Ticket #891 (assigned task) — at Version 4

Opened 3 years ago

Last modified 3 years ago

Resource download worker daemon

Reported by: pudo Owned by: kindly
Priority: critical Milestone: ckan-sprint-2011-11-07
Component: ckan Keywords: queue
Cc: Repository: ckan
Theme: none

Description (last modified by rgrp) (diff)

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to an OFS repository.

Questions

  • Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
  • Should we have clever ways to dump APIs? ANS: no.
  • Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
  • Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
    • Complete support for ETags
    • Expires, Max-Age etc.
  • Check

Functionality

  • Download files via HTTP, HTTPS (will not do FTP)
  • Respect robots.txt

Process:

  1. [Archiver.Update checks queue (automated as part of celery)]
  2. Open url and get any info from resource on cache / content-length etc
    1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
    2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get hash from headers and hash unchanged GOTO step 4
      • IF: get content-length and content-length unchanged GOTO step 4
      • IF: max-age / expires / other cache headers show this has not changed since last check GOTO step 4
  3. Archive it: connect to storage system and store it. Bucket: from config, Key: /{timestamp}/{resourceid}/filename.ext
    • Add cache url to resource and updated date
    • Add other relevant info to resource such as md5, content-type etc
  4. Update task_status

Optional functionality

  • If result object is HTML, search for references to "proper data" (CSV download pages etc.)
  • Download from POST forms (accepting licenses or weird proprietary systems)
  • Support running on Google Apps Engine to save traffic costs.

Existing work

Change History

comment:1 Changed 3 years ago by pudo

  • Description modified (diff)

comment:2 Changed 3 years ago by pudo

  • Milestone changed from ckan-v1.3 to iati-4

comment:3 Changed 3 years ago by rgrp

  • Status changed from new to assigned
  • Description modified (diff)
  • Repository set to ckan
  • Theme set to none
  • Milestone set to ckan-v1.6
  • Owner set to kindly

comment:4 Changed 3 years ago by rgrp

  • Description modified (diff)
Note: See TracTickets for help on using tickets.