http://localhost/ticket/891 <p> Superticket: <a class="closed ticket" href="http://localhost/ticket/1397" title="enhancement: [super] Resource archiving (closed: fixed)">#1397</a> </p> <p> Write a worker daemon to download all resources from a CKAN instance to an OFS repository. </p> <h2 id="Questions">Questions</h2> <ul><li>Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues) </li><li>Should we have clever ways to dump APIs? ANS: no. </li><li>Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving) </li><li>Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache. <ul><li>Complete support for ETags </li><li>Expires, Max-Age etc. </li></ul></li><li>Check </li></ul><h2 id="Functionality">Functionality</h2> <ul><li>Download files via HTTP, HTTPS (will not do FTP) </li><li>Respect robots.txt </li></ul><p> Process: </p> <ol><li>[Archiver.Update checks queue (automated as part of celery)] </li><li>Open url and get any info from resource on cache / content-length etc <ol><li>If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery </li><li>Check headers for content-length and content-type ... <ul><li>IF: content-length &gt; max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?) </li><li>ELSE: check content-type. <ul><li>IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource) </li><li>ELSE: archive it (compute md5 hash etc) </li></ul></li><li>IF: get hash from headers and hash unchanged GOTO step 4 </li><li>IF: get content-length and content-length unchanged GOTO step 4 </li><li>IF: max-age / expires / other cache headers show this has not changed since last check GOTO step 4 </li></ul></li></ol></li><li>Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext <ul><li>Add cache url to resource and updated date </li><li>Add other relevant info to resource such as md5, content-type etc </li></ul></li><li>Update task_status </li></ol><p> </p> <h2 id="Optionalfunctionality">Optional functionality</h2> <ul><li>If result object is HTML, search for references to "proper data" (CSV download pages etc.) </li><li>Download from POST forms (accepting licenses or weird proprietary systems) </li><li>Support running on Google Apps Engine to save traffic costs. </li></ul><h2 id="Existingwork">Existing work</h2> <ul><li><a class="ext-link" href="https://bitbucket.org/okfn/ckanext-qa/overview"><span class="icon">​</span>https://bitbucket.org/okfn/ckanext-qa/overview</a> </li><li>out of date: <a class="ext-link" href="https://bitbucket.org/pudo/ckanextarchive"><span class="icon">​</span>https://bitbucket.org/pudo/ckanextarchive</a> - Old archiver extension, largely experimental. </li><li>out of date: <a class="ext-link" href="https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65"><span class="icon">​</span>https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65</a> - Openness scores by ollyc </li></ul> en-us http://assets.okfn.org/p/ckan/img/ckan_logo_shortname.png http://localhost/ticket/891 Trac 0.12.3 pudo Mon, 03 Jan 2011 11:18:10 GMT http://localhost/ticket/891#comment:1 http://localhost/ticket/891#comment:1 <ul> <li><strong>description</strong> modified (<a href="/ticket/891?action=diff&amp;version=1">diff</a>) </li> </ul> Ticket pudo Mon, 03 Jan 2011 11:21:32 GMT http://localhost/ticket/891#comment:2 http://localhost/ticket/891#comment:2 <ul> <li><strong>milestone</strong> changed from <em>ckan-v1.3</em> to <em>iati-4</em> </li> </ul> Ticket rgrp Thu, 13 Oct 2011 18:13:28 GMT http://localhost/ticket/891#comment:3 http://localhost/ticket/891#comment:3 <ul> <li><strong>status</strong> changed from <em>new</em> to <em>assigned</em> </li> <li><strong>description</strong> modified (<a href="/ticket/891?action=diff&amp;version=3">diff</a>) </li> <li><strong>repo</strong> set to <em>ckan</em> </li> <li><strong>theme</strong> set to <em>none</em> </li> <li><strong>milestone</strong> set to <em>ckan-v1.6</em> </li> <li><strong>owner</strong> set to <em>kindly</em> </li> </ul> Ticket rgrp Fri, 14 Oct 2011 13:46:02 GMT http://localhost/ticket/891#comment:4 http://localhost/ticket/891#comment:4 <ul> <li><strong>description</strong> modified (<a href="/ticket/891?action=diff&amp;version=4">diff</a>) </li> </ul> Ticket rgrp Fri, 14 Oct 2011 14:22:08 GMT http://localhost/ticket/891#comment:5 http://localhost/ticket/891#comment:5 <ul> <li><strong>milestone</strong> changed from <em>ckan-v1.6</em> to <em>ckan-sprint-2011-10-24</em> </li> </ul> <p> May only do link-checker and not do full storage in this sprint. </p> Ticket rgrp Fri, 14 Oct 2011 14:22:55 GMT http://localhost/ticket/891#comment:6 http://localhost/ticket/891#comment:6 <ul> <li><strong>description</strong> modified (<a href="/ticket/891?action=diff&amp;version=6">diff</a>) </li> </ul> Ticket johnglover Wed, 19 Oct 2011 10:39:39 GMT http://localhost/ticket/891#comment:7 http://localhost/ticket/891#comment:7 <ul> <li><strong>keywords</strong> <em>queue</em> added </li> <li><strong>owner</strong> changed from <em>kindly</em> to <em>johnglover</em> </li> <li><strong>description</strong> modified (<a href="/ticket/891?action=diff&amp;version=7">diff</a>) </li> </ul> Ticket johnglover Tue, 01 Nov 2011 10:30:24 GMT http://localhost/ticket/891#comment:8 http://localhost/ticket/891#comment:8 <ul> <li><strong>milestone</strong> changed from <em>ckan-sprint-2011-10-24</em> to <em>current-ckan-sprint-2011-11-07</em> </li> </ul> <p> Almost finished (see <a class="ext-link" href="http://github.com/okfn/ckanext-archiver"><span class="icon">​</span>http://github.com/okfn/ckanext-archiver</a>). </p> <p> Still to address: </p> <ul><li>check headers to see if hash / cache / max-age / expires indicates that the resource does not need to be downloaded. </li><li>add cache url to resource </li></ul> Ticket johnglover Tue, 01 Nov 2011 12:17:21 GMT http://localhost/ticket/891#comment:9 http://localhost/ticket/891#comment:9 <ul> <li><strong>status</strong> changed from <em>assigned</em> to <em>closed</em> </li> <li><strong>resolution</strong> set to <em>fixed</em> </li> <li><strong>description</strong> modified (<a href="/ticket/891?action=diff&amp;version=9">diff</a>) </li> </ul> <p> Added cache_url and cache_last_updated to resources after archiving. </p> <p> Not checking for hash value in headers. This process will generally only run when a new resource is added or someone updates a URL, so we don't expect to be regularly downloading the same resource. </p> <p> We will need something along these lines if this is running as a regular cron job, but in that case the logic will be added to the cron job itself (probably a paster command). </p> Ticket