CKAN: Ticket #891: Resource download worker daemon

description changed

pudo — Mon, 03 Jan 2011 11:18:10 GMT

description modified (diff)

milestone changed

pudo — Mon, 03 Jan 2011 11:21:32 GMT

milestone changed from ckan-v1.3 to iati-4

status, description changed; repo, theme, milestone, owner set

rgrp — Thu, 13 Oct 2011 18:13:28 GMT

status changed from new to assigned
description modified (diff)
repo set to ckan
theme set to none
milestone set to ckan-v1.6
owner set to kindly

description changed

rgrp — Fri, 14 Oct 2011 13:46:02 GMT

description modified (diff)

milestone changed

rgrp — Fri, 14 Oct 2011 14:22:08 GMT

milestone changed from ckan-v1.6 to ckan-sprint-2011-10-24

May only do link-checker and not do full storage in this sprint.

description changed

rgrp — Fri, 14 Oct 2011 14:22:55 GMT

description modified (diff)

owner, description changed; keywords set

johnglover — Wed, 19 Oct 2011 10:39:39 GMT

keywords queue added
owner changed from kindly to johnglover
description modified (diff)

milestone changed

johnglover — Tue, 01 Nov 2011 10:30:24 GMT

milestone changed from ckan-sprint-2011-10-24 to current-ckan-sprint-2011-11-07

Almost finished (see http://github.com/okfn/ckanext-archiver).

Still to address:

check headers to see if hash / cache / max-age / expires indicates that the resource does not need to be downloaded.
add cache url to resource

status, description changed; resolution set

johnglover — Tue, 01 Nov 2011 12:17:21 GMT

status changed from assigned to closed
resolution set to fixed
description modified (diff)

Added cache_url and cache_last_updated to resources after archiving.

Not checking for hash value in headers. This process will generally only run when a new resource is added or someone updates a URL, so we don't expect to be regularly downloading the same resource.

We will need something along these lines if this is running as a regular cron job, but in that case the logic will be added to the cron job itself (probably a paster command).