[super] Resource archiving

[rgrp]

We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: ​http://ckan.okfnpad.org/queue (most relevant parts inlined here)

Preliminaries

• Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)

Tasks

1. Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383
2. ckanext-archiver implements IResourceUrlChange and sends tasks to celery. [0.25d][0.25d] - ???
3. Archiver daemon #891
   1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
   2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
4. Write to resource and task status table.[1d][0.75d]
5. Make archived data available in WUI - #892

Archiver process

Archiver:

1. A resource is added to CKAN
2. IResourceCreate event generated
3. IF: resource url points to ckan storage or falls within some other set of exclusion conditions then END else continue
4. Generate a Archiver.Update task with resource.id

Archiver.update: see #891

Link checker: same as Archiver.update up to 2.1