CKAN: Ticket #1037: More Robust Harvesting for DGU

repo, theme, milestone set

thejimmyg — Mon, 28 Mar 2011 09:47:18 GMT

thejimmyg — Mon, 04 Apr 2011 09:41:55 GMT

thejimmyg — Mon, 18 Apr 2011 08:56:40 GMT

We spent last week integrating the new harvesting architecture and testing the code but there are still some areas that need looking at

The source type and label should be part of the plugin, not named in DGU.
Need warnings if a document changes but its date doesn't -> do we have these?
I noticed there are some tests in DGU, should these perhaps be in ckanext-harvest?
If active is False, the job should not be put on the queue
Log if the wrong type of URL is entered as an error the user can see
Deny if the source is already registered
Overwrite all extras, not just merge new ones.
During the import stage use iswms.py to add an extra during import if it is a WMS so that we can add a link to the WMS later https://gist.github.com/900878
Can errors/warnings be logged in the import stage? Do all fetched documents get passed to import in one go?

thejimmyg — Mon, 09 May 2011 10:40:01 GMT

Closing this now, any outstanding small issues will be logged in new tickets.