Custom Query (2152 matches)

Reported by thejimmyg, 3 years ago.

CKAN's harvesting facility is now live on DGU but there are some major improvements that could be made to make it more robust and better fit the generic CKAN harvesting framework proposed in #987.

Some of the key issues:

• Error reports do not currently contain the ID or title of the document with the error.
• We only have "added" and "error" logging on jobs when we really need a report of "added", "updated", "not changed" and "errors" with the items in each referencing a real metadata document for which harvesting was attempted
• We need deletion and editing of sources, without deleting the harvested documents or packages
• We need a more robust harvesting mechanism than a cron job or we need to deal with the case of multiple cron jobs running at once.
• We need to know the last time a list of documents was scheduled for harvest and the last time each one was fetched.

Reported by amercader, 3 years ago.

It could be interesting for the publicdata.eu project to add some CSW datasources. Thanks to the INSPIRE directive, there are quite a few national catalogs of geodata that could be worth harvesting. Work on this should include:

• Compose Endpoint List (1d)
• Harvest on staging system and evaluate which sources have useful material (2d)

First point is being performed in this GDoc: ​https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&authkey=CNu4hsEB&hl=en_US#gid=0

Reported by amercader, 3 years ago.

We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city).

Relevant discussion: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html