Custom Query (2152 matches)

Reported by pudo, 3 years ago.

We've fixed a few things in CKAN that were discovered and tracked by HRI in their own issue tacker. This includes:

• add authz checks to package edit links (cset:0752316cd2fe)
• replace gettext with _ in controllers, to support unicode error msgs (cset:822340e6077e)
• handle broken html in notes field without crashing on package read (cset:4b6be037dda0)
• update i18n (cset:37d57dc3c492, cset:ea03173f5e77, cset:f16f4ee40fe7)

Reported by pudo, 3 years ago.

• data.london.gov.uk (1d, ping DERI)
• catalogues from Paris and Vienna (1d)

Others?

• digitaliser.dk (1d)
• data-publica.com (2d)
• data.suomi.fi (2d)
• dati.piedmonte.it (1d)
• portalu.de (2d)

Reported by pudo, 3 years ago.

We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include:

• CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
• Catalogue scraping for LOD2 experiments (yields RDF graphs)
• Atom/DCat for LOD2 production (yields RDF graphs)
• OAI-PMH for ​http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios.

In general, such a system would have the following stages:

• Source selection: find what to download/scrape/harvest/parse
• Index retrieval (i.e. package index)
• Item retrieval (i.e. package entity)
• (Optional: Serialization)
• Normalisation
• Loading/Merging? into CKAN

Exisiting harvesters are at:

• CSW: ​https://bitbucket.org/okfn/ckanext-csw/src/
• Scraper+CKAN: ​https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/