Custom Query (2152 matches)

Reported by pudo, 3 years ago.

We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include:

• CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
• Catalogue scraping for LOD2 experiments (yields RDF graphs)
• Atom/DCat for LOD2 production (yields RDF graphs)
• OAI-PMH for ​http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios.

In general, such a system would have the following stages:

• Source selection: find what to download/scrape/harvest/parse
• Index retrieval (i.e. package index)
• Item retrieval (i.e. package entity)
• (Optional: Serialization)
• Normalisation
• Loading/Merging? into CKAN

Exisiting harvesters are at:

• CSW: ​https://bitbucket.org/okfn/ckanext-csw/src/
• Scraper+CKAN: ​https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/

Reported by pudo, 3 years ago.

Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class.

cf. ​https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16

Reported by pudo, 3 years ago.

We're currently setting the user preferences links via a javascript snippet that also evals the name. This should be removed as we're not displaying the user name any longer.

We should also have page fragment caching in Genshi, which is not currently implmented.

cf. ​http://api.rubyonrails.org/classes/ActionController/Caching/Fragments.html