CKAN: Ticket #987: Common harvesting framework http://localhost/ticket/987 <p> We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include: </p> <ul><li>CSW/WFS for INSPIRE/UKLII (yields CKAN packages) </li><li>Catalogue scraping for LOD2 experiments (yields RDF graphs) </li><li>Atom/DCat for LOD2 production (yields RDF graphs) </li><li>OAI-PMH for <a class="ext-link" href="http://datadryad.org/"><span class="icon">​</span>http://datadryad.org/</a> and other dspace (yields CKAN packages) </li></ul><p> We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios. </p> <p> In general, such a system would have the following stages: </p> <ul><li>Source selection: find what to download/scrape/harvest/parse </li><li>Index retrieval (i.e. package index) </li><li>Item retrieval (i.e. package entity) </li><li>(Optional: Serialization) </li><li>Normalisation </li><li><a class="missing wiki">Loading/Merging?</a> into CKAN </li></ul><p> Exisiting harvesters are at: </p> <ul><li>CSW: <a class="ext-link" href="https://bitbucket.org/okfn/ckanext-csw/src/"><span class="icon">​</span>https://bitbucket.org/okfn/ckanext-csw/src/</a> </li><li>Scraper+CKAN: <a class="ext-link" href="https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/"><span class="icon">​</span>https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/</a> </li></ul> en-us CKAN http://assets.okfn.org/p/ckan/img/ckan_logo_shortname.png http://localhost/ticket/987 Trac 0.12.3 thejimmyg Wed, 20 Jul 2011 16:01:45 GMT status changed; repo, theme, resolution set http://localhost/ticket/987#comment:1 http://localhost/ticket/987#comment:1 <ul> <li><strong>repo</strong> set to <em>ckan</em> </li> <li><strong>status</strong> changed from <em>new</em> to <em>closed</em> </li> <li><strong>theme</strong> set to <em>none</em> </li> <li><strong>resolution</strong> set to <em>duplicate</em> </li> </ul> <p> This has largely been implemented now for publicdata.eu. There is a CREP <a class="new ticket" href="http://localhost/ticket/1134" title="CREP: CREP0003: Description and Configuration of Harvesters (new)">#1134</a> outstanding to take the harvesting to the next level so marking this one as duplicate for now. </p> Ticket