CKAN: Ticket #1797: [super] Webstore v2 - Integrated and based on ElasticSearch

CKAN: Ticket #1797: [super] Webstore v2 - Integrated and based on ElasticSearch http://localhost/ticket/1797 <p> Main tasks: </p> <ul><li>Integrated seamlessly into CKAN (authorization etc) </li><li>Based off <a class="missing wiki">ElasticSearch?</a> </li></ul><p> Etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/webstore"><span class="icon"></span>http://ckan.okfnpad.org/webstore</a> </p> <h1 id="Tasks">Tasks</h1> <ul><li><del>New API point (config + controller) [0.75d]</del> <ul><li>/api/data/{resource-id} </li></ul></li><li><del>Authorization [0.5d]</del> </li><li><del>Documentation [0.25d]</del> </li><li>Upgrade data viewer to use ES [0.5d] </li><li>Convert webstorer to new setup - <a class="ext-link" href="http://github.com/okfn/ckanext-webstorer"><span class="icon"></span>http://github.com/okfn/ckanext-webstorer</a> [1d] <ul><li>See below </li></ul></li><li>"Create Webstore Resource" in Resource creation (??) </li><li>Deploy [0.5d] </li></ul><h1 id="Details">Details</h1> <h2 id="LayoutonES">Layout on ES</h2> <pre class="wiki">elasticsearch/{index}/{type-id} index = ckan-{ckan-id} type-id = resource-id (1) </pre><h2 id="Requestpath">Request path</h2> <p> Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results </p> <p> See as an example: <a class="ext-link" href="https://github.com/okfn/elastic-proxy"><span class="icon"></span>https://github.com/okfn/elastic-proxy</a> </p> <h2 id="Questions">Questions</h2> <ol><li>How do we decide what to use in the viewer e.g. Webstore data versus <a class="missing wiki">DataProxy?</a> <ul><li>Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't. </li></ul></li><li>How do we decide when webstore is activated <ul><li>Ans: see previous. </li></ul></li><li>Do we (and how do we) allow for resources which are just for webstores <ul><li>Ans: we have to set webstore_url = True ... on save. Or add a big button saying: "Create Blank Resource" ... </li></ul></li><li>How does webstorer work ... <ul><li>CSV import - NS: make sure we add e.g. rownum or <span class="underline">id</span> = rownum ... </li><li>Separate mini-service (webstorer.thedatahub.org ...) </li></ul></li><li>How do we selectively disable recline where not appropriate (e.g. query support makes no sense for <a class="missing wiki">DataProxy?</a>) <ul><li>Ans: do not bother at present. Focus on Webstore route. </li></ul></li><li>Where does geodata support come in (in Recline or separately) <ul><li>This is a separate issue </li></ul></li></ol><h2 id="UserStory1">User Story 1</h2> <p> User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save </p> <p> => Result = New Dataset + Resource + Data API to resource and nice Preview of the data </p> <p> Solution: </p> <ol><li>Allow normal upload process </li><li>Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database. </li><li>Update the resource in some way (set webstore_url) </li></ol><h1 id="UserStories">User Stories</h1> <h2 id="UserStory2">User Story 2</h2> <p> User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise) </p> <p> => Result = New Dataset + Resource + nice Preview of the data (wherever possible) </p> <h2 id="UserStory3">User Story 3</h2> <p> User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store. </p> <p> => Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user </p> <p> Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...) </p> en-us CKAN http://assets.okfn.org/p/ckan/img/ckan_logo_shortname.png http://localhost/ticket/1797 Trac 0.12.3 rgrp Wed, 15 Feb 2012 11:44:35 GMT keywords set http://localhost/ticket/1797#comment:1 http://localhost/ticket/1797#comment:1 <ul> <li><strong>keywords</strong> <em>[3d]</em> added </li> </ul> Ticket rgrp Wed, 29 Feb 2012 21:16:59 GMT keywords, description changed http://localhost/ticket/1797#comment:2 http://localhost/ticket/1797#comment:2 <ul> <li><strong>keywords</strong> <em>[4d]</em> added; <em>[3d]</em> removed </li> <li><strong>description</strong> modified (<a href="/ticket/1797?action=diff&version=2">diff</a>) </li> </ul> Ticket rgrp Wed, 29 Feb 2012 21:18:24 GMT description changed http://localhost/ticket/1797#comment:3 http://localhost/ticket/1797#comment:3 <ul> <li><strong>description</strong> modified (<a href="/ticket/1797?action=diff&version=3">diff</a>) </li> </ul> <p> New API endpoint, Authorization and documentation done and merged to master. Time so far. 1.5d. <a class="ext-link" href="https://github.com/okfn/ckan/commit/a054071e2e29e70e7cfa69df8c117ad5d5871a24"><span class="icon"></span>https://github.com/okfn/ckan/commit/a054071e2e29e70e7cfa69df8c117ad5d5871a24</a> </p> Ticket rgrp Sun, 04 Mar 2012 11:19:41 GMT description, milestone changed http://localhost/ticket/1797#comment:4 http://localhost/ticket/1797#comment:4 <ul> <li><strong>description</strong> modified (<a href="/ticket/1797?action=diff&version=4">diff</a>) </li> <li><strong>milestone</strong> changed from <em>ckan-v1.7</em> to <em>current-ckan-sprint-2012-03-05</em> </li> </ul> Ticket rgrp Sun, 04 Mar 2012 12:20:39 GMT description changed http://localhost/ticket/1797#comment:5 http://localhost/ticket/1797#comment:5 <ul> <li><strong>description</strong> modified (<a href="/ticket/1797?action=diff&version=5">diff</a>) </li> </ul> <p> Data Viewer support for new <a class="missing wiki">DataStore?</a> in <a class="ext-link" href="https://github.com/okfn/ckan/commit/9ab8b0283bb086eb4cd663ff73c27066bdd3c79a"><span class="icon"></span>https://github.com/okfn/ckan/commit/9ab8b0283bb086eb4cd663ff73c27066bdd3c79a</a> </p> Ticket rgrp Mon, 05 Mar 2012 00:47:44 GMT description changed http://localhost/ticket/1797#comment:6 http://localhost/ticket/1797#comment:6 <ul> <li><strong>description</strong> modified (<a href="/ticket/1797?action=diff&version=6">diff</a>) </li> </ul> Ticket rgrp Sat, 10 Mar 2012 20:47:49 GMT status, description changed; resolution set http://localhost/ticket/1797#comment:7 http://localhost/ticket/1797#comment:7 <ul> <li><strong>status</strong> changed from <em>new</em> to <em>closed</em> </li> <li><strong>resolution</strong> set to <em>fixed</em> </li> <li><strong>description</strong> modified (<a href="/ticket/1797?action=diff&version=7">diff</a>) </li> </ul> <p> All done!! </p> <p> Help for data api done in <a class="ext-link" href="https://github.com/okfn/ckan/commit/1d8e464f8542d4c33286bb93f4de50060665799f"><span class="icon"></span>https://github.com/okfn/ckan/commit/1d8e464f8542d4c33286bb93f4de50060665799f</a> </p> <p> Checkbox for datastore enabled in <a class="ext-link" href="https://github.com/okfn/ckan/commit/3f1320cd92ae0e775fde1b5eada156260c55e0a6"><span class="icon"></span>https://github.com/okfn/ckan/commit/3f1320cd92ae0e775fde1b5eada156260c55e0a6</a> </p> Ticket rgrp Sat, 10 Mar 2012 20:50:44 GMT http://localhost/ticket/1797#comment:8 http://localhost/ticket/1797#comment:8 <p> Merge fix <a class="ext-link" href="https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547"><span class="icon"></span>https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547</a> </p> Ticket