Ticket #1797 (closed enhancement: fixed)

Opened 2 years ago

Last modified 2 years ago

[super] Webstore v2 - Integrated and based on ElasticSearch

Reported by: rgrp Owned by: rgrp
Priority: critical Milestone: ckan-sprint-2012-03-05
Component: ckan Keywords: [4d]
Cc: Repository: ckan
Theme: none

Description (last modified by rgrp) (diff)

Main tasks:

  • Integrated seamlessly into CKAN (authorization etc)
  • Based off ElasticSearch?

Etherpad: http://ckan.okfnpad.org/webstore

Tasks

  • New API point (config + controller) [0.75d]
    • /api/data/{resource-id}
  • Authorization [0.5d]
  • Documentation [0.25d]
  • Upgrade data viewer to use ES [0.5d]
  • Convert webstorer to new setup - http://github.com/okfn/ckanext-webstorer [1d] - #1753
    • See below
  • "Create DataStore? Resource" in Resource creation (??)
    • Just creates a resource with blank url and datastore enabled (webstore_url non null)
  • Tick box to enable DataStore? in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)
    • Also need to use this info in DataStore? extension?
  • Create nice instructions on Resource page about using the DataStore?
  • Rename webstore -> datastore
  • Deploy [0.5d]

Details

Layout on ES

elasticsearch/{index}/{type-id}

index = ckan-{ckan-id}
type-id = resource-id (1)

Request path

Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results

See as an example: https://github.com/okfn/elastic-proxy

Questions

  1. How do we decide what to use in the viewer e.g. Webstore data versus DataProxy?
    • Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't.
  2. How do we decide when webstore is activated
    • Ans: see previous.
  3. Do we (and how do we) allow for resources which are just for webstores
    • Ans: we have to set webstore_url = True ... on save. Or add a big button saying: "Create Blank Resource" ...
  4. How does webstorer work ...
    • CSV import - NS: make sure we add e.g. rownum or id = rownum ...
    • Separate mini-service (webstorer.thedatahub.org ...)
  5. How do we selectively disable recline where not appropriate (e.g. query support makes no sense for DataProxy?)
    • Ans: do not bother at present. Focus on Webstore route.
  6. Where does geodata support come in (in Recline or separately)
    • This is a separate issue

User Story 1

User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save

=> Result = New Dataset + Resource + Data API to resource and nice Preview of the data

Solution:

  1. Allow normal upload process
  2. Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database.
  3. Update the resource in some way (set webstore_url)

User Stories

User Story 2

User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise)

=> Result = New Dataset + Resource + nice Preview of the data (wherever possible)

User Story 3

User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store.

=> Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user

Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...)

Change History

comment:1 Changed 2 years ago by rgrp

  • Keywords [3d] added

comment:2 Changed 2 years ago by rgrp

  • Keywords [4d] added; [3d] removed
  • Description modified (diff)

comment:3 Changed 2 years ago by rgrp

  • Description modified (diff)

New API endpoint, Authorization and documentation done and merged to master. Time so far. 1.5d. https://github.com/okfn/ckan/commit/a054071e2e29e70e7cfa69df8c117ad5d5871a24

comment:4 Changed 2 years ago by rgrp

  • Description modified (diff)
  • Milestone changed from ckan-v1.7 to current-ckan-sprint-2012-03-05

comment:5 Changed 2 years ago by rgrp

  • Description modified (diff)

comment:6 Changed 2 years ago by rgrp

  • Description modified (diff)

comment:7 Changed 2 years ago by rgrp

  • Status changed from new to closed
  • Resolution set to fixed
  • Description modified (diff)
Note: See TracTickets for help on using tickets.