Custom Query (2152 matches)

Reported by rgrp, 3 years ago.

For some time (e.g. 1y+!) we have known that we want to integrate some kind of datastore / data processing system with CKAN. We've had a CREP in progress on this for some months (may copy that here at some point):

​http://wiki.ckan.org/CEP0004

We can distinguish 3 modules that are needed:

1. "Webstore": A datastore with dataapi - #1208

Suggestion is this would be sqlite based with a simple sql based API. ​http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query}

2. Automated conversion of suitable resources into datastore upon resource creation so that e.g. they are accessible via the API. #1398

3. A data processing system which utilizes this datastore. One could

get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) - #1432

More info

• Overview diagram: ​https://docs.google.com/drawings/d/1XK7dcpFXNlMzVFgLPYPZUXOsFRPzgCEvS-w7FcvydEQ/edit?hl=en_GB

Reported by rgrp, 3 years ago.

• SQLite based
• Use Cyclone for async http server on top with auth
• Designed to not be specific to CKAN or anything else

Current code is here: ​https://github.com/okfn/webstore.

API Spec

Read

Two basic ways to query::

    GET: /{owner}/{db-name}/?sql=...
    GET: /{owner}/{db-name}/?table=...&attr=value&attr=value&limit=...

Returns::

  {
      u'keys': [u'id', u'name'],
      u'data': [
          [1, u'jones'],
          [u'aaa', u'jones']
          ]
  }

Write

POST to::

/{owner/{database}/{table}

Payload is json data structured as follows::

{

unique_keys: [list of key attributes] data: {dict of values}

}

Authentication and Authorization

Authentication: use basic auth header.

Authorization:

• Default: all read, owner can write
• Restricted: owner can read and write, everyone can do nothing

Possible future: config file can specify a python method (TODO: method signature)

Integration with Other Systems

TODO: Specify how to delegate authenatication to user database in some other system.

Reported by rgrp, 3 years ago.

See reports on list at:

• ​http://lists.okfn.org/pipermail/ckan-dev/2011-July/001103.html
• ​http://lists.okfn.org/pipermail/ckan-dev/2011-July/001107.html

Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix.