CKAN: Ticket #1737: Expose solr-based search API

keywords set

rgrp — Tue, 31 Jan 2012 12:57:18 GMT

keywords 1d added

description changed

rgrp — Thu, 02 Feb 2012 23:22:34 GMT

description modified (diff)

milestone changed

rgrp — Thu, 02 Feb 2012 23:24:15 GMT

milestone changed from ckan-sprint-2012-02-20 to ckan-v1.7

status, milestone changed

rgrp — Mon, 06 Feb 2012 11:47:12 GMT

status changed from new to assigned
milestone changed from ckan-v1.7 to current-ckan-sprint-2012-02-20

keywords changed

rgrp — Mon, 06 Feb 2012 12:07:39 GMT

keywords [1d] added; 1d removed

owner changed

rgrp — Tue, 07 Feb 2012 00:39:05 GMT

owner changed from zephod to rgrp

dread — Wed, 22 Feb 2012 13:19:41 GMT

SOLR syntax is already accepted in:

api/3/search/package
Action API "package_search"

Actually api/1/search/package and api/1/search/package accept SOLR syntax too, but they also translate old-CKAN search parameters syntax to SOLR as well.

See: http://docs.ckan.org/en/latest/apiv3.html

milestone changed

rgrp — Mon, 05 Mar 2012 00:43:55 GMT

milestone changed from current-ckan-sprint-2012-03-05 to ckan-v1.7

Did not get to this sprint as focused on #1797.

keywords, owner changed

icmurray — Mon, 16 Apr 2012 15:47:26 GMT

keywords [1-2d] added; [1d] removed
owner changed from rgrp to icmurray

See http://ckan.okfnpad.org/feature-1737-expose-solr-based-search-api

Immediate actions

analysis of whether the current action/get.py:package_search() function exposes all we currently need for the use cases described above.
how to return that data (expand the current format?)
analysis of whether the current (action) API v.3 can use query parameters rather than as well as POSTed data.

(controlles/api.py:action() ) uses "self._get_request_data()" which in-turn pulls out data from POST body.

Tasks

Extend the existing package_search action.

pass the facet fields into the logic layer from the request parameters.
- /api/3/action/package_search
facet information not being returned via package_search action (is empty).
- curl -X POST -d '{"q": "{!lucene q.op=AND df=text}tags:health +community -profile"}' 'http://thedatahub.org/api/3/action/package_search'
- figure out why it's not working, and what the facet information should look like
whitelist any GETable api actions, and optionally construct the query from url params rather than body

milestone changed

icmurray — Mon, 16 Apr 2012 15:48:17 GMT

milestone changed from ckan-v1.7 to current-ckan-sprint-2012-04-30

status changed; resolution set

icmurray — Thu, 26 Apr 2012 17:18:03 GMT

status changed from assigned to closed
resolution set to worksforme

Facet results are already available through the package_search action api. The facet fields need to be specified in the query, otherwise no faceting will be done (ie - the default facets specified in the .ini file are not used).

This leaves "whitelist any GETable api actions, and optionally construct the query from url params rather than body" todo, which I've pulled out into another ticket, as it wasn't originally in this ticket. (#2330)

icmurray — Thu, 26 Apr 2012 17:23:17 GMT

... just to add to the above:

>>> pprint(json.loads(requests.post('http://127.0.0.1:8088/api/3/action/package_search', json.dumps({'facet.field': ["tags", "groups"]})).content))
<snip>
u'search_facets': {u'groups': {u'items': [{u'count': 1,
                                                        u'display_name': u"Roger's books",
                                                        u'name': u'roger'},
                                                       {u'count': 2,
                                                        u'display_name': u"Dave's books",
                                                        u'name': u'david'}],
                                            u'title': u'groups'},
                                u'tags': {u'items': [{u'count': 1,
                                                      u'display_name': u'tolstoy',
                                                      u'name': u'tolstoy'},
                                                     {u'count': 2,
                                                      u'display_name': u'russian',
                                                      u'name': u'russian'},
                                                     {u'count': 2,
                                                      u'display_name': u'Flexible \u30a1',
                                                      u'name': u'Flexible \u30a1'}],
                                          u'title': u'tags'}}},
 u'success': True}

status, milestone changed; resolution deleted

rgrp — Sun, 06 May 2012 18:29:07 GMT

status changed from closed to reopened
resolution worksforme deleted
milestone changed from ckan-sprint-2012-04-30 to current-ckan-sprint-2012-05-15

Having now thought about this I'm re-opening this ticket for the following reasons:

No real documentation (other than that in this ticket yet available)
- It would also be nice to know how this maps to SOLR API (can i use all of the facet options solr provides or not ...?)
- And I would again emphasize my preference for having *direct* access to something that looks *exactly* like SOLR API as I can then use client and docs from SOLR to work with it.
No clear resolution of separation between Action and REST API (and search API). Really seems to me there should be convergence between latter 2 (as suggested in the ticket) -- this would also resolve the problem that having GET /api/dataset return all datasets is *not* a great idea
The Action API requires a POST request. Since the primary purpose of the search API would be usage from JS it would be nice if GET and JSONP were supported. (Though given our CORS support we could argue this was optional).

Not saying *all* of this needs fixing but some clear approach here would be useful

keywords, milestone changed

icmurray — Wed, 16 May 2012 09:07:58 GMT

keywords [0.5d] added; [1-2d] removed
milestone changed from ckan-sprint-2012-05-15 to current-ckan-sprint-2012-05-29

icmurray — Mon, 21 May 2012 09:59:12 GMT

package_search action is now documented, with reference to the solr search parameters available. http://docs.ckan.org/en/latest/apiv3.html (auto-docs not working on rtd at the moment).

actions defined in get.py are now GETable. http://docs.ckan.org/en/latest/apiv3.html#get-able-actions

we won't be providing direct access to solr api, as we think the cost outweighs the benefit.

cc set

icmurray — Mon, 21 May 2012 09:59:30 GMT

cc rgrp added

rgrp — Fri, 25 May 2012 10:39:05 GMT

I *strongly* disagree re access to solr API -- i don't really care if it is direct but I want something that looks like it at least for core query parameters and facets ...

Is there some major issue around security etc (e.g. limiting to only public datasets or similar?)

icmurray — Fri, 25 May 2012 11:01:11 GMT

For completeness, package_search docs can be viewed at https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L983

@rgrp: are you happy with this ticket now, does it expose all you need? If so, I'll close it...

milestone changed

ross — Mon, 28 May 2012 11:24:59 GMT

milestone changed from current-ckan-sprint-2012-05-29 to ckan-v1.8

rgrp — Sun, 03 Jun 2012 10:27:00 GMT

Let's close. What would be nice though is ETA on the GET support getting deployed on the DataHub? -- just tried using it today and realized it didn't work :-)

status changed; resolution set

icmurray — Mon, 18 Jun 2012 09:16:41 GMT

status changed from reopened to closed
resolution set to fixed

The GET-able actions ( #2330 ) are in master, and will make it into 1.8

rgrp — Mon, 18 Jun 2012 09:20:03 GMT

But not v1.7.1? (When is v1.8 due?).

Also for the record a couple of things I found when trying to use this:

No support for facet sort order or facet limit afaict ...

icmurray — Mon, 18 Jun 2012 10:34:55 GMT

Replying to rgrp:

But not v1.7.1? (When is v1.8 due?).

Not 1.7.1 as it's not a bug fix (https://docs.google.com/document/d/170fxET3kd9dJ4L6VAj3yZugtK0rrVe44J4HuLbTUsEU/)

I don't know when 1.8 is due.

Also for the record a couple of things I found when trying to use this:

No support for facet sort order or facet limit afaict ...

Thanks, that's good to know.

facet.limit should be working [1], so if it's not, then that's a bug. I can check that.

for facet.sort, I've added ticket #2543

[1] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L1022

icmurray — Tue, 19 Jun 2012 13:32:12 GMT

Replying to icmurray:

Also for the record a couple of things I found when trying to use this:

No support for facet sort order or facet limit afaict ...

Thanks, that's good to know.

facet.limit should be working [1], so if it's not, then that's a bug. I can check that.

facet.limit is working as I'd expect. eg (on master):

import requests
import json
from pprint import pprint as pp
pp(json.loads(requests.get('http://ian-laptop:5000/api/3/action/package_search?q=data&facet.field=tags&facet.limit=1').content))

Returns a result where only the tag with the highest count is returned in the search_facets result dict.

One thing I did spot though was that we aren't able to specify per-field parameters. ie whilst facet.limit sets the facet limit for all facet fields, solr allows that to be overridden on a per-field basis, eg facet.tags.limit. This isn't something we support at the moment. I've created a ticket for this, #2573 . Let me know if this something that you'd find useful, otherwise I'll leave it on the backlog for a future iteration.