CKAN Search Engine

Use Cases

As a user of a CKAN instance I want to be able to make complicated searches, referencing the data fields.

Design

Search technology - Apache SOLR is selected

Architecture: SOLR to work both alongside and as a replacement for the existing full text search in CKAN.

There are two main options for getting data into SOLR:

• POST the records to SOLR in XML format (​docs)
• Direct connection Setup SOLR (​docs)
  • Provide SELECT statements to do queries
  • Process is initiated by doing a GET to a particular SOLR URL

The preference is for the first option as the abstraction provides more flexibility in the db and more control about what gets indexed.

When to index a package? Currently we index it on database after_insert and after_update triggers. But this might seriously slow down a large data import since the indexing requires a POST over the internet. Maybe keep the triggers, but for a batch import we can turn them off and then manually run the indexing. Alternatively store up changes and do an hourly cron.

Tickets

1 Get a SOLR instance running, using basic config.

2 Get indexing and searching working with name and title fields only:

• Harness one of the three python SOLR libraries to send SOLR Update XML of CKAN Packages (triggered on the command-line).
• Write tests for SOLR by sending data with SOLR library and using JSON interface for queries.

3 Get it working with all package fields, optimising the field descriptions in schema.xml.

4 Trigger the indexing sensibly (as decided above).

5 Provide option to connect CKAN's search WUI to SOLR back-end.

6 CKAN Developer docs - description of how to setup SOLR link and schema.xml.