CKAN Search Engine
Use Cases
As a user of a CKAN instance I want to be able to make complicated searches, referencing the data fields.
Design
Search technology - Apache SOLR is selected
Architecture: SOLR to work both alongside and as a replacement for the existing full text search in CKAN.
There are two main options for getting data into SOLR:
- POST the records to SOLR in XML format (docs)
- Direct connection Setup SOLR (docs)
- Provide SELECT statements to do queries
- Process is initiated by doing a GET to a particular SOLR URL
The preference is for the first option as the abstraction provides more flexibility in the db and more control about what gets indexed.
When to index a package? Currently we index it on database after_insert and after_update triggers. But this might seriously slow down a large data import since the indexing requires a POST over the internet. Maybe keep the triggers, but for a batch import we can turn them off and then manually run the indexing. Alternatively store up changes and do an hourly cron.
Tickets
1 Get a SOLR instance running, using basic config.
2 Get indexing and searching working with name and title fields only:
- Harness one of the three python SOLR libraries to send SOLR Update XML of CKAN Packages (triggered on the command-line).
- Write tests for SOLR by sending data with SOLR library and using JSON interface for queries.
3 Get it working with all package fields, optimising the field descriptions in schema.xml.
4 Trigger the indexing sensibly (as decided above).
5 Provide option to connect CKAN's search WUI to SOLR back-end.
6 CKAN Developer docs - description of how to setup SOLR link and schema.xml.