Changes between Initial Version and Version 1 of SearchEngine

Timestamp:

02/24/10 11:50:12 (4 years ago)

Author:

dread

Comment:

--

SearchEngine

@@ +1 @@
+= CKAN Search Engine =
+
+== Use Cases ==
+As a user of a CKAN instance I want to be able to make complicated searches, referencing the data fields.
+
+== Design == 
+
+Search technology - Apache SOLR is selected
+
+Architecture: SOLR to work both alongside and as a replacement for the existing full text search in CKAN.
+
+There are two main options for getting data into SOLR:
+
+ * POST the records to SOLR in XML format ([http://wiki.apache.org/solr/UpdateXmlMessages docs])
+ * Direct connection Setup SOLR ([http://wiki.apache.org/solr/DataImportHandler docs])
+  * Provide SELECT statements to do queries
+  * Process is initiated by doing a GET to a particular SOLR URL
+
+The preference is for the first option as the abstraction provides more flexibility in the db and more control about what gets indexed.
+
+When to index a package? Currently we index it on database after_insert and after_update triggers. But this might seriously slow down a large data import since the indexing requires a POST over the internet. Maybe keep the triggers, but for a batch import we can turn them off and then manually run the indexing. Alternatively store up changes and do an hourly cron.
+
+== Tickets ==
+
+ 1 Get a SOLR instance running, using basic config.
+ 2 Get indexing and searching working with name and title fields only:
+   * Harness one of the three python SOLR libraries to send SOLR Update XML of CKAN Packages (triggered on the command-line). 
+   * Write tests for SOLR by sending data with SOLR library and using JSON interface for queries.
+ 3 Get it working with all package fields, optimising the field descriptions in schema.xml. 
+ 4 Trigger the indexing sensibly (as decided above).
+ 5 Provide option to connect CKAN's search WUI to SOLR back-end.
+ 6 CKAN Developer docs - description of how to setup SOLR link and schema.xml.