CKAN: Ticket #1154: Make ckan robust against solr failure

status, cc, priority changed; owner, milestone, keywords set

johnglover — Mon, 15 Aug 2011 13:03:37 GMT

status changed from new to assigned
cc sebbacon removed
priority changed from awaiting triage to major
owner set to johnglover
milestone set to ckan-current-sprint
keywords search added

status changed; resolution set

johnglover — Tue, 23 Aug 2011 17:28:17 GMT

status changed from assigned to closed
resolution set to fixed

Fixed in core (branch feature-1275-solr-search).

status changed; resolution deleted

dread — Wed, 24 Aug 2011 09:17:57 GMT

status changed from closed to reopened
resolution fixed deleted

Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a useful thing to do? Without search, perhaps CKAN should simply not start?)

I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this time pudo and I get exception emails saying connection failed for request. e.g.

Module ckan.controllers.home:29 in index
<<          query = query_for(model.Package)
               query.run(query='*:*', facet_by=g.facets,
                         limit=0, offset=0, username=c.user)
               c.facets = query.facets
               c.fields = []
>>  limit=0, offset=0, username=c.user)
Module ckan.lib.search.common:115 in run
<<          self.query = QueryParser(query, terms, fields)
               self.query.validate()
               self._run()
               self._format_results()
               return {'results': self.results, 'count': self.count}
>>  self._run()
Module ckanext.solr.backend:63 in _run
<<              # this wrapping will be caught further up in the WUI.
                   log.exception(e)
                   raise SearchError(e)
               finally:
                   conn.close()
>>  raise SearchError(e)
SearchError: [Errno 111] Connection refused
}}} (recently from nl.ckan.net)

description changed

johnglover — Wed, 24 Aug 2011 09:32:05 GMT

description modified (diff)

Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a useful thing to do? Without search, perhaps CKAN should simply not start?)

When discussing the ticket we agreed that we should allow ckan to start without search. It is possible that people simply want to list/view information without searching for it and we decided not to prevent them from doing so.

I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this time pudo and I get exception emails saying connection failed for request.

Ok, other options then:

if an error occurs during search, return 0 results
if an error occurs during search, redirect to a generic 'search is unavailable' page
if an error occurs during search, retry the query a certain number of times before giving up and doing one of the above.

What do you suggest?

dread — Wed, 24 Aug 2011 09:48:10 GMT

Search requests - option one seems like a bad user experience, and option three seems too complicated. I should do option 2, explain it may well be a temporary problem, and let the user press reload, as per any normal web request.

A more difficult problem though is what to do about a failed search indexing. Because of the try/except catch we've had until the patch yesterday, we get no exception emails for this, but I'm worried that this has caused packages to not be indexed, and hence are essentially lost in CKAN (unless someone realises there is a problem and does a reindex at some point).

BTW I added paster command 'search-index check' for looking at this problem, but I don't think it was ever added to solr. Is it easy to add get_all_entity_ids for the purpose of checking for this problem?

johnglover — Wed, 24 Aug 2011 13:11:36 GMT

Ok I'll implement option 2 for search query issues.

Yes it is easy to get a list of IDs from Solr so I'll have a look at that command and make sure that it works. I think a case could be made for not allowing a package to be added if it cannot be indexed however, and making the user simply retry the request.

dread — Wed, 24 Aug 2011 13:25:24 GMT

Yes, forcing a retry may be possible, although I guess on these occasions you'd be looking at rolling back the creation of the package object too, which is not really in the spirit of the notifications system at present. This is why we had the queue for solr for occasions like this - is it here no longer?

johnglover — Wed, 24 Aug 2011 13:32:33 GMT

Not in core, there is a queue feature in the extension still which I could add to core. I think though it was a case of using either the queue or synchronous search, I don't think it would automatically fall back to the queue if there was an indexing problem, but we could look at that. Was there not talk of changing the queue system recently though? There are also no tests for the Solr queue so that would take a bit of work as well.

johnglover — Wed, 24 Aug 2011 13:47:13 GMT

RE search queries failing: Just having a look through the controller code and the package search controller and the search api both perform option 2 already, so I'll just update the home controller.

dread — Wed, 24 Aug 2011 14:21:24 GMT

Ok, I chatted to David about this and we feel it should go like this: notification to search indexer fails and raises an exception, this gets passed through the notification system back up to the code that creates the package, and you do a roll back. Does this tally with what you mentioned in the last comment or is what's there better?

Thanks for the info on the queue and ckanext-solr. It sounds like this isn't deployed successfully anywhere (or does pudo know if it is?). In this case we should be clear that it is not what it says it is: "This extension adds support for package search using Apache Solr to CKAN" and maybe simply mark it as deprecated. Pudo what do you think?

johnglover — Wed, 24 Aug 2011 14:38:03 GMT

Ok, I chatted to David about this and we feel it should go like this: notification to search indexer fails and raises an exception, this gets passed through the notification system back up to the code that creates the package, and you do a roll back. Does this tally with what you mentioned in the last comment or is what's there better?

This sounds good to me, definitely better than what is currently there.

Thanks for the info on the queue and ckanext-solr. It sounds like this isn't deployed successfully anywhere (or does pudo know if it is?).

I'm not actually sure if the queue works or not, but if it does I'll have to write some tests for it before we could add it to core. A working queue system would be useful generally however, as there are other things that it would be useful for (such as QA), but yes, going forward the Solr extension should really be deprecated.

status changed; resolution set

johnglover — Thu, 25 Aug 2011 15:51:59 GMT

status changed from reopened to closed
resolution set to fixed

All attempts to index packages should now throw a SearchIndexError? if Solr is unavailable and the create/update will be rolled back. These errors are caught in the main package controller (redirecting to an error page) and the API controller. I have added tests for both controllers.

Also updated the paster 'search-index check' command to work with solr.