Make ckan robust against solr failure

[nils.toedtmann]

According to pudo, a ckan with activated solr extension throws a 5xx when solr is unreachable. Instead, it should behave more like a ckan without ckanext-solr when this happens.

[johnglover]

Fixed in core (branch feature-1275-solr-search).

[dread]

Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a useful thing to do? Without search, perhaps CKAN should simply not start?)

I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this time pudo and I get exception emails saying connection failed for request. e.g.

Module ckan.controllers.home:29 in index
<<          query = query_for(model.Package)
               query.run(query='*:*', facet_by=g.facets,
                         limit=0, offset=0, username=c.user)
               c.facets = query.facets
               c.fields = []
>>  limit=0, offset=0, username=c.user)
Module ckan.lib.search.common:115 in run
<<          self.query = QueryParser(query, terms, fields)
               self.query.validate()
               self._run()
               self._format_results()
               return {'results': self.results, 'count': self.count}
>>  self._run()
Module ckanext.solr.backend:63 in _run
<<              # this wrapping will be caught further up in the WUI.
                   log.exception(e)
                   raise SearchError(e)
               finally:
                   conn.close()
>>  raise SearchError(e)
SearchError: [Errno 111] Connection refused
}}} (recently from nl.ckan.net)

[johnglover]

Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a useful thing to do? Without search, perhaps CKAN should simply not start?)

When discussing the ticket we agreed that we should allow ckan to start without search. It is possible that people simply want to list/view information without searching for it and we decided not to prevent them from doing so.

I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this time pudo and I get exception emails saying connection failed for request.

Ok, other options then:

• if an error occurs during search, return 0 results
• if an error occurs during search, redirect to a generic 'search is unavailable' page
• if an error occurs during search, retry the query a certain number of times before giving up and doing one of the above.

What do you suggest?