Worker basics

Background workers are small programs that will consume queue notifications generated by CKAN and act upon those. Examples could include web hooks notifiers, email sending processes, the search engine indexer or archival daemons.

• ​API and usage docs
• ​Workers repo

Known workers

• "test" worker (included in ckanextworker)
• ​Ckan EXT Archive - Downloads package data

Current notes

• The search indexer in ckanext really should be a worker and needs to be refactored to reflect that
• Workers are on eu4, ran by the user "worker" - monitored and executed via supervisord.
• Indexers are on eu4, ~okfn/var/ckanext/ckanext/solrindexer - monitored and executed via supervisord.
• We need to handle stale data. Assume, for example, that a package notification is first handeled by a "statisticsworker" that would add some data. The next worker (Emailworker) should now receive an updated version of the package.