Custom Query (2152 matches)

Reported by dread, 4 years ago.

Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead.

The indexing shall run in a separate shell/process, managed by supervisord.

Reported by dread, 4 years ago.

As a

CKAN client program

I want to

be notified when changes to the CKAN metadata occur.

Examples of use

1. An external search engine needing to (re)index a package. (interest: Package)
2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333.
3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: PackageResource)
4. Do some processing on resource (e.g. extract sample data for display) (interest: PackageResource)

Context

The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling.

Design

Split-off into two tickets:

• Notification message - ticket:323
  • Which events to notify on
  • Message format
• Interface for Notifier Service - ticket:322

Testing

To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine.

Related

• Run CKAN search indexing using this system - ticket:324
• Run SOLR indexing using this system - ticket:353
• Web hooks for notifications - ticket:327

Reported by dread, 4 years ago.

Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers.