Custom Query (2152 matches)

Reported by pudo, 4 years ago.

At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls.

This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance.

NB: We should also agree on a maximum request latency.

TODO: Read up on all those QoS tickets to avoid overlapping efforts.

Reported by pudo, 4 years ago.

We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app)

Reported by pudo, 4 years ago.

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

• Naming scheme?
  • Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
    • What happens if names change
  • File: filename? hash?
• Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages.

Extra points for:

• Properly checking for source file modification (Last-modified, Etag)
• Using PIP VCS Backends for retrieval
• OFS/S3 Storage