Custom Query (2152 matches)

Reported by pudo, 4 years ago.

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

• Naming scheme?
  • Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
    • What happens if names change
  • File: filename? hash?
• Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages.

Extra points for:

• Properly checking for source file modification (Last-modified, Etag)
• Using PIP VCS Backends for retrieval
• OFS/S3 Storage

Reported by ross, 22 months ago.

We need to check the archiver to make sure it works as we would expect after the changes in #2732:

• Receive notification of a file that needs processing
• Check the file's DB entry and upload the file to the configured remote service
• Update the URL of the file (both in the file's DB entry and the resource)
• Once we're sure the file is safely archived, we should mark the file as being archived

so that we know it can safely be deleted at some point.

Maybe this could/should be implemented as a service rather than a celery task?

Reported by johnglover, 3 years ago.

For example:

• name and title should have a higher weight than the notes field