Ticket #402 (closed task: duplicate)

Opened 4 years ago

Last modified 3 years ago

Archiving worker to back up package resources from a CKAN instance

Reported by: pudo Owned by: pudo
Priority: critical Milestone: ckan-v1.3
Component: ckan Keywords:
Cc: Repository:
Theme:

Description (last modified by rgrp) (diff)

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

  • Naming scheme?
    • Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
      • What happens if names change
    • File: filename? hash?
  • Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages.

Extra points for:

  • Properly checking for source file modification (Last-modified, Etag)
  • Using PIP VCS Backends for retrieval
  • OFS/S3 Storage

Change History

comment:1 Changed 4 years ago by pudo

  • Description modified (diff)

comment:2 Changed 4 years ago by pudo

This is beginning to go somewhere:

comment:3 Changed 3 years ago by rgrp

  • Owner set to pudo
  • Priority set to critical
  • Component set to ckan
  • Description modified (diff)
  • Milestone set to ckan-v1.3

comment:4 Changed 3 years ago by rgrp

  • Summary changed from Write archival/caching worker to Archiving worker to back up package resources from a CKAN instance

comment:5 Changed 3 years ago by pudo

  • Status changed from new to closed
  • Resolution set to duplicate

De-dup: #891

Note: See TracTickets for help on using tickets.