Ticket #1574 (assigned enhancement) — at Version 11

Opened 2 years ago

Last modified 2 years ago

[super] Storage changes

Reported by: ross Owned by: ross
Priority: awaiting triage Milestone: ckan-v1.7
Component: ckan Keywords: storage,archiver
Cc: Repository: ckan
Theme: none

Description (last modified by ross) (diff)

Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore.

  1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs [5d]
  1. The user is provided the link to the file as currently.
  1. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) [5d] - took 6
  • It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file.
  1. The archiver will continue to move the file into permanent storage as per current practice.

Tickets

  • #1608 Merge storage into Core [5d]
  • #1609 Celery task for ckanext-archiver to write to webstore. [5d] (took 6)
  • #1687: Deploy to datahub [1d]
  • #1681: Core storage documentation [1d?]
    • Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc
    • External: here's how to configure it and here's what you get ... (goes in main sphinx docs)

Change History

comment:1 Changed 2 years ago by ross

Related:

#1608 Merge storage into Core #1609 Celery task for ckanext-archiver to write to webstore.

comment:2 Changed 2 years ago by ross

Added #1611 as relevant to webstore integration

comment:3 Changed 2 years ago by rgrp

@ross: Is it possible to update the detailed description of this ticket so it reflects current reality (whatever that is ;-) ) (BTW: you can edit the main descript of the ticket by scrolling to the bottom of this ticket edit page)

comment:4 Changed 2 years ago by ross

  • Description modified (diff)

Updated description.

comment:5 Changed 2 years ago by ross

  • Description modified (diff)

Some issues to be addressed

  • What about existing content stored on google/s3?
  • Metadata for OFS should use username/userid? If user changes username we could just change the pairtree metadata json file, it should not be automated or easy for users to change username though.
  • OFS Metadata should contain the form data from CKAN upload
  • Remove _owner from OFS metadata (unnecessary)

Some issues to still consider

  • Allow .sqlite upload?
  • Allow .sql upload?

comment:6 Changed 2 years ago by ross

  • Owner set to ross
  • Status changed from new to assigned

comment:7 Changed 2 years ago by kindly

  • Milestone changed from ckan-backlog to ckan-v1.6

comment:8 Changed 2 years ago by rgrp

  • Description modified (diff)

comment:9 Changed 2 years ago by ross

  • Description modified (diff)

Added related ticket #1681 for documentation/announce to be done asap.

comment:10 Changed 2 years ago by rgrp

  • Description modified (diff)

comment:11 Changed 2 years ago by ross

  • Description modified (diff)
Note: See TracTickets for help on using tickets.