[super] Storage changes

[ross]

Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore.

1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs

2. The user is provided the link to the file as currently.

3. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609)

• It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file.

4. The archiver will continue to move the file into permanent storage as per current practice.

Tickets

• #1608 Merge storage into Core
• #1609 Celery task for ckanext-archiver to write to webstore.
• TODO: deploy
• TODO: documentation
  • Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc
  • External: here's how to configure it and here's what you get ... (goes in main sphinx docs)

[ross]

Related:

#1608 Merge storage into Core #1609 Celery task for ckanext-archiver to write to webstore.

[ross]

Added #1611 as relevant to webstore integration

[rgrp]

@ross: Is it possible to update the detailed description of this ticket so it reflects current reality (whatever that is ;-) ) (BTW: you can edit the main descript of the ticket by scrolling to the bottom of this ticket edit page)

[ross]

Updated description.

[ross]

Some issues to be addressed

• What about existing content stored on google/s3?
• Metadata for OFS should use username/userid? If user changes username we could just change the pairtree metadata json file, it should not be automated or easy for users to change username though.
• OFS Metadata should contain the form data from CKAN upload
• Remove _owner from OFS metadata (unnecessary)

Some issues to still consider

• Allow .sqlite upload?
• Allow .sql upload?

[ross]

Added related ticket #1681 for documentation/announce to be done asap.