[super] Storage changes including automated save to Webstore

[ross]

Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore.

1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs [5d]

2. The user is provided the link to the file as currently.

3. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) [5d] - took 6

• It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file.

4. The archiver will continue to move the file into permanent storage as per current practice.

Tickets

• #1608 Merge storage into Core [5d]
• #1609 Celery task for ckanext-archiver to write to webstore. [5d] (took 6)
• #1687: Deploy to datahub [1d]
• #1681: Core storage documentation [1d?]
  • Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc
  • External: here's how to configure it and here's what you get ... (goes in main sphinx docs)

[ross]

Related:

#1608 Merge storage into Core #1609 Celery task for ckanext-archiver to write to webstore.

[ross]

Added #1611 as relevant to webstore integration

[rgrp]

@ross: Is it possible to update the detailed description of this ticket so it reflects current reality (whatever that is ;-) ) (BTW: you can edit the main descript of the ticket by scrolling to the bottom of this ticket edit page)

[ross]

Updated description.

[ross]

Some issues to be addressed

• What about existing content stored on google/s3?
• Metadata for OFS should use username/userid? If user changes username we could just change the pairtree metadata json file, it should not be automated or easy for users to change username though.
• OFS Metadata should contain the form data from CKAN upload
• Remove _owner from OFS metadata (unnecessary)

Some issues to still consider

• Allow .sqlite upload?
• Allow .sql upload?

[ross]

Added related ticket #1681 for documentation/announce to be done asap.