Changes between Initial Version and Version 4 of Ticket #1574


Ignore:
Timestamp:
01/06/12 10:52:44 (2 years ago)
Author:
ross
Comment:

Updated description.

Legend:

Unmodified
Added
Removed
Modified
  • Ticket #1574 – Description

    initial v4  
    1 It would be great to allow uploading of files to push the data into webstore.  Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore. 
     1Allow uploading of files to push the data into webstore.  Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore. 
    22 
    3 1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. 
     31. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs 
    44 
    5 2. The file upload is performed by ckanext-storage to whichever data sink was configured. 
     52. The user is provided the link to the file as currently.  
    66 
    7 3. The user is provided the link to the file as they are currently, except the link is to a short-link, a la bit.ly, which resolves to the file itself. [Note: this may not be necessary, we may be able to manage this with resource properties] 
     73. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) 
    88 
    9 4. A configurable celery task checks the uploaded content and decides what to do with the file based on the mime-type or the file size, or a combination of the two.  In some cases this will send the file up to  webstore from wherever it was uploaded to. 
     9        3a. It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading    
     10            the original file. 
    1011 
    11 5. The short link code [or resource]  is updated to point to the new location (e.g. changed  from http://ckaninstance/ to http://webstoreinstance/) so that future requests will go to the correct location. 
    12  
    13 6. After each file has been processed, the archiver will determine whether the file is deleted, kept or moved to an archive [Do we need to make sure some tasks only happen in sequence?] 
    14  
    15 This would require some change to the webstore to have it interoperate more cleanly with CKAN - Username handling should be modified to allow use of IDs rather than  trying to mangle CKAN usernames that don't fit the current scheme, and we need to change to using the API rather than the DB directly (see #1550) 
    16  
    17 A new celery task would be necessary in ckanext-archiver although it would beat some resemblance to ckanext-webstorer. 
    18  
    19          
     124. The archiver will continue to move the file into permanent storage as per current practice.