http://localhost/ticket/1574 <p> It would be great to allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore. </p> <ol><li>A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. </li></ol><ol start="2"><li>The file upload is performed by ckanext-storage to whichever data sink was configured. </li></ol><ol start="3"><li>The user is provided the link to the file as they are currently, except the link is to a short-link, a la bit.ly, which resolves to the file itself. [Note: this may not be necessary, we may be able to manage this with resource properties] </li></ol><ol start="4"><li>A configurable celery task checks the uploaded content and decides what to do with the file based on the mime-type or the file size, or a combination of the two. In some cases this will send the file up to webstore from wherever it was uploaded to. </li></ol><ol start="5"><li>The short link code [or resource] is updated to point to the new location (e.g. changed from <a class="ext-link" href="http://ckaninstance/"><span class="icon">​</span>http://ckaninstance/</a> to <a class="ext-link" href="http://webstoreinstance/"><span class="icon">​</span>http://webstoreinstance/</a>) so that future requests will go to the correct location. </li></ol><ol start="6"><li>After each file has been processed, the archiver will determine whether the file is deleted, kept or moved to an archive [Do we need to make sure some tasks only happen in sequence?] </li></ol><p> This would require some change to the webstore to have it interoperate more cleanly with CKAN - Username handling should be modified to allow use of IDs rather than trying to mangle CKAN usernames that don't fit the current scheme, and we need to change to using the API rather than the DB directly (see <a class="assigned ticket" href="http://localhost/ticket/1550" title="enhancement: Allow simple auth via the API (assigned)">#1550</a>) </p> <p> A new celery task would be necessary in ckanext-archiver although it would beat some resemblance to ckanext-webstorer. </p> <p> </p> en-us http://assets.okfn.org/p/ckan/img/ckan_logo_shortname.png http://localhost/ticket/1574 Trac 0.12.3 ross Tue, 03 Jan 2012 09:18:18 GMT http://localhost/ticket/1574#comment:1 http://localhost/ticket/1574#comment:1 <p> Related: </p> <blockquote> <p> <a class="closed ticket" href="http://localhost/ticket/1608" title="enhancement: Merge storage into Core (closed: fixed)">#1608</a> Merge storage into Core <a class="closed ticket" href="http://localhost/ticket/1609" title="enhancement: Celery task for ckanext-archiver to write to webstore. (closed: fixed)">#1609</a> Celery task for ckanext-archiver to write to webstore. </p> </blockquote> Ticket ross Tue, 03 Jan 2012 11:30:11 GMT http://localhost/ticket/1574#comment:2 http://localhost/ticket/1574#comment:2 <p> Added <a class="closed ticket" href="http://localhost/ticket/1611" title="enhancement: Implement auth API calls for webstore/external use (closed: duplicate)">#1611</a> as relevant to webstore integration </p> Ticket rgrp Thu, 05 Jan 2012 14:38:56 GMT http://localhost/ticket/1574#comment:3 http://localhost/ticket/1574#comment:3 <p> @ross: Is it possible to update the detailed description of this ticket so it reflects current reality (whatever that is ;-) ) (BTW: you can edit the main descript of the ticket by scrolling to the bottom of this ticket edit page) </p> Ticket ross Fri, 06 Jan 2012 10:52:44 GMT http://localhost/ticket/1574#comment:4 http://localhost/ticket/1574#comment:4 <ul> <li><strong>description</strong> modified (<a href="/ticket/1574?action=diff&amp;version=4">diff</a>) </li> </ul> <p> Updated description. </p> Ticket ross Fri, 06 Jan 2012 10:53:50 GMT http://localhost/ticket/1574#comment:5 http://localhost/ticket/1574#comment:5 <ul> <li><strong>description</strong> modified (<a href="/ticket/1574?action=diff&amp;version=5">diff</a>) </li> </ul> <p> <span class="underline">Some issues to be addressed</span> </p> <ul><li>What about existing content stored on google/s3? </li><li>Metadata for OFS should use username/userid? If user changes username we could just change the pairtree metadata json file, it should not be automated or easy for users to change username though. </li><li>OFS Metadata should contain the form data from CKAN upload </li><li>Remove _owner from OFS metadata (unnecessary) </li></ul><p> <span class="underline">Some issues to still consider</span> </p> <ul><li>Allow .sqlite upload? </li><li>Allow .sql upload? </li></ul> Ticket ross Mon, 09 Jan 2012 12:09:09 GMT http://localhost/ticket/1574#comment:6 http://localhost/ticket/1574#comment:6 <ul> <li><strong>owner</strong> set to <em>ross</em> </li> <li><strong>status</strong> changed from <em>new</em> to <em>assigned</em> </li> </ul> Ticket kindly Thu, 12 Jan 2012 18:26:23 GMT http://localhost/ticket/1574#comment:7 http://localhost/ticket/1574#comment:7 <ul> <li><strong>milestone</strong> changed from <em>ckan-backlog</em> to <em>ckan-v1.6</em> </li> </ul> Ticket rgrp Wed, 18 Jan 2012 16:38:03 GMT http://localhost/ticket/1574#comment:8 http://localhost/ticket/1574#comment:8 <ul> <li><strong>description</strong> modified (<a href="/ticket/1574?action=diff&amp;version=8">diff</a>) </li> </ul> Ticket ross Wed, 18 Jan 2012 16:42:45 GMT http://localhost/ticket/1574#comment:9 http://localhost/ticket/1574#comment:9 <ul> <li><strong>description</strong> modified (<a href="/ticket/1574?action=diff&amp;version=9">diff</a>) </li> </ul> <p> Added related ticket <a class="closed ticket" href="http://localhost/ticket/1681" title="enhancement: Core storage document (closed: fixed)">#1681</a> for documentation/announce to be done asap. </p> Ticket rgrp Wed, 18 Jan 2012 16:53:42 GMT http://localhost/ticket/1574#comment:10 http://localhost/ticket/1574#comment:10 <ul> <li><strong>description</strong> modified (<a href="/ticket/1574?action=diff&amp;version=10">diff</a>) </li> </ul> Ticket ross Fri, 20 Jan 2012 10:56:51 GMT http://localhost/ticket/1574#comment:11 http://localhost/ticket/1574#comment:11 <ul> <li><strong>description</strong> modified (<a href="/ticket/1574?action=diff&amp;version=11">diff</a>) </li> </ul> Ticket rgrp Wed, 15 Feb 2012 20:45:33 GMT http://localhost/ticket/1574#comment:12 http://localhost/ticket/1574#comment:12 <ul> <li><strong>summary</strong> changed from <em>[super] Storage changes</em> to <em>[super] Storage changes including automated save to Webstore</em> </li> </ul> Ticket ross Mon, 28 May 2012 11:02:35 GMT http://localhost/ticket/1574#comment:13 http://localhost/ticket/1574#comment:13 <ul> <li><strong>status</strong> changed from <em>assigned</em> to <em>closed</em> </li> <li><strong>resolution</strong> set to <em>fixed</em> </li> </ul> Ticket