CKAN: Ticket #888: Improvements to the dataproxy and the data API

Stiivi — Wed, 12 Jan 2011 11:04:57 GMT

Chages to Data Proxy:

use brewery for transformations (included reference to brewery framework in a new vendor directory)
side effect: code to make google find external packages in vendor directory (from now on, all external packages should go there and be referenced from .hgsub if they are mercurial repositories)

changed response contents: moved from 'headers' to root, renamed 'response' to 'data', added field list as 'fields'
changed way of registering transformers (now class object is used instead of name)
added 'encoding' and 'dialect' parameters for CSV
added optional data audit (parameter 'audit')

owner, status changed; repo, theme set

thejimmyg — Mon, 06 Jun 2011 09:28:57 GMT

I don't think any progress has been made on this for a bit so I'm assigning it to me.

shevski — Fri, 08 Jul 2011 13:56:22 GMT

johnglover — Wed, 27 Jul 2011 13:25:03 GMT

Dataproxy / Dataapi now deprecated in favour of the combination of new QA archive / process commands and the webstore.

Changes in relation to Dataproxy / Dataapi:

Currently only supports CSV files, but plans to add support for excel and google docs spreadsheets soon.
Uses David Raznick's CSV parser instead of Brewery for parsing, handles messy CSV data better.

Changes in relation to old QA functionality:

decoupled archiving (downloading) and QA process
added a new 'process' command which parses downloaded files and adds them to a local webstore

Closing for now, any improvements/feature requests should be in tickets relating to either the QA functionality or the webstore.