Improvements to the dataproxy and the data API

[rgrp]

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

• Testing for dataproxy
  • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
• Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
• Google docs style row/column selections
• Use the swiss library - ​https://bitbucket.org/okfn/swiss
  • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
• Handle redirects for content-length?
• Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

• Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}

[Stiivi]

Chages to Data Proxy:

• tests added with configurable list of known URLs

• use brewery for transformations (included reference to brewery framework in a new vendor directory)
• side effect: code to make google find external packages in vendor directory (from now on, all external packages should go there and be referenced from .hgsub if they are mercurial repositories)

• changed response contents: moved from 'headers' to root, renamed 'response' to 'data', added field list as 'fields'
• changed way of registering transformers (now class object is used instead of name)
• added 'encoding' and 'dialect' parameters for CSV
• added optional data audit (parameter 'audit')

Changes: ​https://bitbucket.org/Stiivi/dataproxy/changeset/fccbdd275be5

Data information: ​http://databrewery.org/doc/data_quality.html#brewery.dq.FieldStatistics

[thejimmyg]

I don't think any progress has been made on this for a bit so I'm assigning it to me.

[johnglover]

Dataproxy / Dataapi now deprecated in favour of the combination of new QA archive / process commands and the webstore.

Changes in relation to Dataproxy / Dataapi:

• Currently only supports CSV files, but plans to add support for excel and google docs spreadsheets soon.
• Uses David Raznick's CSV parser instead of Brewery for parsing, handles messy CSV data better.

Changes in relation to old QA functionality:

• decoupled archiving (downloading) and QA process
• added a new 'process' command which parses downloaded files and adds them to a local webstore

Closing for now, any improvements/feature requests should be in tickets relating to either the QA functionality or the webstore.