Resource format normalization and detection

[dread]

Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (​https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service.

Sub-tasks:

• Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -> "text/csv" and "Name mappings", i.e. "text/csv" -> "Comma-Separated Spreadsheet".
• Collect and map surface forms from all CKANs
• Access this via Swiss and apply, store as a PackageResource? extra field pending #826 (Resource extras).
• Add heuristics for format auto-detections:
  • Map well-known file extensions
  • Recognize obvious magic (Zip, Tar)
  • Peek into Zipfile/Tarfiles?
• Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar)
• See also: #816 (Autocomplete for the resource format field)

[pudo]

The first three sub-items of this ticket are done in datautil and dcat-tools:

Basic GDocs-based normalizer:

• ​https://bitbucket.org/okfn/datautil/src/8bba83b4fa45/datautil/normalization/table_based.py

Example of use:

• ​https://bitbucket.org/okfn/dcat-tools/src/0ec5012bf12a/dcat/core/normalize.py#cl-32

Spreadsheet (as referenced in datautil source, should be a kwarg):

• ​https://spreadsheets.google.com/ccc?key=0AplklDf0nYxWdE8tVlRrN1F3bG9PdDBFUDNZcENDNEE&hl=en#gid=0

Experience so far has been that Google rate limits the current implementation so we should perform all ops in one or two big calls rather than "piece by piece".

[icmurray]

Moved to ckan-future and unassign so that this ticket will be picked up in triage.

[icmurray]

Replying to icmurray:

Moved to ckan-future and unassign so that this ticket will be picked up in triage.

Sorry, this comment bears no resemblemnce to what I actually did! Assigned to tobes for 1.9.