Resource 'extra' fields

[dread]

There are a number of extra properties we want to store in a resource:

• resource size (in bytes, no. triples etc.) - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/673
• resource hash is 'static' (default) or 'changeable' - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/674
• alternative link (e.g. TNA Web Archive) to use the link is broken - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/173

i.e. this is a much needed enhancement

Need to be able to read/write these values in web interface and API.

(At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.)

Implementation

I suggest these are arbitrary key/value pairs, just like package extras, reusing PackageExtra? code as much as possible. (Other much more radical alternative is to move model to RDF.)

[dread]

It's great that the extras are added in-line with other Resource properties (as opposed to package extras which are a dict not off 'package' but off 'package.extras').

However, the resource extra field keys are defined in config option "ckan.extra_resource_fields". This config option should be removed - extras need to be entirely flexible for our purposes. (In the next ticket we should make it possible to add/remove both keys and values from the Web UI or API.)

It would also be good to tidy things in this direction: ​http://wiki.okfn.org/Coding_Standards

I've merged from default to the branch enhancment_826_resource_extra_fields ready for you.

[pudo]

I want to strongly support david in his call for fully flexible extras: one of my use cases for them is to store the various bits of fallout from an archiving process, such as:

• last-status
• last-crawled
• last-etag
• last-expires
• last-md5
• failure-count
• fall-back-url

These are things needed to really archive the data well, but they have nothing to do with CKAN core ops. Essentially, the archiver is a seperate concern and it need not appear in the CKAN config.

[kindly]

There is nothing to stop anyone from putting any extra attributes in the extra_info field dict. So any have the flexibility you need.

The config option is to add some fields that act in exactly the same way as python attributes, having the same semantics as them. i.e if you have an extra field called alturl, you can do obj.alturl = 'fdsffs'.

This is the best of both worlds as far as I can tell.

[kindly]

I forgot to mention that he main advantage of the fixed fields, is that we can make them properly searchable i.e the values searchable. This currently does not work for package extra values as they are jsons.

I have added this searchability for the sql backend.

[dread]

Ah - I understand. I'd like it if you could make all the extra fields work as attributes and be searchable - is that possible?

[kindly]

This would be too much of a hack. You do not want users overwriting any attributes on the object. If they called the attribute "init" it would write over the actual init.

[dread]

Merged into default in cset:613d7bd5fc96.

Future tickets in this area include:

• #978 Full edit in Web UI
• #979 Edit Resource extras in the API

[dread]

This went into ckan release 1.3.2.