Ticket #826 (closed enhancement: fixed)

Opened 3 years ago

Last modified 3 years ago

Resource 'extra' fields

Reported by: dread Owned by: kindly
Priority: critical Milestone: ckan-v1.4-sprint-1
Component: ckan Keywords:
Cc: thejimmyg Repository: ckan
Theme: none


There are a number of extra properties we want to store in a resource:

i.e. this is a much needed enhancement

Need to be able to read/write these values in web interface and API.

(At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.)


I suggest these are arbitrary key/value pairs, just like package extras, reusing PackageExtra? code as much as possible. (Other much more radical alternative is to move model to RDF.)

Change History

comment:1 Changed 3 years ago by pudo

  • Milestone set to iati-4

comment:2 Changed 3 years ago by kindly

  • Owner changed from rgrp to kindly

comment:3 Changed 3 years ago by rgrp

  • Milestone changed from iati-4 to ckan-v1.4-sprint-1

comment:4 Changed 3 years ago by dread

It's great that the extras are added in-line with other Resource properties (as opposed to package extras which are a dict not off 'package' but off 'package.extras').

However, the resource extra field keys are defined in config option "ckan.extra_resource_fields". This config option should be removed - extras need to be entirely flexible for our purposes. (In the next ticket we should make it possible to add/remove both keys and values from the Web UI or API.)

It would also be good to tidy things in this direction: http://wiki.okfn.org/Coding_Standards

I've merged from default to the branch enhancment_826_resource_extra_fields ready for you.

comment:5 Changed 3 years ago by pudo

I want to strongly support david in his call for fully flexible extras: one of my use cases for them is to store the various bits of fallout from an archiving process, such as:

  • last-status
  • last-crawled
  • last-etag
  • last-expires
  • last-md5
  • failure-count
  • fall-back-url

These are things needed to really archive the data well, but they have nothing to do with CKAN core ops. Essentially, the archiver is a seperate concern and it need not appear in the CKAN config.

comment:6 Changed 3 years ago by kindly

There is nothing to stop anyone from putting any extra attributes in the extra_info field dict. So any have the flexibility you need.

The config option is to add some fields that act in exactly the same way as python attributes, having the same semantics as them. i.e if you have an extra field called alturl, you can do obj.alturl = 'fdsffs'.

This is the best of both worlds as far as I can tell.

comment:7 Changed 3 years ago by kindly

I forgot to mention that he main advantage of the fixed fields, is that we can make them properly searchable i.e the values searchable. This currently does not work for package extra values as they are jsons.

I have added this searchability for the sql backend.

comment:8 Changed 3 years ago by dread

Ah - I understand. I'd like it if you could make all the extra fields work as attributes and be searchable - is that possible?

comment:9 Changed 3 years ago by kindly

This would be too much of a hack. You do not want users overwriting any attributes on the object. If they called the attribute "init" it would write over the actual init.

comment:10 Changed 3 years ago by dread

  • Status changed from new to closed
  • Resolution set to fixed

Merged into default in cset:613d7bd5fc96.

Future tickets in this area include:

  • #978 Full edit in Web UI
  • #979 Edit Resource extras in the API

comment:11 Changed 3 years ago by dread

  • Repository set to ckan
  • Theme set to none

This went into ckan release 1.3.2.

Note: See TracTickets for help on using tickets.