Repro:

• Create a new Group, named "test-group".
• Add a dataset to it.
• Verify the dataset belongs to the group by visiting the Group's read-page
• Update the Group through the action api (group_update), using the uid in the "id" field, and a new name in the "name" field.
• Visit the group's read-page. The list of datasets will be empty.

This was an issue when editing a Group through the web interface, which was fixed in [1]. However it only fixes the issue in the group controller.

[1] ​https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335

On each package page in the openness field if the package is not open provide a link to is it open create enquiry page.

Details:

• may want to add to the url a query parameter indicating which package this relates to e.g. ?ckan-package=...
• ​http://isitopen.ckan.net/enquiry/create/?ckan-package=...
• do we provide link even if package is open (e.g. so that people can make enquiries about e.g. adding open data/content buttons ...)

Cost: 0.5h

Current form stuff is not very good (uses formencode). Switch to formalchemy would improve this, especially on validation.

As an extra we could utilize the formalchemy pylons admin interface (pretty much for free).

Cost: 12h

Details

1. Replace htmlfill and formencode extract in controllers/templates with formalchemy
   1. Crude and simple
   2. Suppress unwanted fields (revision, state, all revisions) and sort out ordering to be similar to before.
   3. Sort out tag field with a special renderer
2. Validation - testing definitely required.
3. Fix up description and pretty css etc

Several things:

• Default package search (used e.g. on front page) should deal well with multiple items: "abc xyz" means search for both abc and xyz (as separate searches)
• Should automatically display a list of matching tags in a line at top with number of associated packages
• have an option to only search for fully open material (or openly licensed material ...)

Since we should be moving to a more sophisticated search solution anyway (to allow prioritisation etc) maybe we should cautious as to what we do now.

Add the following attributes to package:

• author, author_email
• maintainer, maintainer_email

Gives us full compatibility to: ​http://docs.python.org/distutils/setupscript.html#additional-meta-data

Column ordering - should come after name, title, url, download_url.

As a user I want to be able to create groups of packages.

This functionality is different from tags:

• The ability to add a package to a given group is restricted whereas anyone can add a given tag to a package
• Specifically groups have owners and only the owners can add a package to that group

Group properties

• id (primarykey, uuid style)
• name (restricted content - same rules as package)
• title (no restrictions)
• owners -- many:many with user object
• description (markdown)
• packages -- many:many with package object

Don't version groups for now.

Groups address in the WUI will be:

• /group - browse list of groups (reuse list action)
• /group/list - same thing for now
• /group/<groupname> - display: group properties with links to packages. No links to user pages (yet).

Group editing and searching will be another ticket.

Cost: 3d

Add package search facility in the rest api at /api/search

Queries can be provided as for the normal package search either by posting to that url or by performing a get with a query string.

E.g. .../api/search/package?q=xyz

Query parameters:

• q is keyword string (searches name, title, tags by default)
  • split by words and ANDed
• additional parameters for specific fields
• qjson is alternative to q for Searching by specific fields (in addition to keyword string). Payload to json is a JSON-encoded dict which is a dictionary with a q field for free text (keywords) and additional key/value pairs for specific fields
  • if q and qjson specified ignore q and just process qjson value

Additional parameters in addition to query ("q" or "qjson") are:

• limit
• offset
• fullinfo=0/1 - return full record for each result (default=0)
• order_by=field_name
• search_notes=0/1 (default 0) do we search notes field in a keyword search

Return value is json encoded dictionary with keys:

• 'results': list of results
• 'count': total number of results

Extras (for the future)

• Prioritisation is not part of this ticket (requires fulltext support in DB or in external app such as Xapian)

Must not alter the db.

Related to ticket:112.

Based on a section of AccessControl design: Reading and writing to packages in the WUI and REST API are now dependent on the authz tables.

Based on a section of AccessControl design: Group reads and edits are controlled by access control. WUI and REST interfaces covered.

Ensure pending packages don't show up in search or browse

cost: 4h

And associated page to browse group.

Add Group editing page.

If no permissions to change group can't edit group. Also cannot view edit page.

Editable attributes: name, title, description

No preview needed

As a Package Admin I want to change the state of the package. In particular I wish to delete and undelete it.

(NB: this is quite separate from "purging" objects which is the term we shall use for irrevocable removal of an object from the domain model).

• Only Package Admins (and sysadmins) should be able to change state

Implementation Suggestions

• 'delete' action should be renamed to 'change-state' (NB: this requires a db migration ...)
• Have new package formalchemy form (created via inheritance?) to incorporate state attribute. Suggest this is rendered as a dropdown (and may be simple object rendering of state, i.e. do NOT need to change it to a single name such 'active').
• This form should then be used when the user satisfies is_authorized(..., model.Action.CHANGE_STATE) instead of the usual fieldset

Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN.

Cost: 30m

Controlling Groups through a REST interface.

Follows on from ticket:148. Sister to ticket:150.

User object should have a "created" attribute initialized to current datetime.

Require a db migration but o/w very simple.

Cost: 1.5h

This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there.

Suggested by Jonathan Gray

Log message on package edit page is no longer persisted through a preview (no doubt due to recent form upgrades). Let's fix this.

cost: 0.5h

Search of form: tags: ... behaves differently depending on whether there is a leading space:

• tags: postcode - tags found correctly but no packages found
• tags:postcode - tags incorrectly found but correct packages found

Let's fix this.

Cost: 0.5h

E.g. ​http://ckan.net/tag/read/postcodes this has no-postcode listed but when you look at that package 'postcodes' isn't a tag ...

I suspect this may be to do with search using deleted tags as I think no-postcode had postcodes as a tag at some point earlier ...

Ensure logging is working on server.

A 'Derived' relationship can be applied from one package to another.

e.g. sussex-demography is derived from census-2001

'Derived' relationship is:

• directional
• many:many
• stateful

'derived' table columns:

• id (primary key)
• source_package (foreign key)
• result_package (foreign key)
• description (markdown text)

Further tickets:

• WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y
• WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate
• REST if - expose reading and writing this property

Cost: 0.5h (?)

Conjecture this went missing in cset:a35db862a841

See also: PackageResources which defines a 'distribution' as a package's payload (perhaps with metadata). In our case, at least within CKAN, the representation of the distribution will just be the reference to get it (plus some minor metadata). We are expanding this functionality to be any kind of 'resource' associated with a package since it is likely in some cases people will want to link to a file containing a part, but not the whole, of a package's data.

As a

User (package editor and package user)

I want to

(package editor) Associate multiple distributions with a package.

(package user) Easily have access to multiple distributions of a package should they exist.

Implementation

There is an object called 'Resource' with the following attributes:

• url (text) (required in UI, but not in db table)
• format (text)
• description (text)
• package (association) - a resource is always associated with 1 package

A package's list of resources will have an order which is preserved. The first in the list will be known as the 'default resource' (the presumption will always be that this is a 'distribution').

How to replace download url

download_url removed and replaced as a proxy attribute. Read returns the url of the default distribution. Write changes the url of the default resource. Ensures backward compatibility.

REST API has new attribute in package dictionary 'resouces' - an ordered list.

In WUI, replace download url in package read with a table of the distributions.

In the WUI package edit, be able to add/remove/edit/reorder distributions.

Example

Example part of package:

Distributions:

Existing data to be migrated to new structure.

Cost: 4 days

As a

admin / user

I want to

Admin: export all HEAD data to JSON / CSV / RDF format dump.

User: wants to get hold of this dump.

Details

• Just do JSON & CSV for now.
• Put this in CLI.
• Just do list of packages - as_dict info. CSV needs a bit of flattening.

Cost: 2h

As a

admin

I want to

input government data in a constrained way

so that

you get consistency in the db which is easily searched, discovered and automatically linked.

implementation

Form has fields suitable for government data. (related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)

Simple fields

• all basic ckan fields
• default license to OKD::Crown Copyright
• co_id, precision, taxonomy url, agency

Date fields

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

• date_released

• date_updated

• temporal_coverage_from, temporal_coverage_to - two date fields combined

Strict selection

Drop down combo

• department

Selection or free text

Editable combo

• update_frequency - selections: never, weekly, monthly, annually, discontinued

• geographic_granularity - selections: national, regional, local authority, ward, point

• temporal_granularity - selections: yes, months, weeks, days, hours, point

Specialised fields

• geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

• national_statisic - UI is check box, store as 'Yes' or 'No'

• resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.

• categories - multi values from strict selection of ONS categories. Stored as comma-separated.

Field data processed

• tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.

Future - optional

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

Obfuscate author email and maintainer email field.

Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@…', encode='javascript')

Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate?

Package edit page package/edit/mypackagename

• Ability to add the package to groups on this page.
• Fields could be grouped.
• Inconsistent capitalisation (Url -> URL).
• Tag editor YUI script appears to break occasionally.
• Perhaps move to two column forms to save vertical space?
• Notes field to monospace?
• Extras fields need some jQuery love -- shouldn't have a fixed number available.
• Typography is a bit of a mess.

• Requires Package Read to not use the side-bar
• Need to sync params to a package object that is taken out of the session so that it doesn't go into the database.

"Alternative formats". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f).

• How do we know if RDF available? Ans: see details in ticket:90.

Cost: 1h

WUI:

• View: show both sides of the relationship (but think carefully -- e.g. a given package may have *many* dependents ...)

Currently use old "bullet-point" style. Should change to now standard style used on e.g. package search results page.

Cost: 0.5h

Revision.packages should not include all changes so why are we getting revisions with no associated changes? See e.g. ​http://ckan.net/revision/read/814e39e1-2540-4449-90d1-ac9a92ab40fb?

Investigate and fix.

NB: if this is bugfix this should go against stable or metastable branch if at all possible and then be merged into default.

Cost: 1.5h (?)

Create a new package with a name 'blah' and resource format 'blah'. Hit preview. There is an error because of the lack of resource url, but in the resource input boxes, there are now four resources with format 'blah'.

When I edit a package on no.ckan.net with lots of extra fields, lots of them completely disappear.

Metastable branch.

Use Nick's examples as a basis to revamp the formalchemy usage, principally in the package edit form.

• ​http://static.whiteink.com/css/examples/forms.html
• ​http://whiteink.com/temp/ckan-forms/

These don't work.

In Group edit page, you can only add one package at a time. It would be better to use some scripting to allow adding more than one at once, to make it easier and quicker when you have a long list to add.

Ticket is split off from ticket:221

See also ticket:155 (duplicate).

Suggest move this below the label and make full width of screen and only 3/4 rows high (more like a wiki site).

• Also change label to: Edit summary (Briefly describe the changes you have made)
• Remove: you can markdown formatting here.
• Move author: if you have not signed in smaller and closer (like markdown instructions are nwo).
• Change commit -> save
• Remove "please save" just have the bullet points

1. Issues with quote in fields: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html

2. Issues with package resource serialization into csv table.

<quote> In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent. </quote>

As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

• ckan.site_title config variable
• set this on g in app_globals.py e.g.
  • from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
• use in head title and in main site title/logo section (use it as alt on logo image)
• Also all other pages (e.g. index, about) which talk about CKAN
  • Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:

• Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
• Probably yes, but not part of this ticket.

Which events to notify on

Listed by domain object, these are the notification message 'change types' that will be sent:

• Package
• PackageResource

Also it is clear that it could be useful to know when db-wide maintenance is carried out:

• db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration)

Ignored domain objects

These parts of the domain model will not carry notifications as no use case has been identified for them:

• Revision
• Group
• Tag
• Rating
• User - list of users is sensitive info
• Relationships - complicated
• Authz - complicated and sensitive info
• License - change of a license's metadata is a question for the 'license service'

Message format

A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity.

For the 'db' notifications there shall be no payload.

As a

CKAN client program

I want to

be notified when changes to the CKAN metadata occur.

Examples of use

1. An external search engine needing to (re)index a package. (interest: Package)
2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333.
3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: PackageResource)
4. Do some processing on resource (e.g. extract sample data for display) (interest: PackageResource)

Context

The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling.

Design

Split-off into two tickets:

• Notification message - ticket:323
  • Which events to notify on
  • Message format
• Interface for Notifier Service - ticket:322

Testing

To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine.

Related

• Run CKAN search indexing using this system - ticket:324
• Run SOLR indexing using this system - ticket:353
• Web hooks for notifications - ticket:327

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

• Python: ckanclient
• Perl: luke closs wrote something
• PHP: drupal library?
• ...?

As a

third-party interface to a CKAN instance

I want to

link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return.

Design

1. The 'return URL' is passed as a parameter to CKAN.
2. CKAN substitutes the package name into the return URL.

Example

Front-end links to: http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/<NAME> (but with the parameter URL-encoded)

When finished editing and the user commits, CKAN redirects the user to: http://datadotgc.ca/dataset/pollution_data

An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in.

​http://wiki.okfn.org/ckan/doc/theme