• [ ] Write up python coding standards as discussed at dev-meetup into CKAN docs. (Mostly written-up as part of dev-meetup notes.)

• [ ] Collate frontend coding standards (​http://aron.github.com/ckan-style/styleguide/) into same doc(s).

Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN.

Cost: 30m

Need to add capacities to member tables.

Ideally the ./bin/less command would be run when the server is started.

• Also it would be good to have a paster command to build the production CSS with {{{ ./bin/less --production }}
• The command could also detect missing node binaries and redirect to the documentation.

Add the following attributes to package:

• author, author_email
• maintainer, maintainer_email

Gives us full compatibility to: ​http://docs.python.org/distutils/setupscript.html#additional-meta-data

Column ordering - should come after name, title, url, download_url.

Super #1669

As groups need to be authorised in some cases, particularly publisher profiles, we need a field to denote how this is managed.

We should add an 'approval_status' field that will be used to denote its status. By default this will be "approved" but this will allow us to override it with a "pending" or "denied" status. The latter should also change the main status of the Group.

Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML.

There needs to be an extension point where extensions can register their own rendering functions for particular activity types.

Before we do this we should implement smarter group activity streams (i.e. what activities should appear in a group's activity stream?) and following of groups.

This requires adding a logic function for emitting an activity stream event, and then editing the logic functions for creating or updating users and making them call the new emit event function. This same emit event function can later be used to emit activity stream events for other types of object as well.

Add tables described in ​http://ckan.okfnpad.org/notifications

We need a way to group resources by various different factors. The most important of which is timeseries.

The model will look like

Package <o2m> resource_group <o2m> resource

The use cases to this are outlined in this email. ​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

Requests (​http://python-requests.org) is really useful to deal with http requests and saves a lot of time.

I propose we include it as a requirement for the tests for the time being, and even for core if necessary at a later stage.

RDFa can be used as a simple way to expose linked data or at least sameAs the API version of the data but we need to make sure we do not expose anything different from the "official" representation.

The JavaScript? just isn't going to work without it. Look into either the newer JSON3:

​http://bestiejs.github.com/json3/

or the old faithful:

​https://github.com/douglascrockford/JSON-js/

Need to add way for extensions to trigger based on resource change and just resource url change.

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

[Reopened: Additional] Use gravatars in the HTML generated by helpers.py to create lists of users. Eg. on user/list and dataset/history.

EU27 option seems to have been removed at some point, needs to be added back to geographical coverage vocabulary and tested.

• Ensure you can add/change/delete extra fields from packages using the REST API.
• Update API docs to cover this.

On Package get, ensure separation of extra attributes from other attributes.

What is the meaning of posting an incomplete dictionary (updating a package)? Option 1 - if field not present, leave field alone

• Delete field if field value is None

Option 2 - user must supply all fields

• Delete field if field absent

Prefer option 1. Document this.

Follows on from ticket:152

Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new "current" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes:

• Model update (See #1726)
• Update queries for getting harvest sources' datasets
• Update ckanext-csw queries for getting the objects
• Update inspire and ckan harvesters
• Document new field

For example, after deleting a dataset old activity stream events will still hyperlink to that dataset.

Focusing on improving Action API as the v3 API:

• have an optional parameter of the data_dict called "options". Options would contain items that would get passed into the context. e.g. {"options" : {"ref_package_by": "id"}}.
• instead of using API version to change the way packages are referenced, use the ref_package_by.
• All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'.
• Action API is v3 of api, replacement for v1 & v2. Default for most urls is still v1, but if url is /api/action then default to v3.

Next steps:

• Add search API (package, resource,
• Add Util API
• Clarify JSONP still applies
• Add doc strings, clarifying parameters

A good service API needs to be discoverable, so you are not always having to refer to the documentation html.

Maybe /api/action should return a list of actions available? (Currently this returns a 404.)

• It would be nice to sort these into get/create/update/delete.
• #1438 Parameters for each of the actions must be discoverable too

/api/action/{action_name} should also return the help text / parameters allowable. (Currently this returns 400 error)

Provide an API to the logic layer at /api/action.

Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: AccessControl.

Cost: 8d

Create in the model basic operation of Access Control.

roles table

name | context | action
-----------------------
admin| package | edit
admin| package | edit-permissions
admin| package | read
editor| package | update
editor| package | read
reader| package | read
This data is set-up on db init and will have no interface.

user-roles table:

username | context_type | objectid | role
rgrp     | system  | n/a | admin
visitor  | package | * | reader
bob      | package | geonames | admin
visitor  | package | geonames | editor
visitor  | package | geonames | reader
john     | group | ukgov | admin
dread    | group | ukgov | editor
visitor  | group | ukgov | reader
This data will be added when someone is given permissions for the system, a package or a group.

Pseudo code:

class Package

def is_allowed(name, action):

is_allowed(name, action, context=self)

class Group

def is_allowed(name, action):

is_allowed(name, action, context=self)

def is_allowed(name, action, context=None): name: string - a username or IP for 'visitor'

action: string - 'read', 'edit', 'delete', 'edit-permissions' context: object - a Group or a Package or None (which means system)

Based on a section of AccessControl design: WUI gives controls to user and administrator to change permissions on a package.

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results.

This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents.

For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCIITRANSLIT).

Add Group editing page.

If no permissions to change group can't edit group. Also cannot view edit page.

Editable attributes: name, title, description

No preview needed

see ​https://github.com/okfn/ckan/pull/5

Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?).

You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q.

For example:

http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93

This "N-dash" (Unicode character 2013) causes this exception:

Module ckan.controllers.api:460 in search
<<                      if ver in u'12':
                               # Otherwise, put all unrecognised ones into the q parameter
                               params = convert_legacy_parameters_to_solr(params)
                           query = query_for(model.Package)
                           results = query.run(params)
>>  params = convert_legacy_parameters_to_solr(params)
Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr
<<      for search_key in non_solr_params:
               value_obj = legacy_params[search_key]
               value = str(value_obj).replace('+', ' ')
               if search_key == 'all_fields':
                   if value:
>>  value = str(value_obj).replace('+', ' ')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128)

This problem affects CKAN 1.5 and 1.5.1 only.

Notes on the CKAN API from Aron:

• Server returns text/html for errors even when "Accept: application/json" header is set.
• DELETE package request requires a Content-Length? Should be 405 Method Not Allowed?
• Tag returned as a JSON object when updating but as a string when requesting. Same with "extras" content.
• How to remove key from "extras" object? Passing None as described in the docs is invalid JSON and using null doesn't work.
• Tags return package names rather than ids when querying GET /tag/{id}
• Doesn't return rendered_text property on package update.
• Relationships use "object" key rather than id or package_id.

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on ​Storate Metadata API.

This could be implemented as a separate API call, or the rendered text versions of the activities could be added into the JSON returned by the existing API call.

This requires setting up templates for rendering activity streams items and detail items as nice, human-readable text.

There are some open questions, e.g.: Do we want the entire activity stream rendered as a block of plain text? As HTML? Or do we want a list of JSON objects, where each object contains its textual and/or HTML representations as fields?

Activity stream items and their related detail items are separate objects that each have their own textual representations.

For a mockup of the kind of text messages we want, see:

​http://datahub.pudo.org/pudo

but note that this ticket is just for creating the text snippets themselves, not rendering then in an HTML page or RSS feed. Also the mockup only show activity items and not their detail items.

The create_activity() logic function already exists, but validation and authorization need to be added before it's hooked up to an API call.

As CKAN grows, more and more optional representations of packages will become used. Besides RDF (which is the motivation for this ticket), support for DSPL, home-grown XML, or CSV listings is imaginable.

To properly support this CKAN should have an extensible representation registry that can be extended when new output types become available (and without changing the API in the process). This needs to integrate in two places:

• We need to add support for file format extensions to the package handler and REST API.
• We need to add HTTP content type negotiation (Accept headers).

To support this we should have a registry with two registers:

• to map format extensions to mime types (e.g. "json" -> "application/json")
• to map mime types and entity types to converter functions (e.g. 8"application/json", ckan.model.Package) -> func(obj, mime_type) )

This should be set up on load_environment so that IConfigurer plugins can feed into it.

The registry should then first be added to _finish in the REST API (which needs to be refactored to be passed the {format} part of the URL if one is given. Based on the format part and HTTP headers, an appropriate representation can be generated by the registry and then be returned to the user.

Forwarding of requests to the regular WUI controllers with Accept headers set or a format specified can be implemented in a separate effort.

When POSTing to the API, if specified, the 'Content-Type' header must be blank or 'application/x-www-form-urlencoded'. Otherwise we get an error like: "Bad request - JSON Error: Could not extract request body data: Bad content type: \'; charset=utf-8\'""

The problem is that this is a very reasonable header to send. Indeed requests 0.14 sends this particular header.

This affects all versions of CKAN.

This is due to webob/requests.py:1248 being pretty basic.

Change the visual style of CKAN to be more like these sites:

• GitHub?
• Quora
• Google Projects

This does not include major UX work.

You get redirected for a number of different reasons - often confuses me let alone the average user!

Cost: 1h

Tests marked decorated "@search_related" should only be run against postgresql, but in fact they don't get run at all.

I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response:

"publish: New Zealand Coastline (new-zealand-coastline)" opening connection to nz.ckan.net... opened <- "POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n" <- "{\"name\":\"new-zealand-coastline\",\"title\":\"New Zealand Coastline\"}" -> "HTTP/1.1 500 Internal Server Error\r\n" -> "Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n" -> "Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n" -> "Vary: Accept-Encoding\r\n" -> "Content-Type: text/html; charset=iso-8859-1\r\n" -> "Cache-Control: proxy-revalidate\r\n" -> "Content-Length: 640\r\n" -> "Connection: close\r\n" -> "\r\n" reading 640 bytes... -> "<!DOCTYPE HTML PUBLIC \"-IETFDTD HTML 2.0EN\">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n<hr>\n<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80</address>\n</body></html>\n" read 640 bytes Conn close

The server should not respond with a 500 error if the package was created.

Regression found by updating datacatalogs.