CKAN: Ticket Query

#1198: Publisher hierarchy

dread — Thu, 23 Jun 2011 09:16:32 GMT

'Publisher' entities in the model. They are hierarchical.

'User-Publisher' connections with one or more roles (e.g. drafter, moderator).

Authorization settings can control who can set what values in a 'published by' type field.

Publishers and User-Publishers available to read in the API.

Future tickets will provide:

API to write Publishers and User-Publishers
UI to edit Publishers and User-Publishers

(This feature deprecates authorization groups)

#1287: NAVL validation errors - Junk fields should be listed explicitly

dread — Wed, 24 Aug 2011 16:25:02 GMT

When you create a package, but specify a key that is not allowed (e.g. 'relationships') then you get error message:

{"__junk": ["The input field __junk was not expected."]}

It should mention the actual key which is not expected. e.g.

{"relationships": ["The input field 'relationships' was not expected."]}

Kindly said that James' version of NAVL was better in this respect, so this might be best solved by moving to that.

#1322: Action API improvements

dread — Thu, 08 Sep 2011 09:39:09 GMT

Focusing on improving Action API as the v3 API:

have an optional parameter of the data_dict called "options". Options would contain items that would get passed into the context. e.g. {"options" : {"ref_package_by": "id"}}.
instead of using API version to change the way packages are referenced, use the ref_package_by.
All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'.
Action API is v3 of api, replacement for v1 & v2. Default for most urls is still v1, but if url is /api/action then default to v3.

Next steps:

Add search API (package, resource,
Add Util API
Clarify JSONP still applies
Add doc strings, clarifying parameters

#1438: Action API - parameter discovery/checking

dread — Tue, 01 Nov 2011 15:38:10 GMT

Many actions in the Action API require parameters. What params are needed should be listed and checked. Because currently, if you get them wrong you simply get a useless 500 error.

Currently they are listed in the docs, extracted from the code manually.

So you could GET /action/api/package_list to receive not only the help text, but a list of arguments.

And if you send an extra or missing argument then an intelligent error message can be returned.

implementation

How about some sort of decorator on the action function:

@logic_params(id, offset, limit)
def get_package_list(context, data_dict):
    ...

This would do the param checking, and is there a way to extract these params from the function? Or do a registration of the logic function?

I'd certainly like to keep the list of the list of params for the function with the function, for ease of reading the code.

Another good thing would be to pass in the params named as themselves, rather than having them contained in the data_dict.

#1439: Action API discoverablility

dread — Tue, 01 Nov 2011 15:39:30 GMT

A good service API needs to be discoverable, so you are not always having to refer to the documentation html.

Maybe /api/action should return a list of actions available? (Currently this returns a 404.)

It would be nice to sort these into get/create/update/delete.
#1438 Parameters for each of the actions must be discoverable too

/api/action/{action_name} should also return the help text / parameters allowable. (Currently this returns 400 error)

#1457: Bug with DataNL instance

jilly.mathews — Thu, 10 Nov 2011 11:37:21 GMT

"when logging into http://register.data.overheid.nl/ with OpenID, the /user/me page gives a 404. CKAN version 1.3.4."

n the manual it says an API key kan be created via http://test.ckan.net/user/me /. However, when I try the corresponding http://register.data.overheid.nl/user/me, I get a 404 error (not found).

#1679: Default roles problem

dread — Tue, 17 Jan 2012 17:57:22 GMT

The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py).

The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different.

We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?

#1787: [super] Improve RESTful API

dread — Wed, 08 Feb 2012 11:54:42 GMT

Lists of entities should be full URLs, rather than just the names
Discoverability - /api/v3/rest should list the entity types that can be listed

This could be v3 of the RESTful interface.

#2412: More than one resource invalidatiing breaks dataset edit form

icmurray — Tue, 22 May 2012 20:05:48 GMT

When attempting to add more than one resource at once, if more than one resource invalidates, this results in a js error, leaving the form in an inconsistent state.

Repro:

Go to /dataset/new
Add a new resource. Fill in one of the fields with an invalid value. eg - last_modified, or size...
Add another resource, doing the same thing: make one of the fields invalid.
Try to save the dataset.
The entered resource information will be lost, and a js error "Uncaught Error: Can't add the same model to a set twice,: backbone.js:586" will be thrown.

#2485: Encourage leak containment by limiting the number of requests a CKAN process serves

nils.toedtmann — Fri, 01 Jun 2012 11:47:09 GMT

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

research ways to limit requests-per-process with the different web servers (see below for a start);
explain these safeguards in the CKAN documentation and encourage users to apply them;
consider the helper script ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of commented out. Or at least let it warn the user to use MaxRequestsPerChild;
Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

Apache:
- Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on datahub.io and the ckan farm
- (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
nginx/supervisord: to be researched.

#2686: enabling datastore & data API breaks recline

shevski — Tue, 17 Jul 2012 09:06:51 GMT

First I noticed that the gold prices dataset preview was not displaying & has data API enabled Secondly I tried enabling datastore for http://datahub.io/dataset/adur_district_spending/resource/281dffa6-ea9b-4446-be41-05dced06591f and after I saved the preview no longer worked. Unticking the datastore & data api checkbox brought it back

Is this a known issue?

#2788: Speed improvements on creating/updating and indexing

amercader — Wed, 01 Aug 2012 14:56:32 GMT

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

#2838: Context variables accepted by action functions need to be documented

seanh — Fri, 10 Aug 2012 20:35:11 GMT

I was doing this:

context = {'model': base.model, 'session': base.model.Session,
                    'user': toolkit.c.user or toolkit.c.author,
                    'extras_as_string': True}
group_dict = logic.get_action('group_show')(context,
                    {'id': group_id})

in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass 'extras_as_string': True in the context. I don't think this or other context variables are documented anywhere.

#3023: New methods on IPackageController to provide access to the data_dict

amercader — Thu, 22 Nov 2012 17:00:57 GMT

Extension hooking into the edit and create methods of the IPackageController interface receive the package object. This may not include all the fields that came from the form. The new extension points will pass the validated data_dict so extensions can have access to it