CKAN: Ticket Query

#2499: Documentation for the internal analytics

icmurray — Thu, 07 Jun 2012 11:58:17 GMT

There's no docs for setting up the internal analytics, nor what it provides.

Setup

Add: ckan.tracking_enabled = true to .ini file

Run: paster tracking -c dev.ini

#2761: Document all the errors you can get when setting up filestore, and how to fix them

seanh — Thu, 26 Jul 2012 11:36:06 GMT

Add it to a 'Troubleshooting' section on the filestore page: http://docs.ckan.org/en/ckan-1.7.1/filestore.html

For the error messages and their solutions, see various threads on ckan-dev

#2762: test_related.py crashes

seanh — Thu, 26 Jul 2012 11:55:53 GMT

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

#2771: Documentation and examples for IDatasetForm and IGroupForm

seanh — Fri, 27 Jul 2012 12:30:38 GMT

Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests.

The IDatasetForm example should use tag vocabularies (two birds with one stone)

The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples.

Tab Vocabularies docs should reference IDatasetForm example.

When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe)

There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date.

#2919: Remove IController from docs

seanh — Wed, 12 Sep 2012 14:47:33 GMT

Still in docs but seems gone from code

#2936: Updating a group via the API clears its packages

seanh — Wed, 19 Sep 2012 09:47:46 GMT

If the group dict that you post to the API does not have any 'packages' key then all the group's packages get removed. I think it would be better if you could just update e.g. the group's description without having to also post the list of packages, and apparently this is how other update API actions work.

Might be worth checking all the update API actions for this behaviour, and making sure they're all consistent

#2958: Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8

seanh — Thu, 04 Oct 2012 09:29:36 GMT

e.g. 2012_08(주요국가).xls

I tested in CKAN 2.0 and it seemed to work fine there.

#2965: Stats extension broken on 1.8

amercader — Mon, 15 Oct 2012 10:15:41 GMT

Graphs not showing (looks like a flot related file is missing)
Wrong groups counts (e.g. Data Explorer Examples show 1800 datasets when it onlu has 8)

#3003: New pg databases should be created with UTF8 encoding rather than system default

amercader — Fri, 19 Oct 2012 09:33:19 GMT

Just to track @tauberer patch on Github. Probably going to 1.8.1

#2537: Test and document ckanbuild

seanh — Fri, 15 Jun 2012 15:48:48 GMT

https://github.com/okfn/ckanbuild

Verify that what's there so far still works, write a README explaining how it works

#2538: Add multiple-instance support to ckanbuild

seanh — Fri, 15 Jun 2012 15:51:39 GMT

Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command.

May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command.

#2539: Investigate the existing ckan debian package for ckanbuild

seanh — Fri, 15 Jun 2012 15:54:21 GMT

Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh?

#2540: Implement a way of upgrading ckan sites using ckanbuild

seanh — Fri, 15 Jun 2012 15:55:40 GMT

When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once.

#2541: Add non-core extensions to ckanbuild

seanh — Fri, 15 Jun 2012 15:57:06 GMT

We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include.

#2542: Create jenkins job to run ckanbuild, and run tests

seanh — Fri, 15 Jun 2012 15:58:08 GMT

It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests.

#2548: Object ownership for groups/package

ross — Mon, 18 Jun 2012 11:02:11 GMT

Requirements

We need to be able to easily determine who the owner of a dataset or group is. Datasets and Groups should have an Owner, who may change over time but is a specific user within the CKAN instance. It should be easy for CKAN components to determine the user and for the initial version we should ignore the can of worms labelled 'ownership transfer'.

At this point migration is likely to be the biggest issue, and would suggest that it is acceptable that the last user to edit a dataset be set as the current owner.

More tickets should arise as a result of this work where we may be able to optimise some queries to use the new feature.

Interface

None

User Stories

None

Tasks

[ ] Analysis/Clarification?

[ ] Tests

[ ] Migration

[ ] Code/Schema? changes

[ ] Documentation

Estimates

#2547: ODS Initial data sets

ross — Mon, 18 Jun 2012 10:55:06 GMT

Requirements

The ODS demo site will need data adding, initially as fixtures but it would also be useful if we started evaluating datasets that we can ship with ODS installations (at least in the UK) from places such as DGU and ONS.

May wish to create a ticket for making sure the datasets within the system are reset every X hours. Perhaps.

Interface

None

User Stories

As a new system administrator for an ODS instance, I don't want to have a site devoid of any data. Geographically relevant datasets would be welcomed.

As a bizdev person I would like to be able to demonstrate how ODS works with real datasets.

Tasks

[ ] Identify relevant sources for datasets

[ ] Pick datasets

[ ] Set them up for import

Estimates

#235: Resource format normalization and detection

dread — Mon, 18 Jan 2010 15:13:24 GMT

Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service.

Sub-tasks:

Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -> "text/csv" and "Name mappings", i.e. "text/csv" -> "Comma-Separated Spreadsheet".
Collect and map surface forms from all CKANs
Access this via Swiss and apply, store as a PackageResource? extra field pending #826 (Resource extras).
Add heuristics for format auto-detections:
- Map well-known file extensions
- Recognize obvious magic (Zip, Tar)
- Peek into Zipfile/Tarfiles?
Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar)
See also: #816 (Autocomplete for the resource format field)

#250: RDF link in Atom feed

dread — Thu, 18 Feb 2010 15:41:35 GMT

Add link to RDF representation of a package in our Atom feed.

#1745: Dataset search UX improvements as of Jan 2012

rgrp — Thu, 02 Feb 2012 23:22:21 GMT

Changes to make search both more exploratory and more satisfying to use

Search query build - #1603
- Ability to add new facet fields "live"
- That is add fields which then contain faceted options (a bit like data.hri.fi)
(??) Autocomplete / drop down on search (i.e. search while you type)
- Dubious about value / cost ratio here

Probably would involve to pure JS and HTML implementation.

Implementation

Probably require

API changes to expose solr style API directly #1737

#1785: Replace 'Revisions' page with site-wide activity stream

seanh — Tue, 07 Feb 2012 16:34:18 GMT

Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead?

#2257: cleanup template vars

toby — Fri, 23 Mar 2012 14:35:07 GMT

look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc

#2291: DataStorer incorrectly attempts to store JSON in DataStore

rgrp — Sat, 14 Apr 2012 19:19:07 GMT

See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

#2320: Introduce new state: archived

rgrp — Mon, 23 Apr 2012 20:12:29 GMT

Some datasets become obsolete: e.g. http://datahub.io/dataset/uncdb Do not want to delete them but want them marked as archived or deprecated or something.

#2321: Use resource names/titles in URLs

rgrp — Tue, 24 Apr 2012 03:00:12 GMT

Would make them much nicer but requires:

Ensure uniqueness (or deal with ambiguity)
(?) Slug/names versus titles
- Aside: what is currently called name should probably be title

#2324: Link API docs to relevant version not just latest

amercader — Wed, 25 Apr 2012 10:58:52 GMT

The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version.

Also, remove the 'API' link (to /api/1) as pointless and misleading

#2328: Link to Catalog API for each Dataset, Group and Tag

rgrp — Thu, 26 Apr 2012 12:15:01 GMT

At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page?

#2333: Introduce new resource type datastore

rgrp — Sat, 28 Apr 2012 22:16:00 GMT

This is a resource that is *just* in the datastore. At the moment there are hacky ways to do this (create resource with meaningless URL etc) but we should do this properly:

New Option: DataStore? Table in add a resource form
URL field is disabled (or points at /api/data or something rather pointless)
Download button is disabled
- Better would be to find a way to serialize an ES table or stream it ...

#2334: Improved data import

rgrp — Sat, 28 Apr 2012 22:21:02 GMT

Placeholder for thinking about this ...

Allow users to configure info for CSV import (e.g. separators, quotes etc)

#2335: Unicode characters don't work in CKAN ini files

seanh — Mon, 30 Apr 2012 12:42:06 GMT

You get a UnicodeDecodeError?, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example.

#2337: Sort out ini file confusion in docs

seanh — Mon, 30 Apr 2012 14:57:56 GMT

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

#2370: Add examples of all config settings in default config file

seanh — Thu, 03 May 2012 10:50:37 GMT

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

#2406: Errors in source install doc

markw — Mon, 21 May 2012 10:11:33 GMT

There are a few errors or unclarities in the source install doc - see

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html

#2407: 'Access denied' message is unhelpful

markw — Mon, 21 May 2012 15:01:40 GMT

This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied':

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html

The problem was he didn't have access to the group. A message like 'You don't have access to the group "LODcloud"' would have been clearer.

#2415: Confusing message on attempting to register new account with upper case

markw — Wed, 23 May 2012 13:19:49 GMT

On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

"The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.

#2426: Get rid of CKAN's formalchemy dependency

seanh — Fri, 25 May 2012 11:47:42 GMT

#2427: Reduce cruft installed into CKAN's virtualenv due to Pylons

seanh — Fri, 25 May 2012 11:48:27 GMT

#2473: Make datstorer store field ordering in _meta field

kindly — Tue, 29 May 2012 12:11:32 GMT

Recline views should have a default table order, being the same as the csv that was imported.

#2475: Upgrade trac.ckan.org to trac 0.12.3

seanh — Tue, 29 May 2012 18:46:16 GMT

This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

http://trac.edgewall.org/wiki/ChangeLog

#2477: Add extensions section to readthedocs

shevski — Wed, 30 May 2012 15:54:04 GMT

Need a central up to date place for currently supported extensions (closest we have is http://wiki.ckan.org/List_of_Extensions) & their documentation. I suggest this gets added to a dedicated section of docs.ckan.org

#2478: Get NeverNotifyUpdaterPlugin installed on trac.ckan.org

seanh — Wed, 30 May 2012 16:52:14 GMT

#2479: Meanings of Author and Maintainer fields are unclear

markw — Thu, 31 May 2012 08:28:18 GMT

CKAN's default schema has fields for Author ('The name of the main contact, for enquiries about this particular dataset') and 'Maintainer ('another important contact person'). The distinction is not clear. Also the fields are often left blank.

The roles that seems most important are those of the original owner/publisher of the data, and the person maintaining the CKAN record/copy of it. So I suggest:

(1) Rename the fields 'Owner' and 'Maintainer'.

(2) Change the explanatory text for the relevant fields:

Owner: 'The person or organisation who create/collect/publish the data in this dataset.' Owner e-mail: 'E-mail address for enquiries to the Owner named above.' Maintainer: 'The person maintaining this dataset on [name of CKAN instance], if different from the above.' Maintainer e-mail: 'E-mail address for enquiries to the Maintainer named above.'

(3) When a logged-in user creates a new dataset, the main form should have a checkbox, checked by default, marked 'I am the maintainer of this dataset'. If checked, the Maintainer name and e-mail fields are populated from the user's profile.

#2480: Better message when dataset has no resources

markw — Thu, 31 May 2012 08:31:33 GMT

If a dataset has no resources the resources list currently says '(none)'.

Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'

#2483: Non-local resources should not have Download links

markw — Thu, 31 May 2012 12:52:14 GMT

At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally.

This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table.

Suggested fix:

Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource)
Disable the Download button unless the resource is stored locally.

#2485: Encourage leak containment by limiting the number of requests a CKAN process serves

nils.toedtmann — Fri, 01 Jun 2012 11:47:09 GMT

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

research ways to limit requests-per-process with the different web servers (see below for a start);
explain these safeguards in the CKAN documentation and encourage users to apply them;
consider the helper script ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of commented out. Or at least let it warn the user to use MaxRequestsPerChild;
Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

Apache:
- Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on datahub.io and the ckan farm
- (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
nginx/supervisord: to be researched.

#2516: Make 'Assign to:' field on trac.ckan.org into a dropdown list

seanh — Wed, 13 Jun 2012 09:07:22 GMT

there's a setting for this

#2520: Document undocumented config options

seanh — Wed, 13 Jun 2012 11:52:48 GMT

There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either:

ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled

There are also some options that are in the default deployment.ini even though they're deprecated:

ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously

See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html

It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358

#2529: DataHub (or CKAN) widgets

rgrp — Fri, 15 Jun 2012 08:47:29 GMT

Simple widgets in pure JS. For example:

Count of datasets in a group (could generalise to a query but not sure how useful that is ...)
Embeddable list of top X (5) datasets for a given query
Embeddable list of *my* datasets

Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query).

Or: we have this attached to areas of site where relevant.

Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page).

Cf. http://okfnlabs.org/ckanjs/

#2543: facet.sort is not available in the package_search action

icmurray — Mon, 18 Jun 2012 09:55:35 GMT

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

#2572: clean up stats plugin

toby — Tue, 19 Jun 2012 11:24:14 GMT

attempt to disengage the stats plugin from core as much as possible

#2579: move sort_by functions into lib.helpers

toby — Thu, 21 Jun 2012 12:29:58 GMT

make these more available but keep existing functionality so not to break any users

remove_field()

drill_down_url()

etc

#2585: Escape solr control characters in search queries, add advanced search screen

seanh — Fri, 22 Jun 2012 10:26:13 GMT

Suggestion from David Read:

We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for "Spend over £25,000 - NHS Leeds" then it would not come up with the dataset with that exact name. It was excluding datasets with the word "NHS" due to the dash/minus sign. It works fine if you escape the minus sign: "Spend over £25,000 \- NHS Leeds".

So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine:

http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/

Perhaps you would consider providing this in CKAN core in future?

I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?

#2619: Omit private datasets from public activity streams

seanh — Thu, 28 Jun 2012 11:49:00 GMT

Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered.

#2625: Add i18n strings from non-core but supported extensions to ckan.pot file

seanh — Mon, 02 Jul 2012 13:48:23 GMT

Have to decide which non-core extensions are going to be supported first.

#2663: h.resource_display_name needs love

toby — Wed, 11 Jul 2012 14:42:26 GMT

This function is shit and needs cleaning up and a doc string

description is markdown and should be treated properly

either we should truncate all or leave it to the templates but work universally

url if no name / desc this is in demo-theme branch

#2708: limit extra data for package/group show

toby — Wed, 18 Jul 2012 14:40:20 GMT

contextpackage_limits? = { 'tags': 5, <- get first 5

'extras': 0, <- get all

}

only get what you ask for have to be explicit

contextgroup_limits? = {} only main item

start with datasets/groups expand if we like it

#2758: file storage gives error if config not available but no useful user information

toby — Thu, 26 Jul 2012 07:28:29 GMT

We get an error which should be improved the actual problem is this but not passed to user

⇝ KeyError?: 'ofs.impl'

View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view

data = func(*args, kwargs)

Module ckan.controllers.storage:407 in auth_form view

authorize(method, bucket, label, c.userobj, self.ofs)

Module ckan.controllers.storage:200 in ofs view

StorageAPIController._ofs_impl = get_ofs()

Module ckan.controllers.storage:71 in get_ofs view

storage_backend = configofs.impl?

Module paste.registry:146 in getitem view

return self._current_obj()[key]

KeyError?: 'ofs.impl'

#2773: About page needs improving

markw — Mon, 30 Jul 2012 11:13:15 GMT

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

#2788: Speed improvements on creating/updating and indexing

amercader — Wed, 01 Aug 2012 14:56:32 GMT

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

#2829: Archiver fails on 403 http response

toby — Thu, 09 Aug 2012 10:34:44 GMT

Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end

$ paster archiver update -c ../ckan/development.ini
2012-08-09 11:01:37,636 INFO  [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources)
2012-08-09 11:01:37,671 INFO  [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13
2012-08-09 11:01:37,900 INFO  [ckan.lib.base]  /api/action/package_show render time 0.043 seconds
Traceback (most recent call last):
  File "/home/toby/okfn/pyenv/bin/paster", line 8, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py", line 98, in command
    response = app.post(api_url + '/package_show', data)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 262, in post
    expect_errors=expect_errors)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 243, in _gen_request
    return self.do_request(req, status=status)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 406, in do_request
    self._check_status(status, res)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 439, in _check_status
    res.body))
paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show)
{"help": "Return the metadata of a dataset (package) and its resources.\n\n    :param id: the id or name of the dataset\n    :type id: string\n\n    :rtype: dictionary\n\n    ", "success": false, "error": {"message": "Access denied", "__type": "Authorization Error"}}

#2838: Context variables accepted by action functions need to be documented

seanh — Fri, 10 Aug 2012 20:35:11 GMT

I was doing this:

context = {'model': base.model, 'session': base.model.Session,
                    'user': toolkit.c.user or toolkit.c.author,
                    'extras_as_string': True}
group_dict = logic.get_action('group_show')(context,
                    {'id': group_id})

in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass 'extras_as_string': True in the context. I don't think this or other context variables are documented anywhere.

#2885: Labels (editor, admin) not translated in authorization pages for datasets or groups

seanh — Wed, 22 Aug 2012 08:49:11 GMT

#2889: Support JSON values in extras, when returning JSON from API

seanh — Fri, 24 Aug 2012 12:35:37 GMT

For example, edit the warandpeace dataset and add an extra with key "comments" and value "[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]", now visit API URLs such as:

/api/search/dataset?sort=metadata_modified+desc&all_fields=1&limit=10

/api/rest/dataset/warandpeace

/api/action/package_show?id=warandpeace

The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects.

#2329: Add back in RSS/Atom links on relevant pages

rgrp — Thu, 26 Apr 2012 12:15:40 GMT

E.g. on dataset page, on revision page, on user page and on search results.

#2654: UI support for ordering groups on group_read page

ross — Tue, 10 Jul 2012 18:11:31 GMT

The group_index page has no support in WUI for ordering the groups displayed. Should allow sorting by name

Add support for this for datahub now, and discuss for new 1.9 UI

#2886: Configurable related items

ross — Wed, 22 Aug 2012 08:55:18 GMT

Related items (Apps & Ideas) have a collection of types, which are currently fixed. Some of these types are ambiguous, such as ideas in that they may not have a link - and if they do it is likely to be to a blog post - another existing type.

Whilst the URL is required, and we think this should stay required, we should also allow users to change the types found in Apps & Ideas to a shortened list that suits their requirements.

#2902: genshi is used in Group controller

ross — Tue, 04 Sep 2012 18:10:30 GMT

The Group controller uses the genshi module where it may be better for it to use Markup() in the template.

See https://github.com/okfn/ckan/pull/117/files#r1524863

#2913: IGroupForm has package_form() method

seanh — Tue, 11 Sep 2012 14:45:39 GMT

Should be group_form() I think

#2915: Refactor form_to_db_schema_options()

seanh — Wed, 12 Sep 2012 10:37:49 GMT

Having two methods form_to_db_schema(self) and form_to_db_schema_options(self, options) seems unnecessary, why not just have form_to_db_schema(self, options=None)?

Fixing this might break existing extensions although I don't think any are using form_to_db_schema_options() so it should be okay.

The same thing goes for db_to_form_schema_options().

Also why are we passing a dict 'options' that always contains the same three keys 'api', 'context' and 'type'? Why not three params api=None, contenxt=None, type=None?

#2917: Organization admins can delete themselves

ross — Wed, 12 Sep 2012 12:10:43 GMT

Organization administrators can delete themselves from the user management pages. We should disable this on the front-end as well as the back-end.

#2920: Genshi text template message extractor no longer needed?

seanh — Thu, 13 Sep 2012 09:57:29 GMT

In setup.py, do we still need:

('templates/.txt', 'genshi', {

'template_class': 'genshi.template:TextTemplate'

}),

in message_extractors? Doesn't like there are any txt files in there anymore.

#2921: Add docstring to top of lib/extract.py file

seanh — Thu, 13 Sep 2012 10:00:07 GMT

I think it couldn't hurt for this module to have a docstring at the top of the file explaining what the module is for. I know from setup.py that it's there to provide the extract_ckan() string extractor function for Babel, but I think looking at the file on its own it's not so obvious.

Also a couple of other small fixes:

jinja2_cleaner looks like a helper function only meant to be used by extract_ckan(), might make things clearer if it was called _jinja2_cleaner(), just so it doesn't look like a public function the module wants to export.

jinja_extensions seems to be a module-level variable that is only used in one function, could be moved into the function, unless you think there might be more functions that want it in future.

#2922: Better docstring for CKANInternationalizationExtension

seanh — Thu, 13 Sep 2012 10:01:47 GMT

I'm unsure about what's going on here. As I understand it, when we run python setup.py extract_messages it's going to use the extract_ckan() function from ckan/lib/extract.py to process the HTML files. For the HTML files that are jinja2 templates, extract_ckan() will call jinja2_cleaner() which will call regularise_html() on the strings. So the strings are regularised when they are extracted from the source files.

But then we have the parse() method of CkanInternationalizationExtension? also calling regularise_html(). I don't get what's happening here. Why do the strings need to be regularised twice?

My guess is that CkanInternationalizationExtension? is used when the strings are extracted from the templates at runtime, and they need to be regularised at this time in order to match them against the regularised strings in the mo files to find the translations to output?

Maybe CkanInternationalizationExtension? needs a better docstring saying what it does?

#2923: Change regularise -> regularize

seanh — Thu, 13 Sep 2012 10:03:02 GMT

The function is called regularise_html(), can't remember what file it's in.

#2924: Better docs for trans js command, and add to release process

seanh — Thu, 13 Sep 2012 10:04:31 GMT

Add a better docstring to for trans js command explaining what it does and why, and how to use it.

Also add this command to the CKAN Release Process as it's needed there

#2925: Remove trans mangle paster command?

seanh — Thu, 13 Sep 2012 10:06:08 GMT

Is trans mangle really necessary? If you upload a pot file to Transifex, it can generate a po file for you with 100% strings translated into a fictional pseudo language where everything is really long strings of unicode characters. I found this worked well for coverage testing, and also tests handling of unicode and long strings all over the place.

#2926: I*Form tests for per-type templates

seanh — Thu, 13 Sep 2012 16:01:21 GMT

Finish the tests for the per-package-type custom template support in IPackageForm, IGroupForm and IOrganizationForm (i.e. the new_template(), read_template() etc. methods).

The new way that template inheritance works in jinja looks like it should allow this to be tested nicely without duplicating entire templates in the example extensions.

#2927: Test new package/group/organization type URLs support

seanh — Thu, 13 Sep 2012 16:02:33 GMT

IPackageForm, IGroupForm and IOrganizationForm allow you to define e.g. new package types with new top-level URLs, but the example extensions don't have tests for this yet.

#2928: Run CKAN tests with example_i*form extensions enabled

seanh — Thu, 13 Sep 2012 16:05:34 GMT

Before releasing CKAN 2.0 we need to run all the CKAN tests with a modified test-core.ini with the example_idatasetform, example_igroupform and example_iorganizationform plugins enabled. If any tests fail, fix the bugs. This needs to be done for each release so add it to the release process.

#2929: Remove is_fallback() from I*Form extension interfaces?

seanh — Thu, 13 Sep 2012 16:09:57 GMT

If a plugin wants to take over the default package type for example, it can return dataset? from its package_types() method. I don't see how there can be a package type that is not the default 'package' and is not returned by the package_types() method of a plugin, because the URLs for such a package type would not be mapped. Am I missing something or is is_fallback() not needed?

Same for IGroupForm and IOrganizationForm

#2931: Better docstring for app_globals.py

seanh — Mon, 17 Sep 2012 14:13:14 GMT

The application's Globals object is not very informative.

#2932: Add docstring to system_info.py

seanh — Mon, 17 Sep 2012 14:50:29 GMT

What is the system_info table for? It looks like a way to override config file settings in the database, perhaps so that admins can use a web interface to change site title etc. Could do with a docstring explaining the intended purpose of the table.

#2933: Update theming docs

seanh — Mon, 17 Sep 2012 15:21:12 GMT

There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs.

I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents.

#2956: Allow for resource editing in CKAN 2.0

johnmartin — Wed, 03 Oct 2012 15:11:40 GMT

On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually.

#2961: Preview plugin endpoint

dominik — Mon, 08 Oct 2012 11:39:27 GMT

Users could write ckan extensions that offer previews for a certain datatype.

Advantages:

We can say that you can add your own previews
Every preview would have to define it's own dependencies
External development (community creates new extensions?)

Questions to answer:

Specification of the interface
Precedence of previews for the same data types
Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible.

As far as I can see, this should be fairly easy to implement by following these instructions: http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface

#2964: Last organization admin can remove herself

seanh — Mon, 15 Oct 2012 10:14:18 GMT

If you are the only admin of an organization you can edit the organization's members and demote yourself, then the org has no admins and no one (except sysadmins maybe) can edit it.

Last admin should not be able to remove or demote herself.

Also applies to groups.

#2966: 'Add' button text is wrong when editing organization members

seanh — Mon, 15 Oct 2012 10:15:49 GMT

e.g. if I just changed a member's capacity I am not adding anything 'save' is better maybe

#2967: Organization members edit page reloads after demoting self

seanh — Mon, 15 Oct 2012 10:17:07 GMT

Edit an organizations members page and demote yourself from organization admin, after saving the members edit page reloads even though you are no longer an admin and should no longer be able to access this page.

#2968: Anyone can access organization members page

seanh — Mon, 15 Oct 2012 10:19:15 GMT

The button will not show if you are not authorized but browse to /organization/members/foo and you can edit the members, it does stop you when you try to save your changes, but you shouldn't be able to get to the page at all

#2969: Group members page 500s

seanh — Mon, 15 Oct 2012 10:20:54 GMT

The group members page (e.g. /group/members/roger) seems broken, IDs instead of user names are shown for the members, and clicking on a member 500s

#2970: Organization and group member links use id not name

seanh — Mon, 15 Oct 2012 10:22:11 GMT

e.g. it the 'Members' button links to /organization/members/0a44... instead of /organization/members/foobar

#2971: "Are ytou sure you want to delete this member?" should say which member

seanh — Mon, 15 Oct 2012 10:27:20 GMT

when deleting members from groups and orgs

#2972: Remove any imports of authz.py and delete file

seanh — Mon, 15 Oct 2012 10:37:22 GMT

it is no longer used

#2973: Move new_authz.py into logic/auth/init.py

seanh — Mon, 15 Oct 2012 10:39:05 GMT

Makes sense to keep all auth stuff under logic/auth.

Also decide which functions from new_authz.pu should be marked private with _.

Also decide which functions from new_authz.py are really helpers for the auth functions in get.py, update.py and delete.py, and should maybe moved to another file, e.g. logic/auth/helpers.py

#2974: General of all auth functions

seanh — Mon, 15 Oct 2012 10:40:12 GMT

From Toby:

A general cleanup of all auth functions (in logic/auth) to check their fitness, error messages, etc.

#2975: Tests for auth functions and new actions

seanh — Mon, 15 Oct 2012 10:41:00 GMT

From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions

#2976: Polish group and organization member pages

seanh — Mon, 15 Oct 2012 10:41:49 GMT

From Toby: A little polish to the member pages and probably a little hardening of the controller and logic actions

#2977: Fix user autocomplete on group and organization member pages

seanh — Mon, 15 Oct 2012 10:42:29 GMT

From Toby: @johnmartin the user autocomplete on member add needs fixing it gets data but does not understand what to do with it

#2978: Tests for permissions for organizations and groups

seanh — Mon, 15 Oct 2012 10:43:26 GMT

From Toby: checking the logic around the who can do what from the user stories - via some tests may be a good approach

#2979: Requesting membership to groups and organizations

seanh — Mon, 15 Oct 2012 10:44:04 GMT

From Toby: The requesting membership user stories and implementation remain needed

CKAN: Ticket Query

#2499: Documentation for the internal analytics

Setup

Related links

#2761: Document all the errors you can get when setting up filestore, and how to fix them

#2762: test_related.py crashes

#2771: Documentation and examples for IDatasetForm and IGroupForm

#2919: Remove IController from docs

#2936: Updating a group via the API clears its packages

#2958: Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8

#2965: Stats extension broken on 1.8

#3003: New pg databases should be created with UTF8 encoding rather than system default

#2537: Test and document ckanbuild

#2538: Add multiple-instance support to ckanbuild

#2539: Investigate the existing ckan debian package for ckanbuild

#2540: Implement a way of upgrading ckan sites using ckanbuild

#2541: Add non-core extensions to ckanbuild

#2542: Create jenkins job to run ckanbuild, and run tests

#2548: Object ownership for groups/package

Requirements

Interface

User Stories

Tasks

Estimates

#2547: ODS Initial data sets

Requirements

Interface

User Stories

Tasks

Estimates

#235: Resource format normalization and detection

#250: RDF link in Atom feed

#1745: Dataset search UX improvements as of Jan 2012

Implementation

#1785: Replace 'Revisions' page with site-wide activity stream

#2257: cleanup template vars

#2291: DataStorer incorrectly attempts to store JSON in DataStore

#2320: Introduce new state: archived

#2321: Use resource names/titles in URLs

#2324: Link API docs to relevant version not just latest

#2328: Link to Catalog API for each Dataset, Group and Tag

#2333: Introduce new resource type datastore

#2334: Improved data import

#2335: Unicode characters don't work in CKAN ini files

#2337: Sort out ini file confusion in docs

#2370: Add examples of all config settings in default config file

#2406: Errors in source install doc

#2407: 'Access denied' message is unhelpful

#2415: Confusing message on attempting to register new account with upper case

#2426: Get rid of CKAN's formalchemy dependency

#2427: Reduce cruft installed into CKAN's virtualenv due to Pylons

#2473: Make datstorer store field ordering in _meta field

#2475: Upgrade trac.ckan.org to trac 0.12.3

#2477: Add extensions section to readthedocs

#2478: Get NeverNotifyUpdaterPlugin installed on trac.ckan.org

#2479: Meanings of Author and Maintainer fields are unclear

#2480: Better message when dataset has no resources

#2483: Non-local resources should not have Download links

#2485: Encourage leak containment by limiting the number of requests a CKAN process serves

#2516: Make 'Assign to:' field on trac.ckan.org into a dropdown list

#2520: Document undocumented config options

#2529: DataHub (or CKAN) widgets

#2543: facet.sort is not available in the package_search action

#2572: clean up stats plugin

#2579: move sort_by functions into lib.helpers

#2585: Escape solr control characters in search queries, add advanced search screen

#2619: Omit private datasets from public activity streams

#2625: Add i18n strings from non-core but supported extensions to ckan.pot file

#2663: h.resource_display_name needs love

#2708: limit extra data for package/group show

#2758: file storage gives error if config not available but no useful user information

#2773: About page needs improving

#2788: Speed improvements on creating/updating and indexing

#2829: Archiver fails on 403 http response

#2838: Context variables accepted by action functions need to be documented

#2885: Labels (editor, admin) not translated in authorization pages for datasets or groups

#2889: Support JSON values in extras, when returning JSON from API

#2329: Add back in RSS/Atom links on relevant pages

#2654: UI support for ordering groups on group_read page

#2886: Configurable related items

#2973: Move new_authz.py into logic/auth/init.py