CKAN: Ticket Query

#2699: workflow for associating datasets with groups

shevski — Wed, 18 Jul 2012 14:02:56 GMT

needs review & speccing out e.g. datasets created by a user who belongs to a certain publisher (group) get auto added to this group

#2780: way for admins to undelete datasets

shevski — Tue, 31 Jul 2012 12:20:48 GMT

Since admins can see deleted datasets - there should be a way for them a) to know they are currently deleted & not viewable by normal users - ticket: #2779 b) way to undelete such datasets - this ticket

I suggest a button on the edit form instead of the delete button i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message?

#2869: user profile not linked on disqus until page refresh

shevski — Thu, 16 Aug 2012 10:58:34 GMT

Reported by Daniel:

Problem: DISQUS comments don't appear with profile details (inc. username). How to reproduce: 1) Go to a dataset, e.g. I went to http://s031.okserver.org:2375/dataset/ff 2) I logged into DISQUS using the Google Authentication, e.g. in my case that is okfn.daniel.lewis@… 3) I made a comment and submitted 4) It comes up with the comment, but no linked profile until the page gets refreshed

#1414: track user log-ins on thedatahub.org

shevski — Mon, 24 Oct 2011 11:13:02 GMT

Set up tracking for user logins so that we have stats about how many active users of thedatahub exist want to be able to see who logged in the the last x months

#2762: test_related.py crashes

seanh — Thu, 26 Jul 2012 11:55:53 GMT

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

#2786: target blank HTML downloads

shevski — Wed, 01 Aug 2012 11:16:54 GMT

e.g. if I click on download here: http://s031.okserver.org:2375/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b can it take me to a new tab instead of navigating away from ckan?

#3027: solr for 2.0

kindly — Tue, 04 Dec 2012 02:22:09 GMT

change mm support solr 3 and 4 add *_date field

#2900: snippets/package_list.html template crashes on dataset that has no 'notes'

seanh — Thu, 30 Aug 2012 14:00:07 GMT

Sometimes a dataset dict comes through package_show() with 'notes': None and then the 'notes' key gets stripped from the dict when validation happens (validation only happens if there's an IDatasetForm plugin with a db_to_form_schema() method) and then package_list.html template crashes, e.g. on user profile pages.

Fix in the template? Or in the schema?

The template also crashes on packages with no tags (empty list gets stripped from dict) and probably other empty fields as well

#1201: seperate out logic in atom feeds to logic layer.

kindly — Fri, 24 Jun 2011 15:21:32 GMT

Simplify the logic in the atom feed an make all feeds use logic layer to return lists.

#2729: searching for tags:[tag] works but tag:[tag] doesn't

shevski — Sun, 22 Jul 2012 09:58:29 GMT

which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work!

http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc

#2896: resource_create crashes on ValidationErrors with KeyError: 'resources'

seanh — Wed, 29 Aug 2012 16:29:14 GMT

ckan/logic/action/create.py:resource_create():

try:

pkg_dict = _get_action('package_update')(context, pkg_dict)

except ValidationError?, e:

errors = e.error_dictresources?[-1] raise ValidationError?(errors)

This assumes that package_update always returns an error dict with 'resources' in it. This assumption doesn't hold.

#1668: repoze version discrepency

dread — Tue, 17 Jan 2012 12:02:26 GMT

There's a discrepency in repoze.who versions between the source and package installs:

repoze.who - package 1.0.18 vs source 1.0.19
repoze.who-friendlyform - package 1.0b3 vs source 1.0.8

We get a test failure [1] with the 1.0b3 version (from the ubuntu 10.04 python-repoze.who-plugins package). But we've not noticed any problems on s057 instances (br, no, ie etc) which have the package versions of repoze.who.

The reason the package install uses the earlier packaged versions rather than the ones we'd like is that repoze uses all sorts of horrendous import hacks, making it too difficult to put into our 'ckan-conflict' source package.

James suggests we 'do something horrible like dynamically patch repoze on CKAN import'.

[1] http://buildbot.okfn.org/builders/builder-ckan/builds/1371/steps/shell/logs/stdio ERROR: ckan.tests.functional.test_user.TestUserController?.test_user_create_unicode

#2573: package_search does not allow solr's per-field facet parameters

icmurray — Tue, 19 Jun 2012 13:27:19 GMT

Solr allows its facet parameters to be specified on a per-field basis, eg. facet.limit applies to all facet fields, but solr allows it to be overriden for a specific field, eg. facet.tags.limit.

We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept. See ckan.lib.search.query.VALID_SOLR_PARAMETERS.

#2768: normalise excel to xls

shevski — Thu, 26 Jul 2012 16:16:22 GMT

so that data proxy works

#2863: non-admin users can edit other's draft datasets

shevski — Thu, 16 Aug 2012 10:54:39 GMT

edit button shows up for daniel lewis for http://s031.okserver.org:2375/dataset/ff

#2854: no way to see which dataset an app is related to on /apps page

shevski — Wed, 15 Aug 2012 16:38:48 GMT

e.g. from http://s031.okserver.org:2375/apps?page=1 you cannot edit a related item as an admin or view what dataset the item is related to

#2857: no UI to follow another user although option on dashboard

shevski — Wed, 15 Aug 2012 16:59:43 GMT

e.g. second option in left hand sidebar is follow another user' http://s031.okserver.org:2375/user/dashboard but if you click on that and then any user, there's no follow button

to me the dashboard user stuff is highly unfinished and probably should not be included unless someone is very keen for it

#2579: move sort_by functions into lib.helpers

toby — Thu, 21 Jun 2012 12:29:58 GMT

make these more available but keep existing functionality so not to break any users

remove_field()

drill_down_url()

etc

#2784: model dictize sensitive data

icmurray — Wed, 01 Aug 2012 09:51:25 GMT

The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not.

#1606: metadata license config option

dread — Mon, 02 Jan 2012 10:45:30 GMT

Add a config option to choose the metadata licence. Set default to Open Database License.

Currently the dataset edit form says "Important: By submitting content, you agree to release your contributions under the Open Database License." This is hard-coded, but not suitable for when DGU uses the CKAN form - they use the OGL.

#2583: make sure that we implement authentication where needed

toby — Fri, 22 Jun 2012 10:04:38 GMT

in development many auth checks may have been lost we need to check they are still working etc

#2282: log api calls for users

toby — Tue, 10 Apr 2012 16:18:45 GMT

This was part of #2251 but is now it's own ticket.

we need clarity on this. The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some? do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me.

#2708: limit extra data for package/group show

toby — Wed, 18 Jul 2012 14:40:20 GMT

contextpackage_limits? = { 'tags': 5, <- get first 5

'extras': 0, <- get all

}

only get what you ask for have to be explicit

contextgroup_limits? = {} only main item

start with datasets/groups expand if we like it

#2810: heroku ckan support

kindly — Tue, 07 Aug 2012 18:40:58 GMT

Get ckan working on heroku

#2663: h.resource_display_name needs love

toby — Wed, 11 Jul 2012 14:42:26 GMT

This function is shit and needs cleaning up and a doc string

description is markdown and should be treated properly

either we should truncate all or leave it to the templates but work universally

url if no name / desc this is in demo-theme branch

#2902: genshi is used in Group controller

ross — Tue, 04 Sep 2012 18:10:30 GMT

The Group controller uses the genshi module where it may be better for it to use Markup() in the template.

See https://github.com/okfn/ckan/pull/117/files#r1524863

#2758: file storage gives error if config not available but no useful user information

toby — Thu, 26 Jul 2012 07:28:29 GMT

We get an error which should be improved the actual problem is this but not passed to user

⇝ KeyError?: 'ofs.impl'

View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view

data = func(*args, kwargs)

Module ckan.controllers.storage:407 in auth_form view

authorize(method, bucket, label, c.userobj, self.ofs)

Module ckan.controllers.storage:200 in ofs view

StorageAPIController._ofs_impl = get_ofs()

Module ckan.controllers.storage:71 in get_ofs view

storage_backend = configofs.impl?

Module paste.registry:146 in getitem view

return self._current_obj()[key]

KeyError?: 'ofs.impl'

#2543: facet.sort is not available in the package_search action

icmurray — Mon, 18 Jun 2012 09:55:35 GMT

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

#2721: deleted groups should not show on 'Add to Groups' dropdown

shevski — Sun, 22 Jul 2012 09:17:12 GMT

Groups previously deleted still show up in the add dataset process in step 3 'Additional info'

http://s031.okserver.org:2375/dataset/new_metadata/ff

#2728: deleted group shows on search index - for admins

shevski — Sun, 22 Jul 2012 09:53:04 GMT

'test-group', which has been deleted,shows up on main search page under groups - and can be filtered by - see http://s031.okserver.org:2375/dataset?groups=test-group

#2815: db_to_form_package_schema() strips tracking summary, isopen

seanh — Wed, 08 Aug 2012 16:47:07 GMT

If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates.

#2897: db_to_form_package_schema() crashes historical dataset page

seanh — Thu, 30 Aug 2012 12:41:35 GMT

If there's an IDatasetForm plugin active with a db_to_form_schema() method based on db_to_form_package_schema(), then viewing historical versions of datasets (e.g. http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191) crashes.

Reason is that revison_id and revision_timestamp need to be added to db_to_form_package_schema(), something like:

schema.update({'revision_id': [validators.ignore_missing, unicode]}) schema.update({'revision_timestamp':

[validators.ignore_missing, unicode]})

#3012: data.gov auth

toby — Thu, 08 Nov 2012 16:50:09 GMT

#2697: create dataset validation

shevski — Wed, 18 Jul 2012 14:00:35 GMT

Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check)

#2726: confusing logic on data preview formats

shevski — Sun, 22 Jul 2012 09:38:47 GMT

If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this

If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28

#3013: common-error-messages is unreadable

dominik — Sat, 10 Nov 2012 13:18:25 GMT

Since the update of the doc theme, the page became unreadable.

http://docs.ckan.org/en/latest/common-error-messages.html

#2257: cleanup template vars

toby — Fri, 23 Mar 2012 14:35:07 GMT

look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc

#2572: clean up stats plugin

toby — Tue, 19 Jun 2012 11:24:14 GMT

attempt to disengage the stats plugin from core as much as possible

#3030: clean up helper functions

toby — Tue, 18 Dec 2012 12:33:58 GMT

#2244: clean up auth

toby — Mon, 19 Mar 2012 17:22:51 GMT

cache admins etc
aim to move check_access so independent of logic function
general spruce up

#2912: ckanext/organizations: Neither public or private initially selected when adding dataset

seanh — Mon, 10 Sep 2012 09:57:56 GMT

When adding a dataset with the organizations extension on, neither the Public or the Private radio button is initially selected. Public should be selected.

#2873: ckanext-qa: dataset summary incorrect

danieljohnlewis — Thu, 16 Aug 2012 14:13:49 GMT

For example see: http://s031.okserver.org:2375/qa/dataset/five_stars

Which has: Example dataset as a score of: 4 After examination of the dataset ( http://s031.okserver.org:2375/dataset/example-dataset ) it is made up of a CSV and an HTML file.

#2718: can't add dataset to more than one group

shevski — Fri, 20 Jul 2012 10:35:50 GMT

trying to add a dataset to another group means it's no longer part of the first group

http://demo.ckan.org/dataset/edit/afterfibre

#2777: bug: user attributes

icmurray — Tue, 31 Jul 2012 09:19:23 GMT

#2748: add 'add new resource' button to sidebar

shevski — Wed, 25 Jul 2012 10:41:18 GMT

When editing a resource you see the current and any other existing resources in sidebar

(e.g. see http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True)

Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page

#1717: [super] Search UX improvements

shevski — Thu, 26 Jan 2012 18:53:01 GMT

Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search)

Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page).

Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it.

More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu

Blue search button should be displayed in line with the search bar, not underneath

Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name

#1578: [super] Re-enable and refactor ratings

rgrp — Mon, 19 Dec 2011 19:20:43 GMT

#1571: [super] Issues Extension

jilly mathews — Mon, 19 Dec 2011 11:30:25 GMT

"Proper" issues extension.

Etherpad: http://ckan.okfnpad.org/issues
Code: https://github.com/okfn/ckanext-issues

Overview

This would be an extension and improvement of existing todo extension - #842 (https://github.com/okfn/ckanext-issues)
- Current extension does work but lacks polish
- Should we polish or do significant improvements?
Functionality - see etherpad

Steps

~~(Rename todo(s) -> issue(s) in existing extension)~~

#1232: [super] Interface improvements

thejimmyg — Wed, 20 Jul 2011 16:11:36 GMT

Child tickets:

#1194 "Welcome back" message for newly registered user
#1202 Links to datapkg utility don't lead to info about it
#925 Change the search box icon to remove the down arrow
#923 Search box doesn't work in leaderboard page in stats extension
#1034 Flash message cached
#737 Markdown syntax summary page
#811 Extra field editing form layout breaks when there are long field names

#1233: [super] Improve wiki-style functionality for history

thejimmyg — Wed, 20 Jul 2011 16:29:52 GMT

At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas:

#191 Searching by modification date
#193 Searching by time-related field
#301 Package discussion pages
#1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages
#1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page
Other improvements as per my word doc.

#1787: [super] Improve RESTful API

dread — Wed, 08 Feb 2012 11:54:42 GMT

Lists of entities should be full URLs, rather than just the names
Discoverability - /api/v3/rest should list the entity types that can be listed

This could be v3 of the RESTful interface.

#1432: [super] Data processing system for CKAN and Webstore

rgrp — Tue, 01 Nov 2011 10:19:07 GMT

Super ticket: #1190

A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)

#1096: [super] CKAN Hosted

pudo — Tue, 19 Apr 2011 17:44:22 GMT

Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

introduce model.Site and c.site
- site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI)
Subdomain detector to activate sites.
use site in Authorizer instead of System, have a NullSite? for global things
allow cross-site search
packages are in a list of sites, m:n rather than 1:n
- list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally.

#1240: [super] API v4

rgrp — Sun, 24 Jul 2011 16:41:00 GMT

(Just creating this ticket as somewhere to keep notes)

Decide on REST api versus action API
- Do we want to support both?
Tidying
- Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
- extras merged into normal field list in package
- Get rid of /rest/ so just have api/v1/package
- Get rid of separation of search api from 'rest' api
  - Propose that GET on REST index is search e.g. /package/?q=...
    - This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
Resource read/write in API (separate from package)
- Does this need authorization work?
user/account API - read/write
Remove autocomplete -- can just use search
- Do not worry about backwards compat as should only be used in our js (if others using it too bad!)

#2234: Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN templates

seanh — Fri, 16 Mar 2012 12:59:15 GMT

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds

Feed items should probably be cached somewhere

Suggested

The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.

For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference

Caching is pretty crucial and should probably obey the ttl of the feed.

#1569: Wordpressser

jilly mathews — Mon, 19 Dec 2011 11:27:36 GMT

How much effort will this be to be ready to use?

#2934: Webtests for CKAN

Dominik — Tue, 18 Sep 2012 11:10:47 GMT

The tests in CKAN only test single pages (functional tests) or single methods (unit tests). In order to make sure that the whole system still works as expected, we need tests that cover whole processes.

An example user story to be tested would be: "As a user with create permissions I want to be able to log in, create a new resource, save it and then make sure that it is in the system." or: "As a user I want to use the search functionality and open a resource from the search page."

These user stories require a different testing system with interaction and processes.

#2909: User profile headers are a bit weird

johnmartin — Thu, 06 Sep 2012 11:35:48 GMT

They seem to be not exactly well thought out. Needs some thought on better display of the information that they contain.

#2910: User listings are not very compelling

johnmartin — Thu, 06 Sep 2012 11:39:47 GMT

They are currently are just a small image / name. They should be: Larger image, Full name (username), description, edits/dataset

#2321: Use resource names/titles in URLs

rgrp — Tue, 24 Apr 2012 03:00:12 GMT

Would make them much nicer but requires:

Ensure uniqueness (or deal with ambiguity)
(?) Slug/names versus titles
- Aside: what is currently called name should probably be title

#370: Use better email encryption for author_email and maintainer_email

shudson@… — Thu, 22 Jul 2010 18:03:39 GMT

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.

#2958: Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8

seanh — Thu, 04 Oct 2012 09:29:36 GMT

e.g. 2012_08(주요국가).xls

I tested in CKAN 2.0 and it seemed to work fine there.

#2475: Upgrade trac.ckan.org to trac 0.12.3

seanh — Tue, 29 May 2012 18:46:16 GMT

This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

http://trac.edgewall.org/wiki/ChangeLog

#2936: Updating a group via the API clears its packages

seanh — Wed, 19 Sep 2012 09:47:46 GMT

If the group dict that you post to the API does not have any 'packages' key then all the group's packages get removed. I think it would be better if you could just update e.g. the group's description without having to also post the list of packages, and apparently this is how other update API actions work.

Might be worth checking all the update API actions for this behaviour, and making sure they're all consistent

#3006: Update translations from Transifex

seanh — Mon, 29 Oct 2012 10:22:24 GMT

There are some new translations already on the CKAN 1.8 resource on Transifex already (e.g. 100% Norwegian translation I think). We should at least pull these from Transifex for 1.8.1, and may as well do a call for translations too.

#2933: Update theming docs

seanh — Mon, 17 Sep 2012 15:21:12 GMT

There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs.

I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents.

#2891: Update or remove DataStore google presentation

seanh — Tue, 28 Aug 2012 10:23:43 GMT

Currently this page:

http://docs.ckan.org/en/ckan-1.8/datastore.html

has an embedded Google docs presentation about data store which contains some outdated information, e.g. links to datastoreclient which was merged into ckanclient.

Suggest moving the google presentation into the documentation proper, so it works with grep etc.

#3020: Update CKAN coding standards

seanh — Fri, 16 Nov 2012 20:11:51 GMT

#2335: Unicode characters don't work in CKAN ini files

seanh — Mon, 30 Apr 2012 12:42:06 GMT

You get a UnicodeDecodeError?, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example.

#2654: UI support for ordering groups on group_read page

ross — Tue, 10 Jul 2012 18:11:31 GMT

The group_index page has no support in WUI for ordering the groups displayed. Should allow sorting by name

Add support for this for datahub now, and discuss for new 1.9 UI

#2988: UI functional tests for CKAN 2.0

seanh — Mon, 15 Oct 2012 11:02:15 GMT

Afaik the new CKAN 2.0 frontend has no functional tests

#653: Trackback links for packages

dread — Tue, 21 Sep 2010 09:40:25 GMT

When people link to a package, a track-back link is auto-created. (Similar system as for blogs).

As suggested by Tim Davies:

Allowing some form of ‘track back’ against datasets When a non-technical user comes to look at a dataset it would be really useful for them to be able to see if anyone has created an interface interpretation of it already.

I found quite a few cases in research of end-users struggling to make sense of a dataset when good interfaces to that data had already been built and blogged about, but without there being any link from the dataset listing to those data uses. Accepting track backs could also make it easier for technical users to find blog posts / shared code etc. relating to a given dataset.

#1561: To do extension

jilly mathews — Mon, 19 Dec 2011 10:52:52 GMT

Can we finish this ready for release on data hub and CKAN Hosted.

#331: Timezone of CKAN timestamps should be configurable

rgrp — Mon, 31 May 2010 10:40:40 GMT

Revisions are timestamped using the server's clock, which may not relate to the expected timezone for the site. e.g. the Norway site has a server on GMT. No timezone info is displayed either.

Would like to set timezone for a CKAN instance to use in rendering revision timestamps. For example, use CET or EST timezone.

#1800: Tidy up _list() and _search() functions in ckan/logic/action/get.py

seanh — Thu, 16 Feb 2012 15:12:09 GMT

For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed.

Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality.

#2978: Tests for permissions for organizations and groups

seanh — Mon, 15 Oct 2012 10:43:26 GMT

From Toby: checking the logic around the who can do what from the user stories - via some tests may be a good approach

#2975: Tests for auth functions and new actions

seanh — Mon, 15 Oct 2012 10:41:00 GMT

From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions

#2927: Test new package/group/organization type URLs support

seanh — Thu, 13 Sep 2012 16:02:33 GMT

IPackageForm, IGroupForm and IOrganizationForm allow you to define e.g. new package types with new top-level URLs, but the example extensions don't have tests for this yet.

#2537: Test and document ckanbuild

seanh — Fri, 15 Jun 2012 15:48:48 GMT

https://github.com/okfn/ckanbuild

Verify that what's there so far still works, write a README explaining how it works

#2893: Tags missing from form_to_db_package_schema()

seanh — Tue, 28 Aug 2012 12:11:04 GMT

With an IDatasetForm plugin active with a form_to_db_package_schema() method that returns a schema based on ckan/logic/schema.py:form_to_db_package_schema() I'm getting validation errors like this:

[ckan.controllers.api] Validation error: "{'type': 'Validation Error', 'junk': [u'The input field junk was not expected.']}"

They seem to be caused when a data dict with key 'tags' is passed to e.g. the package_update API, for example in ckan/tests/functional/api/model/test_vocabulary.py there are various tests that get this error when they call package_update (if you run the tests with a suitable IDatasetForm plugin active).

In ckan/logic/schema.py:form_to_db_package_schema() 'tags' is for some reason deliberately removed from the schema.

#1145: Support the Handle System

timmcnamara — Thu, 19 May 2011 00:26:15 GMT

The Handle System is an initiative to provide persistent references for resources. That is, it's basically a proxy system for preventing link rot.

Its documentation is here: http://www.handle.net/. Servers running CKAN could host a "Local Handle Service", which redirects a hash of a resource to an actual URL.

Some suggested use cases:

Researcher would like to cite where data came from
Agencies would like to have a way to prevent vendor lock-in from CKAN if they decide to move to another platform

#1179: Support tag aliases

timmcnamara — Tue, 07 Jun 2011 06:47:01 GMT

A small number of tags are near-duplicates of each other.

Perhaps we could support word stemming from NLTK and/or manual tag aliases:

statistics statistik ... survey surveying surveys

#1458: Support previewing kml files in data viewer

rgrp — Thu, 10 Nov 2011 14:48:08 GMT

Super ticket: #1151 (viewing geo data)

E.g. preview of http://thedatahub.org/dataset/louisvillecrime should bring up a map

#1184: Support Wuala as CKAN storage option

timmcnamara — Tue, 14 Jun 2011 06:59:11 GMT

Most of CKANs storage options are tied to the USA. This brings concerns of data security for some organisations who may wish to adopt the system. Wuala is a distributed file system that stores data in a peer-to-peer manner. The company behind it, LaCie? sells storage for a fee. However, they also enable clients to have 'free' storage space when machines act as a storage node.

In order to be a storage node, a machine needs to be online for more than 14% of the time - roughly 4h per day. Most CKAN servers are likely to have a far greater uptime than this.

Supporting Wuala would go some way to enabling CKAN to be used in a secure manner. That is, CKAN could be promoted for organisational use where there is lots of data to be stored and large geographic distances to be managed. There is a Python client available and a fairly long Google Tech Talk that overviews the system.

#2889: Support JSON values in extras, when returning JSON from API

seanh — Fri, 24 Aug 2012 12:35:37 GMT

For example, edit the warandpeace dataset and add an extra with key "comments" and value "[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]", now visit API URLs such as:

/api/search/dataset?sort=metadata_modified+desc&all_fields=1&limit=10

/api/rest/dataset/warandpeace

/api/action/package_show?id=warandpeace

The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects.

#1144: Support DSPL

timmcnamara — Thu, 19 May 2011 00:06:49 GMT

DSPL, the Dataset Publishing Language, is being promoted by Google for its "Google Public Data Explorer" system. It is an XML format with metadata.

The format is described on the developer docs ofthe Google Code site.

Google provides a Python script which reads CSV data and generates DSPL

Sample from http://code.google.com/apis/publicdata/docs/dspl_sample.html:

<?xml version="1.0" encoding="UTF-8"?>
<dspl xmlns="http://schemas.google.com/dspl/2010"
    xmlns:geo="http://www.google.com/publicdata/dataset/google/geo"
    xmlns:geo_usa="http://www.google.com/publicdata/dataset/google/geo/us"
    xmlns:time="http://www.google.com/publicdata/dataset/google/time"
    xmlns:quantity="http://www.google.com/publicdata/dataset/google/quantity"
    xmlns:entity="http://www.google.com/publicdata/dataset/google/entity">
  <import namespace="http://www.google.com/publicdata/dataset/google/time"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/quantity"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/entity"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/geo"/>
  <info>
    <name>
      <value>My statistics</value>
    </name>
    <description>
      <value>Some very interesting statistics about countries</value>
    </description>
    <url>
      <value>http://www.stats-bureau.com/mystats/info.html</value>
    </url>
  </info>
  <provider>
    <name>
      <value>Bureau of Statistics</value>
    </name>
    <url>
      <value>http://www.stats-bureau.com</value>
    </url>
  </provider>
  <topics>
    <topic id="geography">
      <info>
        <name><value>Geography</value></name>
      </info>
    </topic>
    <topic id="social_indicators">
      <info>
        <name><value>Social indicators</value></name>
      </info>
      <topic id="population_indicators">
        <info>
          <name><value>Population indicators</value></name>
        </info>
      </topic>
      <topic id="poverty_and_income">
        <info>
          <name><value>Poverty & income</value></name>
        </info>
      </topic>
      <topic id="health">
        <info>
          <name><value>Health</value></name>
        </info>
      </topic>
    </topic>
  </topics>
  <concepts>
    <!-- As noted in the tutorial, this concept should extend quantity:amount.-->
    <concept id="population">
      <info>
        <name>
          <value>Population</value>
        </name>
        <description>
          <value>Size of the resident population.</value>
        </description>
      </info>
      <topic ref="population_indicators"/>
      <type ref="integer"/>
    </concept>
    <!-- This country concept is defined for educational purposes only. A country
    concept exists in the Google geo dataset. See:
    http://code.google.com/apis/publicdata/docs/canonical/geo.html -->
    <concept id="country" extends="geo:location">
      <info>
        <name>
          <value>Country</value>
        </name>
        <description>
          <value>My list of countries</value>
        </description>
      </info>
      <type ref="string"/>
      <property id="name">
        <info>
          <name><value xml:lang="en">Country name</value></name>
          <description>
            <value xml:lang="en">The official name of the country</value>
          </description>
        </info>
        <type ref="string"/>
      </property>
      <table ref="countries_table"/>
    </concept>
    <!-- This US state concept is defined for educational purposes only. A US state
      concept exists in the Google geo US dataset. See:
      http://code.google.com/apis/publicdata/docs/canonical/geo.us.html -->
    <concept id="state" extends="geo:location">
      <info>
        <name>
          <value>State</value>
        </name>
        <description>
          <value>US states</value>
        </description>
      </info>
      <type ref="string"/>
      <property concept="country" isParent="true"/>
      <table ref="states_table"/>
    </concept>
    <concept id="gender" extends="entity:entity">
      <info>
          <name>
          <value>Gender</value>
        </name>
        <description>
          <value>Gender, Male or Female</value>
        </description>
        <pluralName><value>Genders</value></pluralName>
        <totalName><value>Both genders</value></totalName>
      </info>
      <type ref="string"/>
      <table ref="genders_table"/>
    </concept>
    <concept id="unemployment_rate" extends="quantity:rate">
      <info>
        <name>
          <value>unemployment rate</value>
        </name>
        <description>
          <value>The percent of the labor force that is unemployed, not seasonally
            adjusted.</value>
        </description>
        <url><value>http://www.bls.gov/cps/cps_htgm.htm</value></url>
      </info>
      <topic ref="social_indicators"/>
      <type ref="float"/>
      <attribute id="is_percentage">
        <type ref="boolean"/>
        <value>true</value>
      </attribute>
    </concept>
  </concepts>
  <slices>
    <slice id="countries_slice">
      <dimension concept="country"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <table ref="countries_slice_table"/>
    </slice>
    <slice id="states_slice">
      <dimension concept="state"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <metric concept="unemployment_rate"/>
      <table ref="states_slice_table"/>
    </slice>
    <slice id="countries_gender_slice">
      <dimension concept="country"/>
      <dimension concept="gender"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <table ref="countries_gender_slice_table"/>
    </slice>
  </slices>
  <tables>
    <table id="countries_table">
      <column id="country" type="string"/>
      <column id="name" type="string"/>
      <column id="latitude" type="float"/>
      <column id="longitude" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">countries.csv</file>
      </data>
    </table>
    <table id="countries_slice_table">
      <column id="country" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <data>
        <file format="csv" encoding="utf-8">country_slice.csv</file>
      </data>
    </table>
    <table id="states_table">
      <column id="state" type="string"/>
      <column id="name" type="string"/>
      <column id="country" type="string">
        <value>US</value>
      </column>
      <column id="latitude" type="float"/>
      <column id="longitude" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">states.csv</file>
      </data>
    </table>
    <table id="states_slice_table">
      <column id="state" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <column id="unemployment_rate" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">state_slice.csv</file>
      </data>
    </table>
    <table id="genders_table">
      <column id="gender" type="string"/>
      <column id="name" type="string"/>
      <data>
        <file format="csv" encoding="utf-8">genders.csv</file>
      </data>
    </table>
    <table id="countries_gender_slice_table">
      <column id="country" type="string"/>
      <column id="gender" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <data>
        <file format="csv" encoding="utf-8">gender_country_slice.csv</file>
      </data>
    </table>
  </tables>
</dspl>

#1069: Stub datasets (request for datasets)

rgrp — Fri, 01 Apr 2011 14:08:39 GMT

Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki.

We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ...

(Just as we have datasets listed that exist but aren't available ...)

Alternative would be to have a request for datasets subsystem.

I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists.

Implementation

Agree a new dedicated tag. e.g. todo.does-not-exist

This ticket has clear links to http://getthedata.org/

#1564: Structured Data (Data API)

jilly mathews — Mon, 19 Dec 2011 11:07:14 GMT

Basic websotre exists but this may be not what is described yet.

CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first.

#2197: Storage Metadata API: add/update not working with local file storage (Pairtree)

zydio — Tue, 28 Feb 2012 09:29:37 GMT

If OFS is configured with Pairtree to use a local file storage, all POST requests to add/update metadata ( /api/storage/metadata/{label} ) will fail.

This is due to the use of BotoOFS specific private methods in StorageAPIController.set_metadata(), eg: self.ofs._require_bucket(bucket), self.ofs._get_key(b, label), self.ofs._update_key_metadata(k, metadata) ... those methods can't be found in POTFS and this causes errors. The API should use only OFSInterface methods, or should conditionally make calls based off the actual type of self.ofs.

PS: I did set "ckan" as "Component" in the ticket because storage has been integrated back into the core in CKAN 1.6

#2899: Step 2 > Save & add another bug

johnmartin — Thu, 30 Aug 2012 13:15:19 GMT

Goto:

Add dataset

Get to step 2

Click "Save & add another" without adding any data

Ends on step 3

Should: fail to validate and remain on step 2

#2965: Stats extension broken on 1.8

amercader — Mon, 15 Oct 2012 10:15:41 GMT

Graphs not showing (looks like a flot related file is missing)
Wrong groups counts (e.g. Data Explorer Examples show 1800 datasets when it onlu has 8)

#2788: Speed improvements on creating/updating and indexing

amercader — Wed, 01 Aug 2012 14:56:32 GMT

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

#2337: Sort out ini file confusion in docs

seanh — Mon, 30 Apr 2012 14:57:56 GMT

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

#2731: Some sites permanently 'down for maintenance'

markw — Mon, 23 Jul 2012 12:06:08 GMT

A large number of XXX.ckan.net sites give the following message:

"This Site is Down for Maintenance We apologize for the inconvenience. ~ The Open Knowledge Foundation sysadmins."

The message is unhelpful and patently false - the sites do not exist. Some of them were supposed to have been redirected to a relevant group at thedatahub.org in this ticket (now closed):

http://trac.okfn.org/ticket/933

However, the redirection only seems to have worked in one case, http://si.ckan.net.

The problem still affects the following sites - the first 4 of which have supposedly been merged:

Please sort this out by redirecting, removing the sites, giving a more helpful (and accurate) failure message, etc, as appropriate.

#2904: Show more formats/groups/tags shouldn't display if there aren't more

johnmartin — Wed, 05 Sep 2012 14:14:04 GMT

On the search results page when there are no more formats, groups or tags to display it should not display the show more links.

#2486: Should be able to use . in dataset names

seanh — Sat, 02 Jun 2012 16:46:23 GMT

#277: Set some config options / settings in WUI (extension)

dread — Mon, 22 Mar 2010 16:21:01 GMT

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN
## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png
## Site tagline / description (used on front page)
ckan.site_description =
## Used in creating some absolute urls (such as rss feeds, css files) and
## dump filenames
ckan.site_url =
## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico
## An 'id' for the site (using, for example, when creating entries in a common search index)
## If not specified derived from the site_url
# ckan.site_id = ckan.net
## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net
## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end =

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

[namespace]: ? only if KeyValues? (for settings this would then always be settings)
key
label
value (json)
type (e.g. date and to specify in advance what type should be)
description
tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

/ckan-admin/settings
Show label, plus description plus text field

Depends

Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)

#350: Search engine optimisation

dread — Tue, 15 Jun 2010 09:35:41 GMT

Need to research what can easily be done to improve CKAN packages in the search rankings.

Comments from Glen Barnes:

We've been pretty successful at SEO without even really trying (see http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (http://www.ckan.net/package/ascoe -> http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly ("Atmospheric Chemistry Studies in the Oceanic Environment" on the example above). Some basic SEO 101 on page optimisations.

#2962: Search across multiple ckan instances

dominik — Mon, 08 Oct 2012 22:50:22 GMT

Could be done by:

using the solr distributed search
- difficult set up
merging result sets from apis
- make sure that results can be merged properly (score, facets, ...)

#2535: SSL certificate for DataHub + https by default

rgrp — Fri, 15 Jun 2012 11:00:27 GMT

DataHub? is increasingly used and we should ensure it uses ssl as part of general security.

See also #1446 (Need to support https login for multiple instances as part of the CKAN package install)

CKAN: Ticket Query

#2699: workflow for associating datasets with groups

#2780: way for admins to undelete datasets

#2869: user profile not linked on disqus until page refresh

#1414: track user log-ins on thedatahub.org

#2762: test_related.py crashes

#2786: target blank HTML downloads

#3027: solr for 2.0

#2900: snippets/package_list.html template crashes on dataset that has no 'notes'

#1201: seperate out logic in atom feeds to logic layer.

#2729: searching for tags:[tag] works but tag:[tag] doesn't

#2896: resource_create crashes on ValidationErrors with KeyError: 'resources'

#1668: repoze version discrepency

#2573: package_search does not allow solr's per-field facet parameters

#2768: normalise excel to xls

#2863: non-admin users can edit other's draft datasets

#2854: no way to see which dataset an app is related to on /apps page

#2857: no UI to follow another user although option on dashboard

#2579: move sort_by functions into lib.helpers

#2784: model dictize sensitive data

#1606: metadata license config option

#2583: make sure that we implement authentication where needed

#2282: log api calls for users

#2708: limit extra data for package/group show

#2810: heroku ckan support

#2663: h.resource_display_name needs love

#2902: genshi is used in Group controller

#2758: file storage gives error if config not available but no useful user information

#2543: facet.sort is not available in the package_search action

#2721: deleted groups should not show on 'Add to Groups' dropdown

#2728: deleted group shows on search index - for admins

#2815: db_to_form_package_schema() strips tracking summary, isopen

#2897: db_to_form_package_schema() crashes historical dataset page

#3012: data.gov auth

#2697: create dataset validation

#2726: confusing logic on data preview formats

#3013: common-error-messages is unreadable

#2257: cleanup template vars

#2572: clean up stats plugin

#3030: clean up helper functions

#2244: clean up auth

#2912: ckanext/organizations: Neither public or private initially selected when adding dataset

#2873: ckanext-qa: dataset summary incorrect

#2718: can't add dataset to more than one group

#2777: bug: user attributes

#2748: add 'add new resource' button to sidebar

#1717: [super] Search UX improvements

#1578: [super] Re-enable and refactor ratings

#1571: [super] Issues Extension

Steps

#1232: [super] Interface improvements

#1233: [super] Improve wiki-style functionality for history

#1787: [super] Improve RESTful API

#1432: [super] Data processing system for CKAN and Webstore

#1096: [super] CKAN Hosted

#1240: [super] API v4

#2234: Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN templates

#1569: Wordpressser

#2934: Webtests for CKAN

#2909: User profile headers are a bit weird

#2910: User listings are not very compelling

#2321: Use resource names/titles in URLs

#370: Use better email encryption for author_email and maintainer_email

#2958: Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8

#2475: Upgrade trac.ckan.org to trac 0.12.3

#2936: Updating a group via the API clears its packages

#3006: Update translations from Transifex

#2933: Update theming docs

#2891: Update or remove DataStore google presentation

#3020: Update CKAN coding standards

#2335: Unicode characters don't work in CKAN ini files

#2654: UI support for ordering groups on group_read page

#2988: UI functional tests for CKAN 2.0

#653: Trackback links for packages

#1561: To do extension

#331: Timezone of CKAN timestamps should be configurable

#1800: Tidy up *_list() and *_search() functions in ckan/logic/action/get.py

#2978: Tests for permissions for organizations and groups

#2975: Tests for auth functions and new actions

#2927: Test new package/group/organization type URLs support

#1800: Tidy up _list() and _search() functions in ckan/logic/action/get.py