CKAN: {9} Agile Backlog

#2412: More than one resource invalidatiing breaks dataset edit form

icmurray — Tue, 22 May 2012 20:05:48 GMT

When attempting to add more than one resource at once, if more than one resource invalidates, this results in a js error, leaving the form in an inconsistent state.

Repro:

Go to /dataset/new
Add a new resource. Fill in one of the fields with an invalid value. eg - last_modified, or size...
Add another resource, doing the same thing: make one of the fields invalid.
Try to save the dataset.
The entered resource information will be lost, and a js error "Uncaught Error: Can't add the same model to a set twice,: backbone.js:586" will be thrown.

#277: Set some config options / settings in WUI (extension)

dread — Mon, 22 Mar 2010 16:21:01 GMT

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN
## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png
## Site tagline / description (used on front page)
ckan.site_description =
## Used in creating some absolute urls (such as rss feeds, css files) and
## dump filenames
ckan.site_url =
## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico
## An 'id' for the site (using, for example, when creating entries in a common search index)
## If not specified derived from the site_url
# ckan.site_id = ckan.net
## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net
## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end =

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

[namespace]: ? only if KeyValues? (for settings this would then always be settings)
key
label
value (json)
type (e.g. date and to specify in advance what type should be)
description
tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

/ckan-admin/settings
Show label, plus description plus text field

Depends

Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)

#1358: Generate configuration documentation automatically from the deployment_ini_tmpl file

rgrp — Mon, 26 Sep 2011 22:32:30 GMT

At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs.

Suggest we write a script that automatedly generates reference documentation for the config from the source.

May be obsoleted by #277 (some config in db)

#1403: Refactor groups index page

zephod — Mon, 17 Oct 2011 10:31:52 GMT

Groups are listed alphabetically with paging - not an ideal user experience. We would like to list groups in order of 'popularity': The number of datasets they contain.

Following this chain of thought, then, it would be nice to rearrange the groups table by clicking on column headers and having it sort by that column.

Furthermore, then, we'd like to implement a full-fledged groups search feature (if this is at all feasible).

The forthcoming groups refactor will probably have some bearing on this task.

#1411: Force resource format to be lower case (also mimetype)

rgrp — Sat, 22 Oct 2011 21:40:04 GMT

Format should be lowercase. Automatically lower case (for extra points have a bit of javascript to force lower case when entering).

Even more points: do a update on thedatahub repo to make all format lower case (or script this as an update?)

#1507: Minor fixes to dataset add on Group edit form - 0.5d

rgrp — Mon, 05 Dec 2011 12:33:49 GMT

Group edit dataset add form needs some work

Dataset name is not cleared when you add
No way to remove item from list of datasets to be added if I make a mistake
(2nd Apr 2012) It now seems that option to add multiple datasets at once has disappeared (perhaps during the CSS/HTML refactor ...)

#1782: Improvements to datasets page UX (Feb 2012)

rgrp — Tue, 07 Feb 2012 10:10:41 GMT

Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?]
~~Icons for *types* of resources in resource list~~
List of 'Owners' (administrators) at top right of sidebar (a la group page)
- Much prefer owner to administrator ...

#2751: check translations for full demo site

toby — Wed, 25 Jul 2012 11:40:43 GMT

need to check everything gets translated - sean did this before so will have info

#2363: Documentation of best caching practice.

kindly — Tue, 01 May 2012 16:16:57 GMT

Need better documentation on best practices in making page loads faster for non logged in users.

#2326: dataset permalinks

toby — Wed, 25 Apr 2012 16:39:06 GMT

create permalinks

eg \dataset\~a5-DQ\nice-name

~XXXXX defines the permalink the nice-name is ignored

packages table needs a new field for this data

existing datasets need updating

new datasets need the identifier creating

routing needs updating

needs to work for all urls like edit

Once this is implemented we can decide when we will use permalinks vs the normal urls.

#2433: API uses name not id for some version 3 calls

toby — Fri, 25 May 2012 14:54:08 GMT

see https://github.com/okfn/ckan/pull/5

#2461: Create demo revisions page

aron.carroll — Mon, 28 May 2012 13:34:15 GMT

I believe this is to be merged in with site activity.

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts

Implementation:

http://s031.okserver.org:2375/en/revision

#2698: markdown preview

shevski — Wed, 18 Jul 2012 14:01:25 GMT

for description / other fields with markdown support

#2775: Add bin/less to paster serve command

aron.carroll — Mon, 30 Jul 2012 22:01:26 GMT

Ideally the ./bin/less command would be run when the server is started.

Also it would be good to have a paster command to build the production CSS with {{{ ./bin/less --production }}
The command could also detect missing node binaries and redirect to the documentation.

#2822: Resource additional info titles format/i18n

toby — Thu, 09 Aug 2012 09:30:20 GMT

the title for additional info should be translated

capitalised etc

#2416: Normalise resource/data types

aron.carroll — Wed, 23 May 2012 17:09:06 GMT

Currently we have far too many types that are essentially the same format. The new demo theme is using icons for common types. So we need to normalise the common formats into pretty strings. e.g. application/json, JSON, .json and json all should be output as “json”

See the following Basecamp thread for UI examples and discussion.

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056

#2555: Demo site needs a breadcrumb helper

aron.carroll — Mon, 18 Jun 2012 13:43:03 GMT

Something to make building breadcrumbs a bit nicer

#2823: resource additional info title order

toby — Thu, 09 Aug 2012 09:32:53 GMT

Order the items so that none user fields are first from ticket #2707

#2938: Explain bundling in doc/resources.rst

dominik — Mon, 24 Sep 2012 12:13:55 GMT

Since it makes sense to bundle resources into one js file that are commonly used together, there should be some explanations on how separate js scripts are merged into one file for production.

Explanations could include how groups are handled or different resource files.

#2939: Orgs are groups

toby — Tue, 25 Sep 2012 09:07:14 GMT

holder for ticket number

#3012: data.gov auth

toby — Thu, 08 Nov 2012 16:50:09 GMT

#2244: clean up auth

toby — Mon, 19 Mar 2012 17:22:51 GMT

cache admins etc
aim to move check_access so independent of logic function
general spruce up

#2282: log api calls for users

toby — Tue, 10 Apr 2012 16:18:45 GMT

This was part of #2251 but is now it's own ticket.

we need clarity on this. The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some? do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me.

#2411: .ini file is a mess

markw — Tue, 22 May 2012 11:41:40 GMT

The .ini file needs cleaning up in assorted ways which Toby will expand upon.

#2499: Documentation for the internal analytics

icmurray — Thu, 07 Jun 2012 11:58:17 GMT

There's no docs for setting up the internal analytics, nor what it provides.

Setup

Add: ckan.tracking_enabled = true to .ini file

Run: paster tracking -c dev.ini

#2257: cleanup template vars

toby — Fri, 23 Mar 2012 14:35:07 GMT

look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc

#2572: clean up stats plugin

toby — Tue, 19 Jun 2012 11:24:14 GMT

attempt to disengage the stats plugin from core as much as possible

#2579: move sort_by functions into lib.helpers

toby — Thu, 21 Jun 2012 12:29:58 GMT

make these more available but keep existing functionality so not to break any users

remove_field()

drill_down_url()

etc

#2663: h.resource_display_name needs love

toby — Wed, 11 Jul 2012 14:42:26 GMT

This function is shit and needs cleaning up and a doc string

description is markdown and should be treated properly

either we should truncate all or leave it to the templates but work universally

url if no name / desc this is in demo-theme branch

#2758: file storage gives error if config not available but no useful user information

toby — Thu, 26 Jul 2012 07:28:29 GMT

We get an error which should be improved the actual problem is this but not passed to user

⇝ KeyError?: 'ofs.impl'

View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view

data = func(*args, kwargs)

Module ckan.controllers.storage:407 in auth_form view

authorize(method, bucket, label, c.userobj, self.ofs)

Module ckan.controllers.storage:200 in ofs view

StorageAPIController._ofs_impl = get_ofs()

Module ckan.controllers.storage:71 in get_ofs view

storage_backend = configofs.impl?

Module paste.registry:146 in getitem view

return self._current_obj()[key]

KeyError?: 'ofs.impl'

#2718: can't add dataset to more than one group

shevski — Fri, 20 Jul 2012 10:35:50 GMT

trying to add a dataset to another group means it's no longer part of the first group

http://demo.ckan.org/dataset/edit/afterfibre

#2721: deleted groups should not show on 'Add to Groups' dropdown

shevski — Sun, 22 Jul 2012 09:17:12 GMT

Groups previously deleted still show up in the add dataset process in step 3 'Additional info'

http://s031.okserver.org:2375/dataset/new_metadata/ff

#2728: deleted group shows on search index - for admins

shevski — Sun, 22 Jul 2012 09:53:04 GMT

'test-group', which has been deleted,shows up on main search page under groups - and can be filtered by - see http://s031.okserver.org:2375/dataset?groups=test-group

#2780: way for admins to undelete datasets

shevski — Tue, 31 Jul 2012 12:20:48 GMT

Since admins can see deleted datasets - there should be a way for them a) to know they are currently deleted & not viewable by normal users - ticket: #2779 b) way to undelete such datasets - this ticket

I suggest a button on the edit form instead of the delete button i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message?

#2828: Draft datasets are confusing - tickets need creating

shevski — Thu, 09 Aug 2012 09:56:18 GMT

reported as editing datasets incorrect

e.g. clicking on edit here http://s031.okserver.org:2375/dataset/ff takes you to create dataset page http://s031.okserver.org:2375/dataset/edit/ff

but this is the correct behaviour of a draft dataset

We need to show draft datasets correctly

proper tickets need making for the different issues after review of issues with them - who can see, where, admins and viewing, orgs too etc

#2830: Need method to undelete groups

toby — Thu, 09 Aug 2012 10:50:08 GMT

need controller action and front-end method

#2857: no UI to follow another user although option on dashboard

shevski — Wed, 15 Aug 2012 16:59:43 GMT

e.g. second option in left hand sidebar is follow another user' http://s031.okserver.org:2375/user/dashboard but if you click on that and then any user, there's no follow button

to me the dashboard user stuff is highly unfinished and probably should not be included unless someone is very keen for it

#2861: Add qa breadcrumb

shevski — Thu, 16 Aug 2012 10:47:26 GMT

can we add in the breadcrumb navigation to qa pages so that you can go back to http://s031.okserver.org:2375/qa from sub pages such as http://s031.okserver.org:2375/qa/dataset/five_stars

#2862: Revised revised groups description

markw — Thu, 16 Aug 2012 10:52:02 GMT

Revised text for 'What are groups?' box at demo.ckan.org/group (after discussion with IB re #2812):

What are groups?

Groups allow you to group users and data together so that they are easier to manage. Group owners can assign roles and authorisations, giving each project or department control of its own data publishing.

Users can browse or facet by groups, which could be an organisation (for example, the Department of Health) or topic (e.g. Transport, Health), making it easier to find the data they are looking for.

#2863: non-admin users can edit other's draft datasets

shevski — Thu, 16 Aug 2012 10:54:39 GMT

edit button shows up for daniel lewis for http://s031.okserver.org:2375/dataset/ff

#2873: ckanext-qa: dataset summary incorrect

danieljohnlewis — Thu, 16 Aug 2012 14:13:49 GMT

For example see: http://s031.okserver.org:2375/qa/dataset/five_stars

Which has: Example dataset as a score of: 4 After examination of the dataset ( http://s031.okserver.org:2375/dataset/example-dataset ) it is made up of a CSV and an HTML file.

#2583: make sure that we implement authentication where needed

toby — Fri, 22 Jun 2012 10:04:38 GMT

in development many auth checks may have been lost we need to check they are still working etc

#2725: Case sensitivity on tags

shevski — Sun, 22 Jul 2012 09:34:27 GMT

My feeling is that 'country-US' and 'country-us' should be the same tag. However currently tags with caps are treated differently

see http://s031.okserver.org:2375/en/dataset/test-dataset

with TEST and test - there also get indexed twice in the search page

#2726: confusing logic on data preview formats

shevski — Sun, 22 Jul 2012 09:38:47 GMT

If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this

If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28

#2735: Dataset order on user page

shevski — Mon, 23 Jul 2012 17:01:17 GMT

I think the datasets on user pages http://s031.okserver.org:2375/user/me should be ordered by latest updated (with most recent at the top) instead of in alphabetical order.

What do you think?

#2768: normalise excel to xls

shevski — Thu, 26 Jul 2012 16:16:22 GMT

so that data proxy works

#2813: Confusing sidebar on demo dataset page

markw — Wed, 08 Aug 2012 10:03:26 GMT

On a dataset page on demo.ckan.org, the left sidebar is confusing.

It starts with some random links. Actually they are links to groups which the dataset is in, but this isn't clear.
The sidebar elements that are actually part of the dataset are 'Datset extent' and 'License', so these should be right at the top (if they belong in the sidebar at all), instead of which they are right at the bottom in the junk part of the page (i.e. probably lower than the bottom of the main page, and hence lower than anyone will scroll).

#2362: Improve plugin documentaion, including examples.

kindly — Tue, 01 May 2012 16:13:11 GMT

Improve documentation for plugins and add examples. Could use ckanext-example as a base.

#1069: Stub datasets (request for datasets)

rgrp — Fri, 01 Apr 2011 14:08:39 GMT

Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki.

We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ...

(Just as we have datasets listed that exist but aren't available ...)

Alternative would be to have a request for datasets subsystem.

I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists.

Implementation

Agree a new dedicated tag. e.g. todo.does-not-exist

This ticket has clear links to http://getthedata.org/

#235: Resource format normalization and detection

dread — Mon, 18 Jan 2010 15:13:24 GMT

Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service.

Sub-tasks:

Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -> "text/csv" and "Name mappings", i.e. "text/csv" -> "Comma-Separated Spreadsheet".
Collect and map surface forms from all CKANs
Access this via Swiss and apply, store as a PackageResource? extra field pending #826 (Resource extras).
Add heuristics for format auto-detections:
- Map well-known file extensions
- Recognize obvious magic (Zip, Tar)
- Peek into Zipfile/Tarfiles?
Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar)
See also: #816 (Autocomplete for the resource format field)

#1041: Start Using the CKAN Wiki for Tutorial-style documentation

thejimmyg — Wed, 16 Mar 2011 14:11:55 GMT

For example, I will document the following:

A specific guide on using mercurial when working with our branching and merging policy
- DONE - http://wiki.ckan.net/Becoming_a_CKAN_Developer
Policy document on how we use tickets, plan sprints and make releases
- DONE - http://wiki.ckan.net/Becoming_a_CKAN_Developer
How to package software as .deb file for project deployment

I'd love if someone else would write:

An authorisation tutorial covering the core model, the command line tools and examples of every possible way of using the system
A HOWTO guide with screenshots for adding a package

#1168: Test system for deb packaging

dread — Thu, 26 May 2011 20:33:14 GMT

Get buildbot to:

build the deb packages
install them into a fresh virtual machine
run smoke tests on the installed ckan

#2268: UKLP Use same schema as form for harvested records

amercader — Fri, 30 Mar 2012 11:56:46 GMT

Datasets created via the harvesting (UKLP ones) and the ones created via the form use a different schema. That could lead to problems. We need to discuss the best approach.

#143: Most active users listed on homepage

dread — Thu, 08 Oct 2009 13:59:33 GMT

Display league of users' recent activity on homepage.

#1382: Deleted resources are present for harvested package

thejimmyg — Tue, 11 Oct 2011 12:58:09 GMT

Perhaps the importer deletes them before re-importing. We shouldn't have deleted resources, so let's investigate.

#1287: NAVL validation errors - Junk fields should be listed explicitly

dread — Wed, 24 Aug 2011 16:25:02 GMT

When you create a package, but specify a key that is not allowed (e.g. 'relationships') then you get error message:

{"__junk": ["The input field __junk was not expected."]}

It should mention the actual key which is not expected. e.g.

{"relationships": ["The input field 'relationships' was not expected."]}

Kindly said that James' version of NAVL was better in this respect, so this might be best solved by moving to that.

#2644: user dashboard for demo theme

toby — Tue, 10 Jul 2012 09:00:21 GMT

we now have a user dashboard that needs theming not sure if we need sam to look at it

http://localhost:5000/user/dashboard

#2814: Demo: upload file behaves oddly

markw — Wed, 08 Aug 2012 10:06:00 GMT

Uploading a file behaves counter-intuitively (I would suggest wrongly).

When adding a new resource by uploading a file, I select a file called say create-group.png. I expect the following to happen:

the pathname of that file is filled in the box;
nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc).

Instead of this,

the file is immediately and, seemingly, irrevocably uploaded;
the box is filled with a mysterious URL (https://commondatastorage.googleapis.com/ckan-demo ...)
the mysterious URL is so long I can't even see the final element, which is my file name.

#2956: Allow for resource editing in CKAN 2.0

johnmartin — Wed, 03 Oct 2012 15:11:40 GMT

On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually.

#2590: Publisher dashboard

ross — Mon, 25 Jun 2012 10:03:37 GMT

Need proper user stories but ...

Publisher admins/editors may need a more useful group read page showing things like:

The current search

Recent activity

People within the group

Followers

Others?

#2699: workflow for associating datasets with groups

shevski — Wed, 18 Jul 2012 14:02:56 GMT

needs review & speccing out e.g. datasets created by a user who belongs to a certain publisher (group) get auto added to this group

#2702: Future Javascript wishlist for demo

shevski — Wed, 18 Jul 2012 14:07:55 GMT

tooltip on popular datasets with number of views facets to update automatically creating a dataset without reloading page between steps hover on licences information autocomplete on search terms group filtering social share buttons in lightboxes dataset counts on homepage

#2547: ODS Initial data sets

ross — Mon, 18 Jun 2012 10:55:06 GMT

Requirements

The ODS demo site will need data adding, initially as fixtures but it would also be useful if we started evaluating datasets that we can ship with ODS installations (at least in the UK) from places such as DGU and ONS.

May wish to create a ticket for making sure the datasets within the system are reset every X hours. Perhaps.

Interface

None

User Stories

As a new system administrator for an ODS instance, I don't want to have a site devoid of any data. Geographically relevant datasets would be welcomed.

As a bizdev person I would like to be able to demonstrate how ODS works with real datasets.

Tasks

[ ] Identify relevant sources for datasets

[ ] Pick datasets

[ ] Set them up for import

Estimates

#2766: prevent draft datasets making it to activity stream

shevski — Thu, 26 Jul 2012 16:06:35 GMT

The new ckan creates datasets as part of a 3 phase process. To allow for this partially created datasets can have a state that is 'draft' or 'draft-complete'. These datasets should not be seen as active by the activity stream.

If we click 'add dataset' and then complete the first phase of adding a dataset then we end up having a activity stream created. When we add a resource in the next phase (add data) again an activity stream item xxx added resource to dataset is created

We do not want these adding. Essentially if a dataset has a state.startswith('draft') then we want the activity stream to ignore all actions involving it.

finally when the state is changed form state.startswith('draft') to state=='active' we want a xxx has created dataset ... to be added to the stream.

The best way to do this would be to branch from 2375-demo-theme-stable and get it to work there.

Let me know if you need any help with this ticket or a better explanation of the problem.

#2430: Look into marking tickets as dependent on other tickets in trac

seanh — Fri, 25 May 2012 11:58:06 GMT

#2431: Trac git integration

seanh — Fri, 25 May 2012 11:58:48 GMT

#2466: Fix spam handling on trac

seanh — Mon, 28 May 2012 13:51:43 GMT

#2818: Improve related item schema

danieljohnlewis — Thu, 09 Aug 2012 09:22:56 GMT

Problem: When creating a related item (e.g. a Visualisation), if you don't put in a URL it succeeds, but on the related items and apps pages it renders it as a link to the same page. Expected: Always require a URL and it should only submit if one is added

#2952: incorporate javascriopt translations into translations workflow

icmurray — Mon, 01 Oct 2012 19:15:45 GMT

There are now javascript translations. There's a paster command, trans js. This is a ticket to ensure that using trans js is documented in the normal ckan translation workflow (which is followed after a feature freeze, prior to release). And to see if there's any need for further incorporation into the current translation process.

#2301: Replace old-style string formatting with format() method (at least in strings marked for translation)

seanh — Mon, 16 Apr 2012 12:27:44 GMT

Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words.

Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.:

'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible')

http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting

We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated.

#2243: Fix ckanext-example

seanh — Mon, 19 Mar 2012 15:58:30 GMT

#2341: Fix Jenkins issue when testing branches that contain model changes

seanh — Tue, 01 May 2012 12:46:36 GMT

#2342: Fix Jenkins issue when testing branches with different solr schema versions

seanh — Tue, 01 May 2012 12:47:25 GMT

#2336: Move Jenkins' install script into ckan core so it can be versioned

seanh — Mon, 30 Apr 2012 13:35:14 GMT

#2656: Feed with few results has bad paging link, causing exception

dread — Wed, 11 Jul 2012 10:05:12 GMT

This page http://thedatahub.org/feeds/custom.atom?q=wombat has 0 results and contains a link to http://thedatahub.org/feeds/custom.atom?q=wombat&page=0 which the page=0 causes this exception:

ckan.lib.search.common.SearchError'>: SOLR returned an error running query
Error: "'start' parameter cannot be negative"

#2870: 1.8 tag_list not defined

ross — Thu, 16 Aug 2012 10:58:52 GMT

From: <thedatahub.org@…> Date: Thu, Aug 16, 2012 at 3:20 AM Subject: WebApp? Error: <class 'genshi.template.eval.UndefinedError?'>: "tag_list" not defined To: ckan-sysadmin@…, kindly@…

URL: http://ckan.net/dataset/uk-postboxes Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:155 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:131 in call << r'\1', oldpath)

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:239 in call << try:

res = WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

res = WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.package:322 in read << template = template[:template.index('.') + 1] + format

return render(template, loader_class=loader)

def comments(self, id):

return render(template, loader_class=loader)

Module ckan.lib.base:153 in render << try:

return cached_template(template_name, render_template,

loader_class=loader_class)

except ckan.exceptions.CkanUrlException?, e:

raise

loader_class=loader_class)

Module pylons.templating:249 in cached_template << return content

else:

return render_func()

return render_func()

Module ckan.lib.base:102 in render_template << if loader_class == NewTextTemplate?:

return literal(stream.render(method="text", encoding=None))

return literal(stream.render(method=method, encoding=None,

return literal(stream.render(method="text", encoding =None))

Module genshi.core:183 in render << method = self.serializer or 'xml'

generator = self.serialize(method=method, kwargs) return encode(generator, method=method, encoding=encoding, out=out)

def select(self, path, namespaces=None, variables=None):

return encode(generator, method=method, encoding=encoding, out =out)

Module genshi.output:57 in encode << _encode = lambda string: string

if out is None:

return _encode(.join(list(iterator)))

for chunk in iterator:

out.write(_encode(chunk))

return _encode(.join(list(iterator)))

Module genshi.output:569 in call << def call(self, stream):

strip_markup = self.strip_markup for event in stream:

if event[0] is TEXT:

data = event[1]

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module ckanext.googleanalytics.plugin:93 in download_adder << [downloaded %s times]</span>

count = None for mark, (kind, data, pos) in stream:

if mark and kind == START:

href = data[1].get('href')

for mark, (kind, data, pos) in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1175 in call << """

callable_value = hasattr(self.value, 'call') for mark, (kind, data, pos) in stream:

if mark is ENTER:

if callable_value:

for mark, (kind, data, pos) in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.template.base:605 in _include << from genshi.template.loader import TemplateNotFound?

for event in stream:

if event[0] is INCLUDE:

href, cls, fallback = event[1]

for event in stream:

Module genshi.template.base:565 in _flatten << elif kind is EXPR:

result = _eval_expr(data, ctxt, vars) if result is not None:

# First check for a string, otherwise the iterable test

result = _eval_expr(data, ctxt, vars)

Module genshi.template.base:277 in _eval_expr << if vars:

ctxt.push(vars)

retval = expr.evaluate(ctxt) if vars:

ctxt.pop()

retval = expr.evaluate(ctxt)

Module genshi.template.eval:178 in evaluate << traceback_hide = 'before_and_this'

_globals = self._globals(data) return eval(self.code, _globals, {'data': data})

return eval(self.code, _globals, {'data': data})

Module ?:51 in <Expression u"tag_list(c.pkg_dict.get('tags', ))"> << <li py:if="c.pkg_dict.get('tags')" class="sidebar-section">

<h3>Tags</h3> ${tag_list(c.pkg_dict.get('tags', ))}

</li>

${tag_list(c.pkg_dict.get('tags', ))}

Module genshi.template.eval:309 in lookup_name << val = BUILTINS.get(name, val)

if val is UNDEFINED:

val = cls.undefined(name)

return val

val = cls.undefined(name)

Module genshi.template.eval:410 in undefined << """Raise an UndefinedError? immediately."""

traceback_hide = True raise UndefinedError?(key, owner=owner)

raise UndefinedError?(key, owner=owner)

UndefinedError?: "tag_list" not defined CGI Variables CKAN_CURRENT_URL '/dataset/uk-postboxes' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CKAN_PAGE_CACHABLE True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,text/plain,text/xml' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'vi,en-us;q=0.7,en;q=0.3' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'coccoc/1.0 ()' PATH_INFO '/dataset/uk-postboxes' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/dataset/uk-postboxes' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '47065' REQUEST_METHOD 'GET' REQUEST_URI '/dataset/uk-postboxes' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '<address>Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7f5312a51650> beaker.cache <beaker.cache.CacheManager? object at 0x7f5312a51790> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7f5312a516d0>> beaker.session {'_accessed_time': 1345083636.079552, '_creation_time': 1345083636.079552} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: >, ) paste.registry <paste.registry.Registry object at 0x7f52e47477d0> paste.throw_errors True pylons.action_method <bound method PackageController?.read of <ckan.controllers.package.PackageController? object at 0x7f52e477b810>> pylons.controller <ckan.controllers.package.PackageController? object at 0x7f52e477b810> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7f52e477be90> pylons.routes_dict {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'} repoze.who.logger <logging.Logger object at 0x7f53127b2b50> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 139994769488336>, 'friendlyform': <FriendlyFormPlugin? 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin? 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>} routes.route <routes.route.Route object at 0x7f5312955a50> routes.url <routes.util.URLGenerator object at 0x7f52eb8c9090> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f5308983a80> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7f52eb8c9090>, {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'})

#3029: JSONP parameter scuppers Search in API

dread — Tue, 11 Dec 2012 15:00:35 GMT

http://datahub.io/api/2/search/package?jsonp=jsonpcallback&q=canada returns

{"count": 0, "results": []}
I believe this worked in CKAN 1.4 or 1.5, but it is broken on 1.7.1, 1.8 and whatever demo.ckan.org is running. I suspect the jsonpcallback parameter is getting sent to SOLR.

This bug prevents using javascript on another site to search CKAN (although hopefully the action API would work).

#3005: Add following/unfollowing of groups: model, API, frontend, tests

seanh — Tue, 23 Oct 2012 17:35:44 GMT

#3028: Feature: dashboard activity stream filtering

seanh — Mon, 10 Dec 2012 12:07:24 GMT

https://github.com/okfn/ckanext-pdeu/issues/13

#1635: Email notifications (e.g. for activity streams)

seanh — Wed, 11 Jan 2012 17:56:27 GMT

CKAN should be able to send email notifications to users.

Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users.

One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well.

Need to implement (at least some of) #1634 before this can be implemented, in order to have something to send notifications about.

Analysis here: http://ckan.okfnpad.org/27

#1667: Add an extension point for rendering activity streams

seanh — Tue, 17 Jan 2012 10:29:11 GMT

Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML.

There needs to be an extension point where extensions can register their own rendering functions for particular activity types.

#1748: Make activity streams conform to http://activitystrea.ms/ standard

seanh — Sun, 05 Feb 2012 14:06:24 GMT

At the very least we should make our JSON output conform to their spec:

http://activitystrea.ms/

#1778: Replace classmethod's with module-level functions

seanh — Mon, 06 Feb 2012 15:53:34 GMT

In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the get() classmethods in the model/ classes.

Doing it with module functions lets us type module.function() instead of module.Class.method().

In many cases we're importing classes directly with from module import Class and then doing Class.method(), but from module import foo is bad style: http://docs.python.org/dev/howto/doanddont.html

#2200: Add vocabulary_id option to tag_show() logic action function

seanh — Wed, 29 Feb 2012 16:41:18 GMT

Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.

#2771: Documentation and examples for IDatasetForm and IGroupForm

seanh — Fri, 27 Jul 2012 12:30:38 GMT

Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests.

The IDatasetForm example should use tag vocabularies (two birds with one stone)

The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples.

Tab Vocabularies docs should reference IDatasetForm example.

When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe)

There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date.

#2815: db_to_form_package_schema() strips tracking summary, isopen

seanh — Wed, 08 Aug 2012 16:47:07 GMT

If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates.

#1785: Replace 'Revisions' page with site-wide activity stream

seanh — Tue, 07 Feb 2012 16:34:18 GMT

Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead?

#2335: Unicode characters don't work in CKAN ini files

seanh — Mon, 30 Apr 2012 12:42:06 GMT

You get a UnicodeDecodeError?, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example.

#2337: Sort out ini file confusion in docs

seanh — Mon, 30 Apr 2012 14:57:56 GMT

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

#2370: Add examples of all config settings in default config file

seanh — Thu, 03 May 2012 10:50:37 GMT

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

#2475: Upgrade trac.ckan.org to trac 0.12.3

seanh — Tue, 29 May 2012 18:46:16 GMT

This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

http://trac.edgewall.org/wiki/ChangeLog

#2477: Add extensions section to readthedocs

shevski — Wed, 30 May 2012 15:54:04 GMT

Need a central up to date place for currently supported extensions (closest we have is http://wiki.ckan.org/List_of_Extensions) & their documentation. I suggest this gets added to a dedicated section of docs.ckan.org

#2478: Get NeverNotifyUpdaterPlugin installed on trac.ckan.org

seanh — Wed, 30 May 2012 16:52:14 GMT

#2516: Make 'Assign to:' field on trac.ckan.org into a dropdown list

seanh — Wed, 13 Jun 2012 09:07:22 GMT

there's a setting for this

#2520: Document undocumented config options

seanh — Wed, 13 Jun 2012 11:52:48 GMT

There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either:

ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled

There are also some options that are in the default deployment.ini even though they're deprecated:

ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously

See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html

It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358

#2619: Omit private datasets from public activity streams

seanh — Thu, 28 Jun 2012 11:49:00 GMT

Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered.

#2537: Test and document ckanbuild

seanh — Fri, 15 Jun 2012 15:48:48 GMT

https://github.com/okfn/ckanbuild

Verify that what's there so far still works, write a README explaining how it works

#2538: Add multiple-instance support to ckanbuild

seanh — Fri, 15 Jun 2012 15:51:39 GMT

Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command.

May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command.

#2539: Investigate the existing ckan debian package for ckanbuild

seanh — Fri, 15 Jun 2012 15:54:21 GMT

Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh?

#2540: Implement a way of upgrading ckan sites using ckanbuild

seanh — Fri, 15 Jun 2012 15:55:40 GMT

When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once.

#2541: Add non-core extensions to ckanbuild

seanh — Fri, 15 Jun 2012 15:57:06 GMT

We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include.

#2542: Create jenkins job to run ckanbuild, and run tests

seanh — Fri, 15 Jun 2012 15:58:08 GMT

It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests.

#1096: [super] CKAN Hosted

pudo — Tue, 19 Apr 2011 17:44:22 GMT

Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

introduce model.Site and c.site
site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI)
Subdomain detector to activate sites.
use site in Authorizer instead of System, have a NullSite? for global things
allow cross-site search
packages are in a list of sites, m:n rather than 1:n
list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally.

#1565: Admin dashboard finished?

jilly mathews — Mon, 19 Dec 2011 11:11:32 GMT

Is testing complete and ready for release?

#1163: Improvements to Storage Extension

rgrp — Thu, 26 May 2011 11:19:38 GMT

Storage is now working but there are

Integrate with Resources (e.g. create a resource for each file upload and give option to associate with a package)
Should we introduce rule that files *not* associated with a Resource are periodically deleted?
Allow setting of a file name/path before upload
Allow for file overwriting/deleting etc (how should this work -- do we want to allow this sort of thing)
Integrate local file upload stuff in api/auth/*
Different Backend Issues

Local file store is rather different from 'remote' storage in various ways:

For remote you don't want to use many buckets as there are bucket limits while for local you want to. Should we there have a single path that users provide which we then partition differently for different backends.

#1311: Modal user register and login form

rgrp — Tue, 06 Sep 2011 08:20:27 GMT

Subticket of: #1294

Rather than having to visit a dedicated page it would be good if registration and login could be done from a modal form (separate or combined ...).

Why

It could be used from dataset creation page in situations where user needs to be registered / logged in to create a dataset (so we could allow someone to start creating a dataset and only get them to login at the end ...)
It allows for quicker and easier logging in
Implementation

See Friedrich's work on the datahub

#1343: [super] User related improvements (login, user pages etc)

rgrp — Wed, 14 Sep 2011 16:18:18 GMT

Disallow account creation via openid - #1386
Require email field - #1319
Require email confirmation to be activated (?)
Improvements to user page (e.g. show activity and more info about user) - #1396
Modal user login - #1311

#1577: Can't upload file with foreign chars in filename

dread — Mon, 19 Dec 2011 18:00:59 GMT

Looks like uploading a file with foreign characters fails due to encoding reasons.

URL: http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:123 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckanext.storage.controller:2 in auth_form Module ckan.lib.jsonp:26 in jsonpify << Very much modelled after pylons.decorators.jsonify . """ data = func(*args, **kwargs) return to_jsonp(data) >> data = func(*args, **kwargs) Module ckanext.storage.controller:301 in auth_form << method = 'POST' authorize(method, bucket, label, c.userobj, self.ofs) data = self._get_form_data(label) return data >> authorize(method, bucket, label, c.userobj, self.ofs) Module ckanext.storage.controller:79 in authorize << if method != 'GET': # do not allow overwriting if ofs.exists(bucket, key): abort(409) # now check user stuff >> if ofs.exists(bucket, key): Module ofs.remote.botostore:53 in exists << if bucket is None: return False return (label is None) or (label in bucket) def claim_bucket(self, bucket): >> return (label is None) or (label in bucket) Module boto.s3.bucket:87 in __contains__ << def __contains__(self, key_name): return not (self.get_key(key_name) is None) def startElement(self, name, attrs, connection): >> return not (self.get_key(key_name) is None) Module boto.s3.bucket:144 in get_key << response = self.connection.make_request('HEAD', self.name, key_name, headers=headers, query_args=query_args) # Allow any success status (2xx) - for example this lets us # support Range gets, which return status 206: >> query_args=query_args) Module boto.s3.connection:388 in make_request << if isinstance(key, Key): key = key.name path = self.calling_format.build_path_base(bucket, key) boto.log.debug('path=%s' % path) auth_path = self.calling_format.build_auth_path(bucket, key) >> path = self.calling_format.build_path_base(bucket, key) Module boto.s3.connection:88 in build_path_base << def build_path_base(self, bucket, key=''): return '/%s' % urllib.quote(key) class SubdomainCallingFormat(_CallingFormat): >> return '/%s' % urllib.quote(key) Module urllib:1222 in quote << safe_map[c] = (c in safe) and c or ('%%%02X' % i) _safemaps[cachekey] = safe_map res = map(safe_map.__getitem__, s) return ''.join(res) >> res = map(safe_map.__getitem__, s) KeyError: u'\xed' CGI Variables AUTH_TYPE 'cookie' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' HTTP_ACCEPT_ENCODING 'gzip,deflate,sdch' HTTP_ACCEPT_LANGUAGE 'en-US,en;q=0.8' HTTP_CACHE_CONTROL 'max-age=259200' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; ckan_user=elenaibp; ckan_display_name="Elena Mondo"; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; __utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10.1324298316; __utmc=119670349; __utmz=119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' HTTP_HOST 'thedatahub.org' HTTP_REFERER 'http://thedatahub.org/dataset/edit/budget-library-czeck-republic' HTTP_USER_AGENT 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7' HTTP_VIA '1.1 localhost (squid/3.0.STABLE19)' HTTP_X_FORWARDED_FOR '87.114.74.190' HTTP_X_REQUESTED_WITH 'XMLHttpRequest' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' REMOTE_ADDR '193.34.146.142' REMOTE_PORT '55419' REMOTE_USER u'elenaibp' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [''] REQUEST_METHOD 'GET' REQUEST_URI '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' SCRIPT_URL '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' SERVER_ADDR '193.34.146.146' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '<address>Apache/2.2.14 (Ubuntu) Server at thedatahub.org Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.14 (Ubuntu)' WSGI Variables application <beaker.middleware.CacheMiddleware object at 0x7f22601c7dd0> beaker.cache <beaker.cache.CacheManager object at 0x7f22601c7b50> beaker.get_session <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f22601c7a90>> beaker.session {'_accessed_time': 1324298703.071357, '_creation_time': 1324293077.4139669} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan.net' mod_wsgi.reload_mechanism '1' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 8) paste.cookies (<SimpleCookie: __utma='119670349.1809834699.1323782464.1324293066.1324298316.4' __utmb='119670349.3.10.1324298316' __utmc='119670349' __utmz='119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode' ckan_apikey='decd48b1-49ee-4250-bff4-98ccca9c02a5' ckan_display_name='Elena Mondo' ckan_user='elenaibp' hide_welcome_message='1' thedatahub_net='27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e'>, 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; ckan_user=elenaibp; ckan_display_name="Elena Mondo"; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; _ _utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10...)|utmcmd=(none)') paste.registry <paste.registry.Registry object at 0x7f226194df50> paste.throw_errors True pylons.action_method <bound method StorageAPIController.auth_form of <ckanext.storage.controller.StorageAPIController object at 0x7f2261dad990>> pylons.controller <ckanext.storage.controller.StorageAPIController object at 0x7f2261dad990> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext object at 0x7f2261daddd0> pylons.routes_dict {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'} repoze.who.identity <repoze.who identity (hidden, dict-like) at 139785645747120> repoze.who.logger <logging.Logger instance at 0x7f225e23c098> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin 139785625065680>, 'friendlyform': <FriendlyFormPlugin 139785618095248>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f2260874c10>, 'auth_tkt': <AuthTktCookiePlugin 139785625065808>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f2260874c90>} routes.route <routes.route.Route object at 0x7f22601a1090> routes.url <routes.util.URLGenerator object at 0x7f2261dadf50> webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multiprocess' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f2261da9af8> wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7f2261dadf50>, {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'})

#2279: Write tutorial for our DataStore API / Data Explorer

rgrp — Tue, 03 Apr 2012 19:44:23 GMT

Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this.

This is documentation / example only (no code!)

#2874: Clean up bin directory

rgrp — Fri, 17 Aug 2012 08:01:48 GMT

Full of obsolete material

#1384: CKAN wiki needs updating to refer to thedatahub.org instead of ckan.net and datasets instead of packages

shevski — Wed, 12 Oct 2011 10:07:57 GMT

Most articles still refer and link to ckan.net, wiki.ckan.net and to packages

#1429: Provide DOIs for datasets in a CKAN instance

rgrp — Sun, 30 Oct 2011 12:21:45 GMT

DOI = digital object identifier = http://www.doi.org/

As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way.

Details

Probably implement as extension rather than core

#1459: Featured Dataset feature

rgrp — Sat, 12 Nov 2011 15:50:12 GMT

Provide way to mark a dataset as featured. Featured database show up on the front page.

TODO: detail this more.

#1571: [super] Issues Extension

jilly mathews — Mon, 19 Dec 2011 11:30:25 GMT

"Proper" issues extension.

Etherpad: http://ckan.okfnpad.org/issues
Code: https://github.com/okfn/ckanext-issues
Overview

This would be an extension and improvement of existing todo extension - #842 (https://github.com/okfn/ckanext-issues)
Current extension does work but lacks polish
Should we polish or do significant improvements?
Functionality - see etherpad
Steps

~~(Rename todo(s) -> issue(s) in existing extension)~~

#1458: Support previewing kml files in data viewer

rgrp — Thu, 10 Nov 2011 14:48:08 GMT

Super ticket: #1151 (viewing geo data)

E.g. preview of http://thedatahub.org/dataset/louisvillecrime should bring up a map

#1009: Improvements to user accounts sytem

rgrp — Fri, 25 Feb 2011 12:13:11 GMT

Forgot password (email a new password)
Confirm email
Do not show register page if you are logged in (redirect to home page)
ticket:1010 - listing of users.
Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)

#2773: About page needs improving

markw — Mon, 30 Jul 2012 11:13:15 GMT

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

#1581: Blog post about Google Analytics extension for CKAN

johnglover — Tue, 20 Dec 2011 17:40:00 GMT

The CKAN Google Analytics extension has been updated to work with the latest version of CKAN, could make for a nice blog post.

Can ping John Glover in January for any details required.

Key link is: http://thedatahub.org/analytics/dataset/top though this should probably move to be under stats (e.g. http://thedatahub.org/stats/usage)

#1171: Citation instructions on dataset and resource view pages

dread — Wed, 01 Jun 2011 09:33:19 GMT

Some sort of citation helper. Something small on the dataset and resource page that would show how to cite.

wwaites: Some related thoughts on this from opb: http://homepages.inf.ed.ac.uk/opb/papers/ssdbm2006.pdf

timclicks: I'm looking at Dataverse for the first time[0]. It seems very popular in the social sciences. I noticed that there is a recommended citation for each dataset. For example, [1] is has this one: "Targeted Input Programme (TIP) 2000-01", http://hdl.handle.net/1902.1/SSC-MWI-TIP2000-01-M1 V1 [Version]"

Implementation

Add a small box at bottom of dataset / resource page (or in sidebar on dataset page) with title "Cite this" with contents like:

%title. %author. Retrieved %date. %site_title.

For resource: %title = %dataset_title. %resource_name.

Could also add export to ref managers (e.g. to bibtex) but that is for later.

#2796: Need a datahub one-pager

ross — Thu, 02 Aug 2012 16:28:36 GMT

A one-pager explaining what the datahub is and with howto/examples for new users. This would make it much easier to explain the value in using the datahub for storing data.

#1136: Move to SessionExtension in vdm

rgrp — Thu, 12 May 2011 14:34:15 GMT

When vdm was created there was no SessionExtension so we use MapperExtension for doing revisioning. Now that SessionExtension? exists we should use it. We can also follow the existing SQLAlchemy recipe: <http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects>

#1135: Changeset model for vdm

rgrp — Thu, 12 May 2011 14:19:46 GMT

Move to Changeset model for vdm.

A changeset model is like an Audit-Log model in which we just record Changesets with Change-Objects rather than have Revision-Objects for each Object that is revisioned.

This change would also incorporate significant simplication of vdm.

#1137: Remove need for statefulness in vdm

rgrp — Thu, 12 May 2011 14:47:08 GMT

Statefulness, especially statefulness for relation (esp m2m) is cause of most of the complexity in vdm. It is required because, atm, revision objects have FKs to continuity objects.

This ticket proposes the following changes:

NB: this could be limited just to case of join tables (leaving state stuff on other tables)

Remove FKs from revision to continuity (or allow for them to be nullable).
We could just limit this to m2m stuff
Delete of an object leads to:
Deletion of continuity object
Adding an entry in revision table with state set to deleted (we retain state on revision table)
If this is done we will no longer need to worry about filtering on state on relationships as join table will only contain "active" relationships.

If we do this on all tables we remove need for any state awareness in client (e.g. no need to filter tables on active state).

The only disadvantage of this change is that undeletion becomes more problematic (we have to recreate some continuity objects).

#1689: List deleted datasets in API

dread — Fri, 20 Jan 2012 12:23:34 GMT

The admin extension allows deleted datasets to be viewed, but there is no equivalent in the API. The package_list API call filters to just 'state=active' datasets.

'state' could be a parameter on the package_list call

Easy to call and implement
But getting permissions right for the different state values may not be easy - maybe a separate logic function is better?

#1447: disk space leakage

dread — Mon, 07 Nov 2011 11:54:03 GMT

Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it.

One problem in the past has been file handles running out when creating lots of tiny files in the data directory.

Another problem has been several enourmous backups being created every day - pdeu on eu25.

#989: Extending the model from plugins

pudo — Mon, 14 Feb 2011 13:22:04 GMT

We need to support extending the model from plugins. This could involve:

Adding a plugin hook to extend the mapper
Adding an upgrade hook for plugin schema migrations
Documenting how this is to be done
Find a way to avoid conflicts

#2331: Search should AND terms not OR terms

rgrp — Sat, 28 Apr 2012 18:24:45 GMT

Appears current default search in CKAN ORs terms rather than ANDing them (i.e. adding more terms increasing number of items found rather than reducing it).

Not sure when this crept in or if it has been there for a long time.

#2790: logic.action.user_show is slow

toby — Wed, 01 Aug 2012 20:21:23 GMT

This is a very slow call it would benefit from the sort of speed-ups that package-search received

for me locally this is taking 6 seconds for rufus using the datahub data I have. I think a lot of this is the dataset retrival/dictization

can we just grab json blobs from solr?

also is it possible to specify a sort order/paging?

I've put this as a demo-theme ticket as it is an big issue on the demo we are at 25 second page loads - which i can get down to about 8.8 secs so this is the main pain point now

#84: Revert support on versioned objects

rgrp — Thu, 23 Jul 2009 08:59:03 GMT

Basic revert in the classic wiki form is already support by purging a Revision. However may wish to support:

Cases where multiple objects changed in a revision but only want to revert 1 (low priority)
Want to revert but have reversion as a new revision of that object.
Seems low priority at present.

Cost: ?

#2203: Data Viewer support for PDF

rgrp — Sun, 04 Mar 2012 11:24:53 GMT

Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf
propose we use pdf.js - https://github.com/mozilla/pdf.js
We could also try iframing for browsers with native support but this seems more complex

#2294: Unauthorized action on Data API gets 302 rather than 401 or similar

rgrp — Sat, 14 Apr 2012 19:35:04 GMT

Well known issue that 401 *in API* end up with 302 redirect to login page (which makes no sense and is very confusing for clients).

Fix this by ensuring that error messsages on API get passed through correctly.

Minor because affect is annoying but not terminal

#2530: DataHub purge fails on some revisions

rgrp — Fri, 15 Jun 2012 08:54:58 GMT

See http://datahub.io/ckan-admin/trash and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like:

Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (IntegrityError?) update or delete on table "revision" violates foreign key constraint "group_extra_revision_revision_id_fkey" on table "group_extra_revision" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table "group_extra_revision". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'}

#2674: Data preview not loading on s031

shevski — Mon, 16 Jul 2012 10:38:22 GMT

Not loading for all resources as far as I can tell; e.g. http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360

Fine on demo.ckan.org

#2877: Bugs with datastore v2

rgrp — Sat, 18 Aug 2012 00:33:22 GMT

In progress

~~[major] q does not seem to work reliably. e.g. using the setup from this gist https://gist.github.com/1930806 and doing a ?q=DE yields no results (does not work with "q=de" either)~~
~~q=second does work ...~~
[major] q does not work with 2 values (see below)
Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT
Also weird fact that limit is returned but as as as string - should it not be an integer
Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...)
Multiple query values

Try a query such as: "second UK" and you will get 500 error:

http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&q=second%20UK

ProgrammingError: (ProgrammingError) syntax error in tsquery: "second UK" 'select "_id", "id", "date", "x", "y", "z", "country", "title", "lat", "lon", count(*) over() as "_full_count"\n from "4f1299ab-a100-4e5f-ba81-e6d234a2f3bd" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',)
Suggestions

Filter support: should think in more detail about this (may want to follow recline style)

Simple filters in query parameters would be nice too ...

#3027: solr for 2.0

kindly — Tue, 04 Dec 2012 02:22:09 GMT

change mm support solr 3 and 4 add *_date field

#979: Edit Resource extras in the API

dread — Fri, 11 Feb 2011 13:09:37 GMT

Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API.

#1077: Move to simpler vdm system

rgrp — Fri, 08 Apr 2011 23:14:24 GMT

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

The main challenge with this change is schema and data migration
Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset for revtype in [PackageRevision, ...]: for pkgrev in package_revision: changeset = lookupchangeset(package_revision) ChangeObject(cset, (table, id), dictize(pkgrev))
Question:

does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.
Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

Easier support for pending state and similar behaviour
No need to introduce new tables (and hence migrations) when making something revisioned (or not).
Disadvantages

Migration is required
More difficult to query revision history.
Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
Performance (?)
Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table
Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

#1240: [super] API v4

rgrp — Sun, 24 Jul 2011 16:41:00 GMT

(Just creating this ticket as somewhere to keep notes)

Decide on REST api versus action API
Do we want to support both?
Tidying
Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
extras merged into normal field list in package
Get rid of /rest/ so just have api/v1/package
Get rid of separation of search api from 'rest' api
Propose that GET on REST index is search e.g. /package/?q=...
This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
Resource read/write in API (separate from package)
Does this need authorization work?
user/account API - read/write
Remove autocomplete -- can just use search
Do not worry about backwards compat as should only be used in our js (if others using it too bad!)

#1255: Drupal consistancy checks.

kindly — Mon, 01 Aug 2011 17:32:48 GMT

Make a robust way to make sure the drupal database is consistent with the ckan data.

#1341: Delete spam users from ckan

kindly — Wed, 14 Sep 2011 10:10:34 GMT

Spam users where added to thedatahub and we need to clean them.

#1652: How we intergrate with Drupal Multiligual?

ross — Mon, 16 Jan 2012 10:31:34 GMT

Analysis of how/where we can integrate with Drupal Multilingual
Is it likely to access CKAN via /en/data/... or /data/en/... (not sure, probably former) and we need to make sure that it ends up at /en/...
Check with Ian for other issues.

#2729: searching for tags:[tag] works but tag:[tag] doesn't

shevski — Sun, 22 Jul 2012 09:58:29 GMT

which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work!

http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc

#2524: If there are no translation files for selected language fall back to default lang.

kindly — Wed, 13 Jun 2012 17:37:28 GMT

If a user selects a language there are no mo files for then an error is raised. Revert to default language instead.

#2291: DataStorer incorrectly attempts to store JSON in DataStore

rgrp — Sat, 14 Apr 2012 19:19:07 GMT

See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

#2334: Improved data import

rgrp — Sat, 28 Apr 2012 22:21:02 GMT

Placeholder for thinking about this ...

Allow users to configure info for CSV import (e.g. separators, quotes etc)

#2406: Errors in source install doc

markw — Mon, 21 May 2012 10:11:33 GMT

There are a few errors or unclarities in the source install doc - see

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html

#2415: Confusing message on attempting to register new account with upper case

markw — Wed, 23 May 2012 13:19:49 GMT

On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

"The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.

#2426: Get rid of CKAN's formalchemy dependency

seanh — Fri, 25 May 2012 11:47:42 GMT

#2427: Reduce cruft installed into CKAN's virtualenv due to Pylons

seanh — Fri, 25 May 2012 11:48:27 GMT

#2473: Make datstorer store field ordering in _meta field

kindly — Tue, 29 May 2012 12:11:32 GMT

Recline views should have a default table order, being the same as the csv that was imported.

#2708: limit extra data for package/group show

toby — Wed, 18 Jul 2012 14:40:20 GMT

contextpackage_limits? = { 'tags': 5, <- get first 5

'extras': 0, <- get all

}

only get what you ask for have to be explicit

contextgroup_limits? = {} only main item

start with datasets/groups expand if we like it

#2548: Object ownership for groups/package

ross — Mon, 18 Jun 2012 11:02:11 GMT

Requirements

We need to be able to easily determine who the owner of a dataset or group is. Datasets and Groups should have an Owner, who may change over time but is a specific user within the CKAN instance. It should be easy for CKAN components to determine the user and for the initial version we should ignore the can of worms labelled 'ownership transfer'.

At this point migration is likely to be the biggest issue, and would suggest that it is acceptable that the last user to edit a dataset be set as the current owner.

More tickets should arise as a result of this work where we may be able to optimise some queries to use the new feature.

Interface

None

User Stories

None

Tasks

[ ] Analysis/Clarification?

[ ] Tests

[ ] Migration

[ ] Code/Schema? changes

[ ] Documentation

Estimates

#2641: Adapt spatial widgets to new theme

amercader — Mon, 09 Jul 2012 15:02:27 GMT

Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now

#2446: Create demo dataset history/comparison page

aron.carroll — Mon, 28 May 2012 13:12:18 GMT

This is a low priority for the demo site.

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62414120/comments

Implementation:

http://s031.okserver.org:2375/dataset/history/adur_district_spending

#2457: Create demo tags list page

aron.carroll — Mon, 28 May 2012 13:28:55 GMT

This includes the tag page as well for now.

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments

Implementation:

http://s031.okserver.org:2375/en/tag

#2451: Create demo user list page

aron.carroll — Mon, 28 May 2012 13:20:13 GMT

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

http://s031.okserver.org:2375/user

#2785: Allow resources to be re-ordered

aron.carroll — Wed, 01 Aug 2012 10:22:03 GMT

Not sure where this functionality should be added, possibly in one of the sidebar widgets when editing a resource?

Ira, what are your thoughts?

#3021: Logout doesn't work without JS

johnmartin — Tue, 20 Nov 2012 11:19:10 GMT

Essentially, the functionality should be as follows:

Add logout link that has .js-hide attached to it within the header that isn't hidden within a dropdown
See http://plus.google.com/ (when logged in) with and without JS to see an example of the actual sign-out working without JS

#2917: Organization admins can delete themselves

ross — Wed, 12 Sep 2012 12:10:43 GMT

Organization administrators can delete themselves from the user management pages. We should disable this on the front-end as well as the back-end.

#3016: CKAN 2.0 template tweaks

johnmartin — Tue, 13 Nov 2012 13:30:17 GMT

Just a ticket to keep track of a few suggested template changes.

#3018: Load more in activity streams

johnmartin — Wed, 14 Nov 2012 13:34:11 GMT

Activity streams should be able to load more than 15 items within them. Suggest the default amount of loading is around 30 and then click to load more.

#2697: create dataset validation

shevski — Wed, 18 Jul 2012 14:00:35 GMT

Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check)

#3011: Recline fixes and updates for CKAN 2.0

johnglover — Wed, 07 Nov 2012 11:40:55 GMT

Fixes and updates to Recline / Datapreview for CKAN 2.0.

#3017: New stable branch for the demo server

johnglover — Tue, 13 Nov 2012 15:04:48 GMT

#1584: QA report improvements - 2.5d

johnglover — Wed, 21 Dec 2011 09:23:53 GMT

Super: #1594

qa/{username}
qa/{groupname}
paginate QA results
search / filter QA results
~~list organisation report by default, but can disable via config option~~ (done)
UX tidy up of report pages - hide border if no sidebar, etc

#1588: QA - Give SPARQL endpoints a 4 star rating

johnglover — Wed, 21 Dec 2011 15:13:25 GMT

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Besides considering the media type of resources, it would also make sense to check for the presence of a SPARQL endpoint. SPARQL endpoints are recorded for more than 300 datasets on the Data Hub using the pseudo-type "api/sparql". A few more are recorded with the format "SPARQL". I suggest that datasets with such resources should also be considered for the fourth star.

#1589: QA - Give 5 star rating to datasets with link metadata

johnglover — Wed, 21 Dec 2011 15:16:40 GMT

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Regarding the fifth star (is the dataset linked to others?). This cannot be automatically determined just by looking at the format. It either requires inspection of the actual data, or information about links in the metadata. As you're probably aware, we've established conventions for recording information on data links in CKAN [1], as part of the work of the lodcloud group on the Data Hub. Link information is captured for hundreds of datasets. I would claim that we have the majority of four-star datasets covered there, and hence you can determine if they should get the fifth star by checking for the presence of a links:xxx field.

#2733: Datastore logic functions

johnglover — Mon, 23 Jul 2012 15:54:46 GMT

Where does the data go?

In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url.

The user should have rights to create tables.

Whats the api like?

We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern.

What are the initial logic functions?

datastore_create
datastore_delete
datastore_show
What is the JSON input format for datastore_create

To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id.

{ resource_id: resource_id # the data is going to be stored against. fields: a list of dictionaries of fields/columns and their extra metadata. records: a list of dictionaries of the data eg [{"dob": "2005", "some_stuff": ['a', b']}, ..] }
The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
rows: can be empty so that you can just set the fields
fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.
eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though.

Any error results in total failure!! For now pass back the actual error.
Should be transactional
What json does datastore_delete take?

{ resource_id: resource_id # the data is going to be deleted. filters: dictionary of matching conditions to delete e.g {'key1': 'a. 'key2': 'b'} this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ". No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table. }
What json does datastore_search take?

{ resource_id: resource_id # the data is going to be selected. filters : dictionary of matching conditions to select e.g {'key1': 'a. 'key2': 'b'} this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' " q: full text query limit: limit the amount of rows to size default 100 offset: offset the amount of rows fields: list of fields return in that order, defaults (empty or not present) to all fields in fields order. sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc" }
Some free code: https://gist.github.com/3163864

What json does datastore_search return?

{ fields: same type as datastore_create accepts (i.e. with metadata) offset: The same offset that was supplied in datastore_show limit: The original limit filters: The filters that were applied in data_show total: # total matching records without size or offset records: [same as data_create] # list of matching results }
On error will return:

{ __error__: … sql error … }
What types are allowed?

Aim to support as many postgres/postgis types that have string representations.

http://www.postgresql.org/docs/9.1/static/datatype.html

http://www.postgresql.org/docs/9.1/static/sql-createdomain.html

IDs

Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

Other Features

Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc.

#1062: Data preview encoding error

sebbacon — Tue, 29 Mar 2011 10:55:43 GMT

The preview of "Species Misc Turtle Download" at http://ckan.net/package/taxonconcept results in the following error:

Unable to Preview - Had an error from dataproxy: Data Transformation Error (Data transformation failed. Reason: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte

#1259: "Add a row" for Extras on Package form

pudo — Tue, 02 Aug 2011 16:31:33 GMT

The default package form offers 4 empty extras fields. Like the resource section, it should have an "add more" button to add another row.

#1336: License fudge

dread — Tue, 13 Sep 2011 11:07:37 GMT

cset:4b59ab34137d ckan/logic/action/get.py:

- isopen = model.Package.get_license_register()[license_id].isopen() - result_dict['isopen'] = isopen + try: + isopen = model.Package.get_license_register()[license_id].isopen() + result_dict['isopen'] = isopen + except KeyError: + # TODO: create a log message this error? + result_dict['isopen'] = False
This change hides problems with the license server and returns potentially incorrect values for openness.

This has been noted as 'temporary fix' but seems to be forgotten about, since it has been merged to default and gone into release 1.4.3.

I suggest the licenses are cached (I thought this was already the case when CKAN first requests them after start-up?). I suggest failure would return 503.

#1393: Don't skip search tests

dread — Thu, 13 Oct 2011 11:30:53 GMT

Now we don't use postgres search, all the tests involving search now don't need to be skipped when running on sqlite. Should help coders spot earlier if these tests break.

#1203: Moderated edits: html code shows as "changed" although it is not

rolf — Tue, 28 Jun 2011 15:29:30 GMT

I've installed the Moderated Edits extension (ckanext-moderatededits) and am editing a package imported from IATIregistry.org, with an extra field which contains a bit of HTML.

The editor indicates the field has changed, although the content hasn't (see screenshot). All I can find so far is a minor difference: in the field content, there is a code &#8212 and in the rendered table that is an —

#2829: Archiver fails on 403 http response

toby — Thu, 09 Aug 2012 10:34:44 GMT

Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end

$ paster archiver update -c ../ckan/development.ini 2012-08-09 11:01:37,636 INFO [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources) 2012-08-09 11:01:37,671 INFO [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13 2012-08-09 11:01:37,900 INFO [ckan.lib.base] /api/action/package_show render time 0.043 seconds Traceback (most recent call last): File "/home/toby/okfn/pyenv/bin/paster", line 8, in <module> load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')() File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 104, in run invoke(command, command_name, options, args[1:]) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke exit_code = runner.run(args) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 238, in run result = self.command() File "/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py", line 98, in command response = app.post(api_url + '/package_show', data) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 262, in post expect_errors=expect_errors) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 243, in _gen_request return self.do_request(req, status=status) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 406, in do_request self._check_status(status, res) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 439, in _check_status res.body)) paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show) {"help": "Return the metadata of a dataset (package) and its resources.\n\n :param id: the id or name of the dataset\n :type id: string\n\n :rtype: dictionary\n\n ", "success": false, "error": {"message": "Access denied", "__type": "Authorization Error"}}

#2859: Fix the build

icmurray — Thu, 16 Aug 2012 09:26:57 GMT

The requires files have been removed. These were required by buildkit.

We've lost the information regarding which packages were already in ubuntu repos, which conflicted and which were missing. But perhaps we can just package everything up.

#2405: Improvements to data viewer embed

rgrp — Mon, 21 May 2012 06:59:25 GMT

Decent powered by icon for the Embed for the DataHub?? (make this text customizable and allow html ...)
Current icon and text looks really bad
Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
Data viewer embed url includes all the resource info (not needed -- just url and backend info ...)
Embed extension urls do not respect original domain but instead use site_url (I'm guessing). E.g. datahub.io domain ends up with links to embed on thedatahub.org

#2959: Changing a Group's name through the action api disassociates it from its datasets in the index

icmurray — Thu, 04 Oct 2012 15:18:55 GMT

Repro:

Create a new Group, named "test-group".
Add a dataset to it.
Verify the dataset belongs to the group by visiting the Group's read-page
Update the Group through the action api (group_update), using the uid in the "id" field, and a new name in the "name" field.
Visit the group's read-page. The list of datasets will be empty.
This was an issue when editing a Group through the web interface, which was fixed in [1]. However it only fixes the issue in the group controller.

[1] https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335

#2277: Use the new atom feeds in IATI

icmurray — Mon, 02 Apr 2012 17:10:28 GMT

Atom feeds have been implemented in core in #1593. The IATI-specific implementation can now be removed when IATI move to CKAN >= 1.6.1

#2310: Refactor the search-query construction in feeds.py

icmurray — Mon, 16 Apr 2012 16:01:31 GMT

The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication.

#2298: Add sort-by controls to the search results page

icmurray — Mon, 16 Apr 2012 09:39:22 GMT

Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page.

What should the sortBy controls/widget look like?
Which fields should be exposed?
The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master.

[1] https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c

#2350: Hooks in package controller for validating form data

icmurray — Tue, 01 May 2012 14:16:01 GMT

When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data. Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to PackageController? subclasses to do form validation/processing.

Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override.

NB - form processing/validation *is* different to validating data. For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&Cs.

[ ] Add hooks
[ ] Use in ckanext-example

#2603: Remove deprecated 'fields' parameter from resource_search

icmurray — Tue, 26 Jun 2012 17:10:01 GMT

The fields parameter of resource_search was deprecated when fixing #2438. It can be removed in release 1.9, and the action tidied up as a result.

#2621: Remove the deprecated 'fields' parameter from tag_search and tag_autocomplete

icmurray — Thu, 28 Jun 2012 16:22:49 GMT

This was deprecated in 1.8 as it wasn't accessible via GET requests due to being a dict. See #2439

In a future release of CKAN (probably 1.9) it can be removed.

Internal uses of it were removed in #2439, but there are tests that still use it.

#2679: Change default behaviour of TemplateController.view to 404.

icmurray — Mon, 16 Jul 2012 10:55:33 GMT

The current behaviour of TemplateController?.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file.

Although this may be a feature that some instances want. In general, it leads to:

500s when attempting to access a normal template (eg - http://datahub.io/importer/preview)
A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template).
Solution:

Change the controller to 404
Ensure there's a way for existing ckan instances to override that behaviour should they need it.

#2709: Atom feeds are undocumented

markw — Wed, 18 Jul 2012 15:11:50 GMT

There doesn't seem to be any documentation yet for Atom feeds.

#2784: model dictize sensitive data

icmurray — Wed, 01 Aug 2012 09:51:25 GMT

The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not.

#250: RDF link in Atom feed

dread — Thu, 18 Feb 2010 15:41:35 GMT

Add link to RDF representation of a package in our Atom feed.

#2485: Encourage leak containment by limiting the number of requests a CKAN process serves

nils.toedtmann — Fri, 01 Jun 2012 11:47:09 GMT

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

research ways to limit requests-per-process with the different web servers (see below for a start);
explain these safeguards in the CKAN documentation and encourage users to apply them;
consider the helper script ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of commented out. Or at least let it warn the user to use MaxRequestsPerChild;
Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

Apache:
Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on datahub.io and the ckan farm
(This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
nginx/supervisord: to be researched.

#2550: User types

ross — Mon, 18 Jun 2012 11:06:30 GMT

Requirements

In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub).

Required interface

Once changes have been made to the user schema, for a given user we want to be able to:

determine if they have a paid or a free account, and
get a string name of the type of paid account.
Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin.

User Stories

User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases. For now it is adequate that we can manually control these things through paster commands.

Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all.

As a sysadmin I would like to be able to use a paster command to manually set a user's payment level, or remove it entirely.
As a sysadmin I would like to be able to run a paster command to view a list of users who have a payment plan, grouped by the plan that they have.
As a sysadmin I would like to be able to use the API to change the payment status of a specific user through user_create and user_update. This shouldn't be available to anybody else.
As a user, and only if I have one, I'd like to see my current payment level on my user profile page.

Tasks

[x] Tests

[x] Plugin based migration

[x] Code

[x] Model

[x] API

[x] Documentation

Estimates

#3013: common-error-messages is unreadable

dominik — Sat, 10 Nov 2012 13:18:25 GMT

Since the update of the doc theme, the page became unreadable.

http://docs.ckan.org/en/latest/common-error-messages.html

#2961: Preview plugin endpoint

dominik — Mon, 08 Oct 2012 11:39:27 GMT

Users could write ckan extensions that offer previews for a certain datatype.

Advantages:

We can say that you can add your own previews
Every preview would have to define it's own dependencies
External development (community creates new extensions?)
Questions to answer:

Specification of the interface
Precedence of previews for the same data types
Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible.
As far as I can see, this should be fairly easy to implement by following these instructions: http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface

#1558: Publisher Tools

jilly mathews — Mon, 19 Dec 2011 10:46:13 GMT

Summarise final set of requirements for this and finish development and test. Estimated 10 working days.

#1560: Follow extension

jilly mathews — Mon, 19 Dec 2011 10:51:19 GMT

Estimate 2 days to finish dev and test.

David can you add any info needed here?

#1561: To do extension

jilly mathews — Mon, 19 Dec 2011 10:52:52 GMT

Can we finish this ready for release on data hub and CKAN Hosted.

#1564: Structured Data (Data API)

jilly mathews — Mon, 19 Dec 2011 11:07:14 GMT

Basic websotre exists but this may be not what is described yet.

CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first.

#1567: Finish QA extension

jilly mathews — Mon, 19 Dec 2011 11:19:59 GMT

Requires change to celeryd. Estimated 4 weeks.

#1569: Wordpressser

jilly mathews — Mon, 19 Dec 2011 11:27:36 GMT

How much effort will this be to be ready to use?

#1572: Meta data Harvester

jilly mathews — Mon, 19 Dec 2011 11:35:09 GMT

Need to write custom harvesters for each client. Is it worth having one for data hub?

#1573: Apps and Ideas

jilly mathews — Mon, 19 Dec 2011 11:36:33 GMT

Estimate 2 weeks for someone to finish and test.

#1557: Complete Webstore Preview Extension

jilly mathews — Mon, 19 Dec 2011 10:40:53 GMT

Finish any work out standing on web store preview extension to be able to package and release.

Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted.

#2357: Create build script for front-end resources

aron.carroll — Tue, 01 May 2012 15:29:41 GMT

Should minify and concatenate JavaScript? and CSS as part of #2354

#2493: Demo site 404 needs some love

toby — Wed, 06 Jun 2012 12:34:41 GMT

templates/error_document_template.html

I've converted the 404 page to the new style but we probably want to serve this as a 'full page' ie content is 100% page width not sidebar and primary content

just go to a non-existant url to see

Discusion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63374042/

Implementation:

http://s031.okserver.org:2375/404

#2949: Reenable Data API button on the new theme

amercader — Mon, 01 Oct 2012 16:04:24 GMT

The checks to show or not the button need to be updated for the latest datastore version

#728: CSW Harvesting shall be optimised in respect of reharvesting only records that have changed

johnbywater — Thu, 21 Oct 2010 15:35:40 GMT

Hi Will, this is important again because some CSW servers we use have over 300 documents in. Could you take a look at modifying the filter please?

#794: Investigate reconciling UKLP Publisher and Provider with DGU

johnbywater — Mon, 08 Nov 2010 14:50:11 GMT

This needs more analysis, but the GEMINI2 attribute "metadata point of contact" must be reconciled with the registered publisher (or agent).

This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider.

#1152: True support for generic CSW servers

amercader — Mon, 23 May 2011 09:02:14 GMT

The CSW harvesters implemented at the moment were developed with the DGU project in mind, and they assume all remote CSW servers to implement the Gemini 2 specification. Gemini 2 is the profile defined in the UK for INSPIRE complying metadata, so obviously catalogs from other countries or non-INSPIRE complying ones won't be able to be harvested.

The changes needed to support generic CSW servers (i.e. those implementing the ISO 19139 profile) are:

Handling the validators (right now are hardcoded in the harvester
code). This probably involves issues discussed in the CREP 3 (ticket #1134)

Changes in the model to adapt the specification to ISO 19139
Renaming objects and classes which are now Gemini-centric
List of CSW servers tested:

https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&hl=en_US&authkey=CNu4hsEB#gid=0

#1818: Spatial metadata editor

amercader — Mon, 20 Feb 2012 18:52:21 GMT

Estimate 2.5d

Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset.

We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers? [1], [2].

Things to take into account:

Integration with the form sections.
Allow some degree of configuration (default extent, limit geometry types...)
[1] http://openlayers.org/dev/examples/draw-feature.html [2] http://openlayers.org/dev/examples/modify-feature.html

#3026: Support icons on nav_named_link

amercader — Fri, 30 Nov 2012 18:51:56 GMT

nav_named_link won't include the nice icon even when passing the icon keyword param.

#3023: New methods on IPackageController to provide access to the data_dict

amercader — Thu, 22 Nov 2012 17:00:57 GMT

Extension hooking into the edit and create methods of the IPackageController interface receive the package object. This may not include all the fields that came from the form. The new extension points will pass the validated data_dict so extensions can have access to it

#3025: Add requests to core requirements

amercader — Tue, 27 Nov 2012 21:35:24 GMT

Because yes please

#1815: Reenable Sparql endpoint on publicdata.eu

amercader — Mon, 20 Feb 2012 17:08:43 GMT

Estimate: ?

The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset.

#2382: Investigate options for basic geocoding

amercader — Mon, 14 May 2012 17:39:20 GMT

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)

#2407: 'Access denied' message is unhelpful

markw — Mon, 21 May 2012 15:01:40 GMT

This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied':

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html

The problem was he didn't have access to the group. A message like 'You don't have access to the group "LODcloud"' would have been clearer.

#2788: Speed improvements on creating/updating and indexing

amercader — Wed, 01 Aug 2012 14:56:32 GMT

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

#1562: Finish Geo Spatial

jilly mathews — Mon, 19 Dec 2011 10:56:33 GMT

Estimated 4 weeks of Adria's time. I guess this will need to be broken down into more tickets. This feature is being requested by a number of potential customers and we have some ideas of requirements between Rufus and Jilly for this. This is the most popular new feature we talk about to new clients.

#2582: Do not hide notes / readme on dataset pages

rgrp — Thu, 21 Jun 2012 20:59:00 GMT

Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?)

Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket.

#1198: Publisher hierarchy

dread — Thu, 23 Jun 2011 09:16:32 GMT

'Publisher' entities in the model. They are hierarchical.

'User-Publisher' connections with one or more roles (e.g. drafter, moderator).

Authorization settings can control who can set what values in a 'published by' type field.

Publishers and User-Publishers available to read in the API.

Future tickets will provide:

API to write Publishers and User-Publishers
UI to edit Publishers and User-Publishers
(This feature deprecates authorization groups)

#1466: Need to support https login for multiple instances as part of the CKAN package install

thejimmyg — Tue, 15 Nov 2011 16:52:58 GMT

#1832: dataset purge API

dread — Fri, 24 Feb 2012 10:02:25 GMT

Purging datasets (deleting them fully, not just changing the state to 'deleted') is important for users testing dataset creation over the API on a test CKAN instance.

Without this, they need to resort to more difficult methods such as:

cleaning and reloading the database
setting the test datasets to state 'deleted' and also appending a suffix '_00' and incrementing the number until there is no clash of names.
Requested for NHSIC.

Implementation

This could slot into the Action API.
Of course we would need to ensure the user's had been given the specific right to purge.
I suggest we log the full details of the dataset being purged.

#2686: enabling datastore & data API breaks recline

shevski — Tue, 17 Jul 2012 09:06:51 GMT

First I noticed that the gold prices dataset preview was not displaying & has data API enabled Secondly I tried enabling datastore for http://datahub.io/dataset/adur_district_spending/resource/281dffa6-ea9b-4446-be41-05dced06591f and after I saved the preview no longer worked. Unticking the datastore & data api checkbox brought it back

Is this a known issue?

#1244: Notes field carriage-returns converted to CRLF

dread — Tue, 26 Jul 2011 14:10:56 GMT

When you edit a package in the web form, if the notes field had \n as the End Of Line symbol, it gets lost when you preview or save the package, and the notes field is displayed all on one line.

This can be seen when editing annakarenina (as created by 'paster create-test-data'). The diff shows for example:

- Some test notes + Some test notes ? +
but it would more clearly be shown as:

- Some test notes\n + Some test notes ? ++
This is a significant problem with DGU, since a lot comes in via the API.

It's not clear what we should do about it. We could standardise on \n or \r\n when the form submission comes in. Do different browsers on different platforms do different things with EOLs?

Analysis

Displaying the package: the Markdown processor respects both EOLs when displaying the field, putting each line in a <p> tag.

Creating the package edit form: placed into <textfield>.

Browser displaying package edit form: <textfield> displays \n and \r\n as EOL. But \n\n gets compressed to one EOL. But on submission, both are returned as \r\n.

Receiving the edited package: Somewhere along the line the EOL gets converted to \n\n.

#1322: Action API improvements

dread — Thu, 08 Sep 2011 09:39:09 GMT

Focusing on improving Action API as the v3 API:

have an optional parameter of the data_dict called "options". Options would contain items that would get passed into the context. e.g. {"options" : {"ref_package_by": "id"}}.
instead of using API version to change the way packages are referenced, use the ref_package_by.
All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'.
Action API is v3 of api, replacement for v1 & v2. Default for most urls is still v1, but if url is /api/action then default to v3.
Next steps:

Add search API (package, resource,
Add Util API
Clarify JSONP still applies
Add doc strings, clarifying parameters

#1460: Improve extensions documentation

rgrp — Sat, 12 Nov 2011 16:15:23 GMT

Current extensions documentation needs some work: http://docs.ckan.org/en/latest/plugins.html

Queue extension section may now be out of date (?)
Think about how it integrates with https://github.com/okfn/ckanext-example (especially tutorial and example extension)
Document all plugin points (auto-extract from CKAN source??)

#2635: Non-destructive SOLR reindex

dread — Mon, 09 Jul 2012 10:23:14 GMT

You can't run the search-index reindex on a live server because it will give us bad results for 2 to 3 hours while it runs. Can there be an option that doesn't delete the entire index at the start?

Instead it could just delete any items that don't exist any more, then delete them and regenerate them one by one. So the total number of datasets doesn't change much.

#3019: Cannot delete dataset extras

seanh — Wed, 14 Nov 2012 18:44:38 GMT

Deleting extras in the web interface is broken

#256: Package relationships - 3. Edit in WUI

dread — Tue, 23 Feb 2010 12:36:01 GMT

WUI:

Editable as part of package or separately? (e.g. like authz)
Do we normalize to only one type name of the pair?
Do we allow create of relationship from both ends (e.g. only from dependency to dependent or either way?)

#1134: CREP0003: Description and Configuration of Harvesters

amercader — Wed, 11 May 2011 10:14:28 GMT

Proposer: Adrià Mercader

Abstract

The new harvester interface allows to create harvesters for different sources, but right now harvesters don't have many ways to describe and configure themselves. We need a way of allowing them to:

Expose their type and other details so they can be used internally and on the UI.
Define configuration settings for particular harvester instances.
The Problem

Harvester description

The current UI for adding and editing harvest sources is the same used in ckanext-dgu, and thus the 3 harvester types used in DGU to harvest various GEMINI realted sources are hardcoded in the form. The form will be migrated to a DGU-independent one, so we need the harvesters to provide all the necessary data. There is a current get_type method that returns the harvester type, but for make it compatible with the DGU forms, it returns a machine-readable string (e.g. "CSW Server"), making it error prone.

Arbitrary configuration

In the current implementation, when the harvest process is started, ckanext-harvest looks for all the available plugins that implement the IHarvester interface and calls the appropiate methods for the current stage (gather_stage,fetch_stage,import_stage). At these stages, harvesters have no way of applying arbitrary configuration options, so all harvesters of the same type behave on the same way. For instance, the CKAN harvester needs a way to define the API version to use when harvesting remote instances (Right now, the version 2 is hardcoded on the code).

Specification

Harvester description

Harvesters will need to provide the following information so the UI form can be built:

name: machine-readable name (e.g. "waf"). This will be the value stored in the database, and the one used by ckanext-harvest to call the appropiate harvester.
title: human-readable name (e.g. "Web Accessible Folder (WAF)"). This will appear in the form's select box.
description: a description of what the harvester does (e.g. "A Web Accessible Folder (WAF) displaying a list of GEMINI 2.1 documents"). This will appear on the form as a guidance to the user.
The way to provide it will be an info method that all harvesters must implement, which will return a dictionary with the previous elements:

{ 'name': 'csw', 'title': 'CSW Server', 'description': 'A server that implements OGC's Catalog Service for the Web (CSW) standard' }
Arbitrary configuration

As different harvesters will have very different needs, we need to provide a way to persist arbitrary configuration flags for each harvest source. The more flexible way given the current architecture in my opinion would be to store the configuration options as a JSON encoded object as a property of the harvest source (There already is an unused DB field called config in the database) (Maybe using JsonType??).

This will mean adding an extra field in the harvest source form to allow entering the configuration. This could be just a simple text field where users enter the JSON encoded object or a more clever mechanism (i.e an "Add a configuration flag" link that adds two new text fields for the key and value for each flag, and a mechanism to later build the JSON object). In any case, this should probably be hidden in an "Advance options" section.

Why do it this way

Harvester description

The info method would provide a single point to get all the information related to the harvester, and future properties could be added to the dictionary returned without having to modify the interface.

Arbitrary configuration

There is an already existing config field in the database, so we won't need to change the model. Harvesters could access the config object at any of the stages. Of course they could provide default values in their implementations so users don't need to enter them everytime.

Implementation plan

Deliverables

Risks and mitigations

The highest risk on the harvesters info method side is that harvester implementation don't offer one of the necessary properties (namely name and title). This could fire a warning when showing the UI form or using the CLI.

Participants

Adrià Mercader to do it.

Progress

None yet.

#1257: Anti-Spam tools

dread — Tue, 02 Aug 2011 11:12:45 GMT

We are getting more and more spam on ckan.net and we need to improve our strategy of combating it. It is bad because google ranks who we link to (which we do want for legitimate links), and our front page contains the latest package edits, so spam is immediately very visible.

Spam:

Package creation
Package edit
User creation
Group creation
Systems to consider:

Automatic
captcha
bayesian scoring
Sysadmin
a tool to quickly analyse and remove packages, edits and users
Helpful users
button to click indicating spam, leading to any/all of:
add TODO item (which emails package admin and sysadmins)
quarantine the package/user so it doesn't show up on the front page and a warning is shown when you view it
General thoughts:

We should learn from the wikipedia tools, as they are the experts
This is a serious problem that is only going to get worse and is a threat to the whole site.

#1262: Enforce "create-user" permission

pudo — Wed, 03 Aug 2011 12:41:36 GMT

This does not seem to have any implications at the moment, it should lock down registration and remove all related links.

#1286: Remove remaining formalchemy stuff

rgrp — Tue, 23 Aug 2011 16:29:56 GMT

Stuff I've spotted:

forms/*
template/group/edit_form.html
template/package/edit_form.html
This can go once new DGU form is in.

#1352: Use logic functions instead of as_dict when indexing entities

amercader — Wed, 21 Sep 2011 14:29:57 GMT

The current search implementation uses the output of the the as_dict method of the domain Package object to update the index

https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48

It also uses package_to_api1 in the SynchronousSearch? plugin:

https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93

This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model).

The search should use the logic function to get the package properties:

get_action('package_show')(context,data_dict)

#1439: Action API discoverablility

dread — Tue, 01 Nov 2011 15:39:30 GMT

A good service API needs to be discoverable, so you are not always having to refer to the documentation html.

Maybe /api/action should return a list of actions available? (Currently this returns a 404.)

It would be nice to sort these into get/create/update/delete.
#1438 Parameters for each of the actions must be discoverable too
/api/action/{action_name} should also return the help text / parameters allowable. (Currently this returns 400 error)

#1489: Updating example theme/extension

dread — Thu, 24 Nov 2011 12:32:00 GMT

ckanext-example needs updating for CKAN 1.5:

theme changes
new forms
About: 'ckanext-exampletheme' was created in Spring 2011 as an example CKAN extension that showed how to customise the look & operation of CKAN. This moved to github and renamed 'ckanext-example'.

#1534: Change revisions to record userid rather than username

rgrp — Wed, 07 Dec 2011 17:26:30 GMT

The use of username is problematic because username's can change.

Change all revision creation code to use user id (simplest is to change c.author field in lib/base.py (?))
(?) Add a field ipaddr for ip address of anonymous users? (or just keep putting this in author field on Revision and then acception that those won't match when we do a look up against user table)
Change user view page to look up against user id rather than name
Perform migration on existing Revision objects
Match should probably be against both openid and username when searching Revisions' author field (especially true on CKAN where some people have already changed their username from being their openid)

#1542: Buttons to purge spam datasets and groups

dread — Thu, 08 Dec 2011 17:22:10 GMT

A sysadmin should be able to easily examine a suspect group or package, determine if it was created by a spammer (as opposed to being a legitimate object that has been graffitied by a spammer) and purge it.

The existing two-stage revision delete is currently unreliable and perhaps too laborious.

Olav and Richard have needs along this line.

#1661: Wrong Routes version installed by CKAN package

dread — Mon, 16 Jan 2012 12:47:41 GMT

Jaakko Louhio reported that the wrong Routes version got installed during CKAN's package install.

He is using Ubuntu 10.04 and he believes it install Routes 1.12.3 instead of what we use which is 1.11.

#1827: 'Register' link should be hidden if you not allowed to register

dread — Wed, 22 Feb 2012 15:35:39 GMT

I have just deny visitors the create-user permission: sudo -u ckanstd paster --plugin=ckan roles deny reader create-user -c /etc/ckan/std/std.ini sudo -u ckanstd paster --plugin=ckan roles deny anon_editor create-user -c /etc/ckan/std/std.ini and after restarting, the register link is *not* hidden, but now when you access the register page, it shows you this message "Unauthorized to create a user" (when not logged in). But anyway that is an improvement.

#1235: [super] Search Improvements

thejimmyg — Wed, 20 Jul 2011 17:24:01 GMT

Child tickets:

#234 UI Review - Autocomplete package names & tags in search
#193 Searching by time-related field
#191 Searching by modification date
#905 Unable to search with accented characters in package names
#906 Ability to search without accents for accented words
#924 Search box has no search button
Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two.

#1366: Search inside extra fields

dread — Thu, 29 Sep 2011 10:09:52 GMT

SOLR search doesn't support searching for part of an extra field, but it does for other fields.

i.e. title="One Two Three" matches q=one AND q=title:one and geographic_coverage="England Scotland" matches q=England BUT NOT q=geographic_coverage:England

This problem emerged when we went to SOLR in #1275 (CKAN 1.5a). Tests were skipped.

This is could be a problem for DGU and maybe elsewhere.

#1438: Action API - parameter discovery/checking

dread — Tue, 01 Nov 2011 15:38:10 GMT

Many actions in the Action API require parameters. What params are needed should be listed and checked. Because currently, if you get them wrong you simply get a useless 500 error.

Currently they are listed in the docs, extracted from the code manually.

So you could GET /action/api/package_list to receive not only the help text, but a list of arguments.

And if you send an extra or missing argument then an intelligent error message can be returned.

implementation

How about some sort of decorator on the action function:

@logic_params(id, offset, limit) def get_package_list(context, data_dict): ...
This would do the param checking, and is there a way to extract these params from the function? Or do a registration of the logic function?

I'd certainly like to keep the list of the list of params for the function with the function, for ease of reading the code.

Another good thing would be to pass in the params named as themselves, rather than having them contained in the data_dict.

#2963: Timeout on tag pages with lots of datasets

amercader — Mon, 15 Oct 2012 10:13:19 GMT

e.g. http://thedatahub.org/tag/lod

Tags with less datasets work fine (e.g. http://thedatahub.org/tag/railways)

#2887: "Welcome to CKAN!" on front page is untranslatable

seanh — Wed, 22 Aug 2012 09:17:59 GMT

ckan/templates/home/index.html contains:

<h1 class="page_heading">Welcome to ${g.site_title}!</h1>

which means that just "Welcome to" ends up in the ckan.po files for translation. This makes the phrase untranslatable in languages that have a different word ordering than English where the site title needs to be somewhere other than at the end of the sentence.

#2745: Password reset returns an exception if the key parameter is missing

amercader — Tue, 24 Jul 2012 16:05:31 GMT

Instead of showing a notice, the password reset page throws an exception if the key parameter is missing:

Module ckan.controllers.user:329 in perform_reset c.reset_key = request.params.get('key') if not mailer.verify_reset_link(user_obj, c.reset_key): h.flash_error(_('Invalid reset key. Please try again.')) abort(403) if not mailer.verify_reset_link(user_obj, c.reset_key): Module ckan.lib.mailer:100 in verify_reset_link if not user.reset_key or len(user.reset_key) < 5: return False return key.strip() == user.reset_key return key.strip() == user.reset_key AttributeError: 'NoneType' object has no attribute 'strip'
Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8

instead of:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8

#1328: Unicode & paster commands

minspamboks@… — Mon, 12 Sep 2011 10:25:10 GMT

A possible bug in CKAN when I tried deleting users using "paster --plugin=ckan user delete" command.

To reproduce the bug do the following:

Create a user with an ID (which in my case was a user's full name)
that contains non-unicode caracters like Norwegian "æ", "ø", or "å".

Make sure that you can see something like the example below:
(pyenv) rm@mycomputer:$ paster --plugin=ckan user Users: name=Rustæm

Then try deleting the user with following command:
(pyenv) rm@mycomputer:$ paster --plugin=ckan user delete "Rustæm"

You should now get a python encoding error. I know that this is quite rare case, but in our case it caused some trouble. Could you guys have a look at this bug?

CKAN ver. 1.3.3.

#1509: Mis-dated old revisions of datasets

dread — Mon, 05 Dec 2011 13:06:38 GMT

e.g. http://thedatahub.org/dataset/osm%402011-07-12T12%3A16%3A47.590358 gives:

This is the current revision of this dataset, as edited at 2011-07-12 12:16.
but it should say the date 2011-07-12 (which is the data being displayed).

The problem is looking at PackageRevision?, when it might be a tag or extra.

#2673: simplify set of options for resources

rgrp — Sat, 14 Jul 2012 21:15:59 GMT

Far too many resource options. Lets restrict back to data file and API. Visualizations etc can either get linked in description or in the Related items.

#2719: Feeds controller does not catch NotAuthorized exception

dread — Sat, 21 Jul 2012 12:14:23 GMT

Results in bad user experience and WebApp? errors emailed out. Seen in 1.7.1

#2947: Redirect to the resource page from /resource/{res-id}

dominik — Mon, 01 Oct 2012 13:15:25 GMT

Use story: As a user of the datasore I want to be able to put the resource id without the dataset name in a url and see the resource page.

Proposed redirect:

/resource/{res-id} -> dataset/foo-bar/resource/{res-id}

#2831: Create a limited subset of markdown that's supported

aron.carroll — Thu, 09 Aug 2012 11:41:07 GMT

Allowing people to use the full range of markdown results in extremely messy output across the site. I'd suggest limiting support to only a subset of common use cases.

Allow all inline elements, this allows bold, italic, code and links.
Allow lists.
Disallow

Horizontal Rules
Headings
Block quote and code (this may turn out to be useful and so could be included)
This way you get markdowns paragraph handling and a few inline styles without breaking the entire layout of the page.

Here's the full syntax if anyone is interested http://daringfireball.net/projects/markdown/syntax

#2951: Paster command for building css from less

icmurray — Mon, 01 Oct 2012 19:12:20 GMT

With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is:

the less files should be compiled to css (main.css, not just main.debug.css)
This ticket is to provide a paster command which will compile the .less into the main.css file. The idea of the paster command is that it will be run by developers, and they will check in the resulting .css files. It will also provide a convenient mechanism for production scenarios with node.js installed on the production server, as they will be able to compile the .less there as well.

## Background

The .css files that need to be served are built using less. The css files that are generated *are* checked-in to the repo; but they are a build artifact. So the general workflow is:

commit changes to the less files
build the main.css file and commit
This paster command slots into the above workflow for convenience.

## Notes

doc/frontend-development.rst
bin/less
to run this paster command, Node will be required. So that dependency should be checked.

## The paster command

This is the proposed behaviour of the paster command:

ensure custom.less does not exist
for each colour in{fuchsia,green,maroon,red}:
generate a custom.less file for $colour. There's a paster command that does this already: `paster color <color name>.
generate the css from the less files. ie - the equivelant of running bin/less --production.
this will generate a main.css file, which should be renamed to $color.css.
ensure custom.less does not exist
generate the css from the less files. This will create a main.css with the default colour scheme.
call the paster command that minifies css and js files. (This command does not exist yet, see #2950)

#2996: Fix behaviour of View Profile button in user popover when already on user's profile

seanh — Wed, 17 Oct 2012 14:30:38 GMT

The user popovers that appear when you hover the mouse over a user in an activity stream include a View Profile button that takes you to that user's profile page. If you are already on that user's profile page then it simply reloads the same page, but it sends you to the Datasets tab if you were on another tab.

Possible solutions:

Don't show button if already on user's page?

Button just makes popover disappear, if already on user's page?

Button does reload the page, but reloads the same tab (datasets, followers, activity stream) that you were on.

Marking this low priority

#924: Search box has no search button

dread — Mon, 24 Jan 2011 11:12:13 GMT

The search box at the top-right of CKAN's page doesn't have a 'go' button. I feel that a larger percentage of users expect a 'go' or 'search' button on the right-hand side of the box to press to start searching. Techies tend to know the keyboard shortcut of pressing 'carriage-return' but it might be better to follow standard practise on this.

Examples with 'search' button: Internet Explorer, Firefox, Google, Amazon, trac Examples without: ?

#1288: Package edit/creation can't include 'relationships' field

dread — Wed, 24 Aug 2011 16:34:55 GMT

When you create or edit a package (via the API), you aren't able to specify the relationships it has. (If you do you get 409 {"__junk": ["The input field __junk was not expected."]} )

The normal way to create relationships is via /api/rest/relationships/ and this works. But when you GET a package, the dictionary lists all relationship details. So this bug creates a problem for editing a package that has relationships - you want to GET it, make any edits and then PUT it back. The work-around is to delete the 'relationships' key from the dict before you PUT it back.

Options

Ideally, CKAN would read the 'relationships' key and make the necessary changes. This is a chunk of work.

Another good option is to allow an unchanged 'relationships' value, but barf it is edited. This is also a chunk of work.

A bad option would be to just ignore the 'relationships' value, since users will get frustrated changing this value and wonder why it never saves, not understanding it is different to all the rest, without error message.

A final option is to get rid of relationships altogether.

#1314: ckanclient search - generator improvements

dread — Wed, 07 Sep 2011 11:36:50 GMT

Apparently the search generator always makes two requests, even if you don't want to see the search results, which might be slow. Can this be optimised?

Maybe we should also provide a second search function that doesn't use the generator - the original simple search function (that leaves the user to deal with limit & offset).

#1326: Write a set of auth plugin functions to integrate with Druapl

thejimmyg — Mon, 12 Sep 2011 09:51:24 GMT

Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor.

#1424: Openness notice should be clearer

dread — Wed, 26 Oct 2011 16:54:49 GMT

ckan-discuss discussion suggests changes to the 'openness' indicator ( http://lists.okfn.org/pipermail/ckan-discuss/2011-October/001786.html )

Dataset view page:

If there is an explicit but non-OKD compliant license, such as CC-BY-NC, then this should be stated explicitly, perhaps: “This dataset is Not Open. License: Creative Commons Attribution Noncommerical. This is not an open license as it does not meet the Open Knowledge Definition.”
If the license is marked as “Other::License Not Specified”, then this should be stated explicitly, perhaps: “This dataset is Not Open. It is published without an explicit license, the publisher reserves all rights to the dataset.”
3. If the license field was left empty by the contributor of the Data Hub record, then again this should be stated explicitly, perhaps: “This dataset is Not Open. The license of this dataset is unknown or unspecified. Start an enquiry on IsItOpenData? »
There is a bug so that non-open licenses doesn't have an openness notice.
If downloadable resources are not available, this should not affect 'openness' - check this has been removed.

#1544: delete old git branches

dread — Mon, 12 Dec 2011 15:10:10 GMT

We have 150 odd branches (git branch -a) - most of them old - we should prune them. At very least, branches that have been merged in should be deleted. Look at old branches that haven't been merged in and wonder why.

May be of some use:

git branch --merged git branch --no-merged

#2247: Resource preview glitch in some browsers

zephod — Tue, 20 Mar 2012 12:30:14 GMT

From Ira: Preview for google spreadsheets are not displaying correctly for me in Firefox v.10.0.02, fine in Chrome. http://i.imgur.com/KJaqz.png

#2395: paster db clean/init don't work when spatial extension enabled

dread — Wed, 16 May 2012 09:16:33 GMT

If you have a spatial enabled database then if you don't disable the spatial extension in the CKAN config temporarily then you get errors when you run paster db clean and paster db init.

Can't you just modify the clean and init functions to run without extensions enabled?

The wider problem is that extensions do their own inits every time you do load_environment, which seems crazy and inefficient to me, since this occurs every time a request comes into CKAN. But that is another problem/ticket.

#763: Read-only mode - Setup

dread — Tue, 26 Oct 2010 11:11:46 GMT

Admin configures entering read-only mode in one of two places:

CKAN config file (e.g. ckan.ini)
environment variable from Apache config
Once enabled, no writes can occur to the database (including user ratings and other usage stats).

#948: Highlight (to a sysadmin) which packages are deleted

dread — Wed, 02 Feb 2011 11:32:49 GMT

When a customer logs in as a sysadmin then he/she see all packages, including deleted and pending ones. These are hidden to the average user, but the sysadmin has no idea of this until he clicks on the package and sees at the bottom 'state: deleted'.

It should be more obvious than that on the search view - an icon, message or crossed-out name to packages are deleted.

#1285: Errors cause emails

dread — Tue, 23 Aug 2011 16:29:04 GMT

Currently a sysadmin gets an email when an exception is not caught. But there are occasions when we DO want to catch an exception so we can fail nicely for the user, but the sysadmin STILL gets an email to know to fix something. e.g. if there is an exception when search indexes a package. You want to catch the exception and still run any other notify calls.

#2348: Java client library for CKAN

ross — Tue, 01 May 2012 13:46:20 GMT

Start a Java library for interacting with the CKAN Action API.

Currently requires support for Groups, Tags, Resources, and Search.

An example app would be very useful.

#3002: API v1/2 'legacy' search parameters must be escaped before they are put into a Solr query string

amercader — Fri, 19 Oct 2012 09:32:22 GMT

Just to track @tauberer patch on Github. Would be nice to write a test for it. Probably going to 1.8.1

#2834: Defer all publish calls until all modules have been initialised

aron.carroll — Thu, 09 Aug 2012 17:08:13 GMT

Currently if a module uses sandbox.publish() in its initilaize() function then any modules initialised afterwards will miss the event.

We should queue all calls to .publish() in the pubsub module in an array until all modules have loaded then iterate over and execute them.

#2850: Add JSON support for IE7

aron.carroll — Tue, 14 Aug 2012 10:38:12 GMT

The JavaScript? just isn't going to work without it. Look into either the newer JSON3:

http://bestiejs.github.com/json3/

or the old faithful:

https://github.com/douglascrockford/JSON-js/

#2833: Load module templates before calling .initialize()

aron.carroll — Thu, 09 Aug 2012 17:05:39 GMT

I think this would be a nice feature for remote loading templates if the options.template value ends in ".html".

ckan.module('my-module', { options: { template: 'my-template.html' }, initialize: function () { this.template // This is the loaded template. } });

#2835: Client module needs a template loading method

aron.carroll — Thu, 09 Aug 2012 17:10:33 GMT

Client#getTemplate(name, params, success, error);
Where params, success and error are optional arguments. test/index.html already has an implementation called loadFixture().

#2901: Language Dropdown bug in footer with IE7

johnmartin — Tue, 04 Sep 2012 10:31:14 GMT

The language dropdown in Internet Explorer 7 behaves oddly. Essentially I think the problem is with the way that the dropdown decides to go into "drop-up mode" (e.g there aint enough screen space below the initial state so it has to drop up).

#2906: Wrong flags in language dropdown

dominik — Wed, 05 Sep 2012 16:05:44 GMT

The flags in the dropdown on details pages are not right.

You can see the error on http://s031.okserver.org:2375/dataset/malawi-aid-projects/resource/b717c20e-2006-4ad4-82d2-59b57ebc1ab0.

#1260: Remove duplicate functions from _util.html

pudo — Wed, 03 Aug 2011 10:17:32 GMT

There seems to be both a list view for dictized and non dictized data structures for all entities in _util.html at the moment. Probably in the back of someone's mind already, but cleanup here would be nice.

#1261: Investigate dots in extras search

pudo — Wed, 03 Aug 2011 10:19:28 GMT

It seems that searching for extras_foo:value works with solr, but extras_foo.bar:value doesn't. No theory on why.

#1317: password reset - improve user search

dread — Wed, 07 Sep 2011 17:12:19 GMT

In password reset, it gets confused if you have two similar users. This is because with the string the user provides, it searches several fields, not just name but also fullname and email address, allowing you to search for these. But only name is unique.

Specific problem: Ira searches for "Irina" then it finds both: <User name=irina fullname=Irina email=irina.bolychevsky@okfn.org ] and <User name=shevski fullname=Ira email=> (I think)

Maybe need to choose which field it searches?

#1647: add links to ckan discuss & dev to thedatahub

shevski — Mon, 16 Jan 2012 00:30:52 GMT

In the footer as well as more clearly & directly on the About page

#1823: Spring clean bin directory

rgrp — Mon, 20 Feb 2012 21:28:51 GMT

Huge number of accumulated (and likely unnecessary) scripts in /bin directory.

#285: Paginate list of packages on tag read page

rgrp — Wed, 07 Apr 2010 18:23:26 GMT

Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ...

#301: Package discussion pages

rgrp — Mon, 26 Apr 2010 16:57:13 GMT

A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package.

It provides a way for people to make suggestions about a package without needing access to main package.

#1130: First time users

lucychambers — Mon, 09 May 2011 10:59:21 GMT

Send users to FAQ first time on CKAN

#2535: SSL certificate for DataHub + https by default

rgrp — Fri, 15 Jun 2012 11:00:27 GMT

DataHub? is increasingly used and we should ensure it uses ssl as part of general security.

See also #1446 (Need to support https login for multiple instances as part of the CKAN package install)

#2622: Login fails in Opera 12

seanh — Thu, 28 Jun 2012 16:56:42 GMT

Try to login to CKAN using Opera 12, get "Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

#2683: Add no-cache header to _tracking API call's response to make sure it doesn't get cached

seanh — Mon, 16 Jul 2012 13:49:37 GMT

#2688: Allow ordering of groups in WUI

ross — Tue, 17 Jul 2012 10:27:55 GMT

Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name. It would be better if it could be sortable by name (or reversed) or by package_count (or reversed)

#2731: Some sites permanently 'down for maintenance'

markw — Mon, 23 Jul 2012 12:06:08 GMT

A large number of XXX.ckan.net sites give the following message:

"This Site is Down for Maintenance We apologize for the inconvenience. ~ The Open Knowledge Foundation sysadmins."

The message is unhelpful and patently false - the sites do not exist. Some of them were supposed to have been redirected to a relevant group at thedatahub.org in this ticket (now closed):

http://trac.okfn.org/ticket/933

However, the redirection only seems to have worked in one case, http://si.ckan.net.

The problem still affects the following sites - the first 4 of which have supposedly been merged:

http://lt.ckan.net
http://hu.ckan.net
http://be.ckan.net
http://gr.ckan.net
http://nz.ckan.net
http://bg.ckan.net
http://fi.ckan.net
http://katalogdanych.centrumcyfrowe.pl
http://pl.ckan.net
http://southampton.ckan.net
http://sl.ckan.net
Please sort this out by redirecting, removing the sites, giving a more helpful (and accurate) failure message, etc, as appropriate.

#2777: bug: user attributes

icmurray — Tue, 31 Jul 2012 09:19:23 GMT

#2844: SQL-only (no solr) version of CKAN

rgrp — Mon, 13 Aug 2012 11:59:28 GMT

~~Search needs to run of local DB (again)~~
paster db clean attemps to connect to SOLR (still works as does db first but then excepts which is not nice UX)

#2851: Preview of PDF tries to connect to datastore

ross — Wed, 15 Aug 2012 12:12:46 GMT

e.g. http://datahub.io/dataset/ccc-gistemp/resource/80ebdbd6-d91b-4fef-9db4-d3dfbd7e868e

CKAN attempts to render the PDF as a datastore tabular date even though the mimetype of the PDF is correctly specified.

#2888: Datapreview in Iframe

kindly — Thu, 23 Aug 2012 18:11:01 GMT

Try and put datapreview in Iframe.

#2895: QA pages (e.g. broken resource links) are not paginated

rgrp — Tue, 28 Aug 2012 17:38:05 GMT

And as a result timeout on e.g. the datahub ...

#2911: Internal documentation of Organization Groups

kindly — Thu, 06 Sep 2012 14:23:04 GMT

A summary/user story doc of how organizations and groups are expected to work.

#2934: Webtests for CKAN

Dominik — Tue, 18 Sep 2012 11:10:47 GMT

The tests in CKAN only test single pages (functional tests) or single methods (unit tests). In order to make sure that the whole system still works as expected, we need tests that cover whole processes.

An example user story to be tested would be: "As a user with create permissions I want to be able to log in, create a new resource, save it and then make sure that it is in the system." or: "As a user I want to use the search functionality and open a resource from the search page."

These user stories require a different testing system with interaction and processes.

#2943: Chrome does not resize preview

dominik — Mon, 01 Oct 2012 11:08:06 GMT

Chrome does not resize iframe after a full refresh/ on first load

#2944: Recline preview does not work in Opera

dominik — Mon, 01 Oct 2012 11:26:42 GMT

Opera cannot show a recline preview. I shows 0 records.

#2945: Pdf preview does not load in IE

dominik — Mon, 01 Oct 2012 11:27:49 GMT

The pdf preview does not load in IE 9.

#2948: Negative range breaks datastore backend

dominik — Mon, 01 Oct 2012 14:30:54 GMT

Recline allows negative ranges. The backend returns an error 500 for that which breaks the recline preview.

#2957: New datastore on postgres prior to 9.0

dominik — Wed, 03 Oct 2012 19:23:34 GMT

The GRANT SELECT ON ALL and ALTER DEFAULT PRIVILEGES are not available in Postgres prior to version 9.0.

#2992: Delete resource should send me back to edit dataset page

johnmartin — Tue, 16 Oct 2012 17:19:16 GMT

#3020: Update CKAN coding standards

seanh — Fri, 16 Nov 2012 20:11:51 GMT

#3030: clean up helper functions

toby — Tue, 18 Dec 2012 12:33:58 GMT

#2329: Add back in RSS/Atom links on relevant pages

rgrp — Thu, 26 Apr 2012 12:15:40 GMT

E.g. on dataset page, on revision page, on user page and on search results.

#2654: UI support for ordering groups on group_read page

ross — Tue, 10 Jul 2012 18:11:31 GMT

The group_index page has no support in WUI for ordering the groups displayed. Should allow sorting by name

Add support for this for datahub now, and discuss for new 1.9 UI

#2886: Configurable related items

ross — Wed, 22 Aug 2012 08:55:18 GMT

Related items (Apps & Ideas) have a collection of types, which are currently fixed. Some of these types are ambiguous, such as ideas in that they may not have a link - and if they do it is likely to be to a blog post - another existing type.

Whilst the URL is required, and we think this should stay required, we should also allow users to change the types found in Apps & Ideas to a shortened list that suits their requirements.

#2902: genshi is used in Group controller

ross — Tue, 04 Sep 2012 18:10:30 GMT

The Group controller uses the genshi module where it may be better for it to use Markup() in the template.

See https://github.com/okfn/ckan/pull/117/files#r1524863

#2913: IGroupForm has package_form() method

seanh — Tue, 11 Sep 2012 14:45:39 GMT

Should be group_form() I think

#2915: Refactor form_to_db_schema_options()

seanh — Wed, 12 Sep 2012 10:37:49 GMT

Having two methods form_to_db_schema(self) and form_to_db_schema_options(self, options) seems unnecessary, why not just have form_to_db_schema(self, options=None)?

Fixing this might break existing extensions although I don't think any are using form_to_db_schema_options() so it should be okay.

The same thing goes for db_to_form_schema_options().

Also why are we passing a dict 'options' that always contains the same three keys 'api', 'context' and 'type'? Why not three params api=None, contenxt=None, type=None?

#2920: Genshi text template message extractor no longer needed?

seanh — Thu, 13 Sep 2012 09:57:29 GMT

In setup.py, do we still need:

('templates/.txt', 'genshi', {

'template_class': 'genshi.template:TextTemplate'

}),

in message_extractors? Doesn't like there are any txt files in there anymore.

#2921: Add docstring to top of lib/extract.py file

seanh — Thu, 13 Sep 2012 10:00:07 GMT

I think it couldn't hurt for this module to have a docstring at the top of the file explaining what the module is for. I know from setup.py that it's there to provide the extract_ckan() string extractor function for Babel, but I think looking at the file on its own it's not so obvious.

Also a couple of other small fixes:

jinja2_cleaner looks like a helper function only meant to be used by extract_ckan(), might make things clearer if it was called _jinja2_cleaner(), just so it doesn't look like a public function the module wants to export.

jinja_extensions seems to be a module-level variable that is only used in one function, could be moved into the function, unless you think there might be more functions that want it in future.

#2922: Better docstring for CKANInternationalizationExtension

seanh — Thu, 13 Sep 2012 10:01:47 GMT

I'm unsure about what's going on here. As I understand it, when we run python setup.py extract_messages it's going to use the extract_ckan() function from ckan/lib/extract.py to process the HTML files. For the HTML files that are jinja2 templates, extract_ckan() will call jinja2_cleaner() which will call regularise_html() on the strings. So the strings are regularised when they are extracted from the source files.

But then we have the parse() method of CkanInternationalizationExtension? also calling regularise_html(). I don't get what's happening here. Why do the strings need to be regularised twice?

My guess is that CkanInternationalizationExtension? is used when the strings are extracted from the templates at runtime, and they need to be regularised at this time in order to match them against the regularised strings in the mo files to find the translations to output?

Maybe CkanInternationalizationExtension? needs a better docstring saying what it does?

#2923: Change regularise -> regularize

seanh — Thu, 13 Sep 2012 10:03:02 GMT

The function is called regularise_html(), can't remember what file it's in.

#2924: Better docs for trans js command, and add to release process

seanh — Thu, 13 Sep 2012 10:04:31 GMT

Add a better docstring to for trans js command explaining what it does and why, and how to use it.

Also add this command to the CKAN Release Process as it's needed there

#2925: Remove trans mangle paster command?

seanh — Thu, 13 Sep 2012 10:06:08 GMT

Is trans mangle really necessary? If you upload a pot file to Transifex, it can generate a po file for you with 100% strings translated into a fictional pseudo language where everything is really long strings of unicode characters. I found this worked well for coverage testing, and also tests handling of unicode and long strings all over the place.

#2926: I*Form tests for per-type templates

seanh — Thu, 13 Sep 2012 16:01:21 GMT

Finish the tests for the per-package-type custom template support in IPackageForm, IGroupForm and IOrganizationForm (i.e. the new_template(), read_template() etc. methods).

The new way that template inheritance works in jinja looks like it should allow this to be tested nicely without duplicating entire templates in the example extensions.

#2927: Test new package/group/organization type URLs support

seanh — Thu, 13 Sep 2012 16:02:33 GMT

IPackageForm, IGroupForm and IOrganizationForm allow you to define e.g. new package types with new top-level URLs, but the example extensions don't have tests for this yet.

#2928: Run CKAN tests with example_i*form extensions enabled

seanh — Thu, 13 Sep 2012 16:05:34 GMT

Before releasing CKAN 2.0 we need to run all the CKAN tests with a modified test-core.ini with the example_idatasetform, example_igroupform and example_iorganizationform plugins enabled. If any tests fail, fix the bugs. This needs to be done for each release so add it to the release process.

#2929: Remove is_fallback() from I*Form extension interfaces?

seanh — Thu, 13 Sep 2012 16:09:57 GMT

If a plugin wants to take over the default package type for example, it can return dataset? from its package_types() method. I don't see how there can be a package type that is not the default 'package' and is not returned by the package_types() method of a plugin, because the URLs for such a package type would not be mapped. Am I missing something or is is_fallback() not needed?

Same for IGroupForm and IOrganizationForm

#2931: Better docstring for app_globals.py

seanh — Mon, 17 Sep 2012 14:13:14 GMT

The application's Globals object is not very informative.

#2932: Add docstring to system_info.py

seanh — Mon, 17 Sep 2012 14:50:29 GMT

What is the system_info table for? It looks like a way to override config file settings in the database, perhaps so that admins can use a web interface to change site title etc. Could do with a docstring explaining the intended purpose of the table.

#2933: Update theming docs

seanh — Mon, 17 Sep 2012 15:21:12 GMT

There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs.

I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents.

#2964: Last organization admin can remove herself

seanh — Mon, 15 Oct 2012 10:14:18 GMT

If you are the only admin of an organization you can edit the organization's members and demote yourself, then the org has no admins and no one (except sysadmins maybe) can edit it.

Last admin should not be able to remove or demote herself.

Also applies to groups.

#2966: 'Add' button text is wrong when editing organization members

seanh — Mon, 15 Oct 2012 10:15:49 GMT

e.g. if I just changed a member's capacity I am not adding anything 'save' is better maybe

#2967: Organization members edit page reloads after demoting self

seanh — Mon, 15 Oct 2012 10:17:07 GMT

Edit an organizations members page and demote yourself from organization admin, after saving the members edit page reloads even though you are no longer an admin and should no longer be able to access this page.

#2968: Anyone can access organization members page

seanh — Mon, 15 Oct 2012 10:19:15 GMT

The button will not show if you are not authorized but browse to /organization/members/foo and you can edit the members, it does stop you when you try to save your changes, but you shouldn't be able to get to the page at all

#2969: Group members page 500s

seanh — Mon, 15 Oct 2012 10:20:54 GMT

The group members page (e.g. /group/members/roger) seems broken, IDs instead of user names are shown for the members, and clicking on a member 500s

#2970: Organization and group member links use id not name

seanh — Mon, 15 Oct 2012 10:22:11 GMT

e.g. it the 'Members' button links to /organization/members/0a44... instead of /organization/members/foobar

#2971: "Are ytou sure you want to delete this member?" should say which member

seanh — Mon, 15 Oct 2012 10:27:20 GMT

when deleting members from groups and orgs

#2972: Remove any imports of authz.py and delete file

seanh — Mon, 15 Oct 2012 10:37:22 GMT

it is no longer used

#2973: Move new_authz.py into logic/auth/__init__.py

seanh — Mon, 15 Oct 2012 10:39:05 GMT

Makes sense to keep all auth stuff under logic/auth.

Also decide which functions from new_authz.pu should be marked private with _.

Also decide which functions from new_authz.py are really helpers for the auth functions in get.py, update.py and delete.py, and should maybe moved to another file, e.g. logic/auth/helpers.py

#2974: General of all auth functions

seanh — Mon, 15 Oct 2012 10:40:12 GMT

From Toby:

A general cleanup of all auth functions (in logic/auth) to check their fitness, error messages, etc.

#2975: Tests for auth functions and new actions

seanh — Mon, 15 Oct 2012 10:41:00 GMT

From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions

#2976: Polish group and organization member pages

seanh — Mon, 15 Oct 2012 10:41:49 GMT

From Toby: A little polish to the member pages and probably a little hardening of the controller and logic actions

#2977: Fix user autocomplete on group and organization member pages

seanh — Mon, 15 Oct 2012 10:42:29 GMT

From Toby: @johnmartin the user autocomplete on member add needs fixing it gets data but does not understand what to do with it

#2978: Tests for permissions for organizations and groups

seanh — Mon, 15 Oct 2012 10:43:26 GMT

From Toby: checking the logic around the who can do what from the user stories - via some tests may be a good approach

#2979: Requesting membership to groups and organizations

seanh — Mon, 15 Oct 2012 10:44:04 GMT

From Toby: The requesting membership user stories and implementation remain needed

#2981: Remove config from jinja templates

seanh — Mon, 15 Oct 2012 10:46:00 GMT

From Toby: remove the config from jinja2 templates get all such stuff via g. (before 2.0)

#2982: Move functionality from controllers into template helpers

seanh — Mon, 15 Oct 2012 10:46:42 GMT

From Toby: strip more functionality from controllers and add via template helpers eg activity streams

#2983: Refactor lib/base.py to remove circular import issues

seanh — Mon, 15 Oct 2012 10:47:24 GMT

From Toby: c) refactor lib/base.py to remove the circular import issues (render functions to lib/render.py - would fix much of that)

#2984: Auto-populate context with user, model, session, etc.

seanh — Mon, 15 Oct 2012 10:48:21 GMT

From Toby: make the context auto-populated with things like user/model/session if not user supplied - especially for extensions (pre 2.0) maybe via a helper or else a decorator on the action - helper seems nicer maybe done via get_action()

#2985: Make lib/helpers.py more template-specific

seanh — Mon, 15 Oct 2012 10:49:48 GMT

From Toby: make lib/helpers more template specific and less generally shared with none template uses (some functions are shared but many shouldn't be) - again help avoid circular imports

#2986: Make lib/base.py more like it was originally intended

seanh — Mon, 15 Oct 2012 10:50:45 GMT

From Toby: make lib/base.py be more like it was originally intended eg defines stuff that is shared eg _ but maybe try to include stuff like render/get_action if the circular import stuff can be solved - which should be doable if hard and likely needs an extra file or two for the externals _ etc

#2987: Remove all direct calls to logic action and auth functions

seanh — Mon, 15 Oct 2012 10:51:27 GMT

From Toby

#2988: UI functional tests for CKAN 2.0

seanh — Mon, 15 Oct 2012 11:02:15 GMT

Afaik the new CKAN 2.0 frontend has no functional tests

#2989: "Add dataset to organization" should auto-select the organization

seanh — Mon, 15 Oct 2012 11:04:12 GMT

'Add dataset to organization' button, when you get through to the third stage of the new dataset form the organization you came from is not selected.

I don't see any option to choose the group when adding or updating a dataset, but if I add a dataset via the "Add dataset to group" button on a group's page, then the dataset seems to get added to that group. (And I can also add/remove existing datasets by editing the group.) I wonder if organizations should work the same way, instead of having an Organization drop-down when creating or updating a dataset. The add dataset page needs to somehow indicate that you're adding a dataset to a certain group or organization though, doesn't currently.

#2990: Fix descriptions of groups and organizations on /groups and /organizations pages

seanh — Mon, 15 Oct 2012 11:04:51 GMT

#2991: Add blocks to header.html template

seanh — Tue, 16 Oct 2012 11:10:20 GMT

Currently if you want to customise the header.html template you have to copy the whole thing, it isn't broken up into blocks like other templates are.

#2993: "logged_in" and "visitor" show in user list at /users

seanh — Wed, 17 Oct 2012 09:42:02 GMT

#2994: Add dataset and user popovers throughout the site

seanh — Wed, 17 Oct 2012 14:26:04 GMT

If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button.

These popovers should appear wherever datasets or users are listed, throughout the site.

#2995: Popovers for resources, groups, organizations, tags...

seanh — Wed, 17 Oct 2012 14:27:21 GMT

If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button.

It would be nice to add similar popovers for other types of object in CKAN, e.g. resources, groups, organizations, tags...

#2997: Add activity streams to dataset pages

seanh — Wed, 17 Oct 2012 14:31:46 GMT

Add an activity stream tab to dataset pages, like we have on user profile pages. Dataset activity streams are already implemented in the backend.

#2998: Add activity streams to group pages

seanh — Wed, 17 Oct 2012 14:33:14 GMT

Before we do this we should implement smarter group activity streams (i.e. what activities should appear in a group's activity stream?) and following of groups.

#2999: Add activity streams to organization pages

seanh — Wed, 17 Oct 2012 14:34:17 GMT

Before we do this we need to merge the organizations branch into master, we need to implement smarter activity streams for organizations, and we need to implement following of organizations.

#3000: Add number of followers to dataset pages

seanh — Wed, 17 Oct 2012 14:36:00 GMT

User profile pages now show some interesting stats about the user: num. datasets, num. edits and num. followers. Would be nice to add some similar info to dataset pages. At least num. followers can be done (already implemented in backend).

#3001: Multilingual plugin crashes CKAN on add dataset when some languages are default

seanh — Thu, 18 Oct 2012 16:50:48 GMT

Enable the multilingual plugins:

ckan.plugins = stats synchronous_search multilingual_dataset multilingual_group multilingual_tag

and set your default language to one not supported by the multilingual plugin, e.g.

ckan.locale_default = cs_CZ

now run CKAN and try to add a dataset:

File '/home/seanh/Projects/ckan171/ckan/ckanext/multilingual/plugin.py', line 141 in before_index

text_field_items+ default_lang?.extend(all_terms)

KeyError?: 'text_cs_CZ'

It doesn't matter what language you are viewing the site in in your browser, the default language setting in the ini file determines whether it crashed or not.

A number of supported languages are defined at the top of ckanext/multilingual/plugin.py. I think if the default language is not one of these it crashes.

I think this affects all versions of CKAN since the multilingual plugin was added so at least 1.7, 1.8 and 2.0

#3007: Adding a dataset creates multiple activities

seanh — Mon, 29 Oct 2012 13:15:07 GMT

Adding a new dataset creates multiple activity stream activities, e.g. seanh created the dataset foo, seanh add the resource bar to the dataset foo, and seanh updated the dataset foo are all created when I add a new dataset. I wonder if these can be collapsed into a single activity.

#3010: Pin images don't appear in data explorer

seanh — Wed, 31 Oct 2012 17:45:06 GMT

When minified files are in use (set debug = false in ini file) the pin images for points don't show up in the data explorer map view when viewing a resource with geolocation data.

set debug = true will fix the problem by using unminified files instead but also turns on a bunch of debug stuff you wouldn't want on a production site.

middleware.py, around line 73, is where the decision to turn minified files on or off based on the debug setting is made. This can be hacked on a production site to run with unminified files but otherwise run on production mode thereby getting around the issue with the pins. But that doesn't fix the underlying bug.

#3014: Crash when deleting a non-empty vocabulary

seanh — Tue, 13 Nov 2012 10:50:08 GMT

From Knud Möller:

when I try to delete a non-empty tag vocabulary via the API (through HTTP), I get an internal server error. Checking the logs, this turns out to be a consistency error raised by sqlalchemy:

Error - <class 'sqlalchemy.exc.IntegrityError?'>: (IntegrityError?) update or delete on table "vocabulary" violates foreign key constraint "tag_vocabulary_id_fkey" on table "tag" DETAIL: Key (id)=(21421955-7560-467c-af30-9f790b73e6ae) is still referenced from table "tag".

'DELETE FROM vocabulary WHERE vocabulary.id = %(id)s' {'id': u'21421955-7560-467c-af30-9f790b73e6ae'}

URL: http://33.33.33.10:5000/api/action/vocabulary_delete

The error makes sense, but I'm wondering if it would be useful to extend the API to also allow the deletion of non-empty vocabularies, possibly via a parameter (not sure what best practice in API design is). At the very least, it would be cool if the error message coming back in the response had more information in it.

#140: News section on front page

rgrp — Wed, 07 Oct 2009 08:02:21 GMT

Have a news section (suggest as a sidebar item).

News section will link to latest 3/4 blog posts on CKAN from blog.okfn.org.

Details:

Suggest pulling via rss or similar.
Will want to cache this ...
Cost: 4h?

#253: Package relationships

dread — Mon, 22 Feb 2010 16:05:21 GMT

Overview

Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management.

Examples

There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial.
In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.)
OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it.
IPCC collection of data linked / mirrored. Not sure if there are useful relationships here?
Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM.
See more examples discussed here: http://trac.ckan.org/ticket/253

Implementation

This is split into four tickets:

Model: ticket:254
Read in WUI: ticket:255
Edit in WUI: ticket:256
API: ticket:257
No need for write access to be provided API for the moment.

This ticket also encompasses ticket:169 (Package derivations) and ticket:176 (Package dependencies).

#331: Timezone of CKAN timestamps should be configurable

rgrp — Mon, 31 May 2010 10:40:40 GMT

Revisions are timestamped using the server's clock, which may not relate to the expected timezone for the site. e.g. the Norway site has a server on GMT. No timezone info is displayed either.

Would like to set timezone for a CKAN instance to use in rendering revision timestamps. For example, use CET or EST timezone.

#350: Search engine optimisation

dread — Tue, 15 Jun 2010 09:35:41 GMT

Need to research what can easily be done to improve CKAN packages in the search rankings.

Comments from Glen Barnes:

We've been pretty successful at SEO without even really trying (see http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (http://www.ckan.net/package/ascoe -> http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly ("Atmospheric Chemistry Studies in the Oceanic Environment" on the example above). Some basic SEO 101 on page optimisations.

#351: Homepage: list new, updated and 'hot' packages

dread — Tue, 15 Jun 2010 09:56:56 GMT

Have a simpler list of exciting data, as opposed to the big revision list.

For example:

Hot data =========== New packages: package1, package2, package3 Updated resources: package1, package2, package3 Popular packages:

#369: "Package Listing Key" should appear on Tag results

shudson@… — Thu, 22 Jul 2010 18:00:34 GMT

Currently there's a nice legend titled "Package Listing Key" that appears in right side of "Browse Packages" results. The same key should show on other search results like when searching for a tag.

#370: Use better email encryption for author_email and maintainer_email

shudson@… — Thu, 22 Jul 2010 18:03:39 GMT

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.

#653: Trackback links for packages

dread — Tue, 21 Sep 2010 09:40:25 GMT

When people link to a package, a track-back link is auto-created. (Similar system as for blogs).

As suggested by Tim Davies:

Allowing some form of ‘track back’ against datasets When a non-technical user comes to look at a dataset it would be really useful for them to be able to see if anyone has created an interface interpretation of it already.

I found quite a few cases in research of end-users struggling to make sense of a dataset when good interfaces to that data had already been built and blogged about, but without there being any link from the dataset listing to those data uses. Accepting track backs could also make it easier for technical users to find blog posts / shared code etc. relating to a given dataset.

#737: Markdown syntax summary page

dread — Fri, 22 Oct 2010 16:59:09 GMT

I suggest we produce a quick Markdown cheat-sheet page, showing the key runes: e.g. create a title and quote some text. This page can link to the full Markdown docs for advanced users.

A user going to the Markdown docs that we link will have to read a couple of pages of the raison-d'etre of Markdown before he gets to the syntax. And it's not very easy to read, and being white on black it looks like proper geek stuff.

#765: Read-only mode - API usage

dread — Tue, 26 Oct 2010 11:18:17 GMT

All writes to the API are captured and you are returned an error explaining the reason.

Possible errors:

503 temporary maintenance
403 forbidden (if server if permanently read-only)

#811: Extra field editing form layout breaks when there are long field names

cygri — Wed, 17 Nov 2010 11:53:32 GMT

The layout of the editing section for extra fields breaks when a field name is slightly too long. Field names jump over to the right. See http://ckan.net/package/edit/dbpedia for examples.

#812: Package edit form only allows three extra fields

cygri — Wed, 17 Nov 2010 11:56:50 GMT

Rationale

The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview).

Implementation

A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

#818: Rethinking the author and maintainer fields

cygri — Wed, 17 Nov 2010 14:18:44 GMT

The semantics of the Author and Maintainer fields are really unclear at the moment. This leads to very inconsistent usage. Also, perhaps Name and Email are not the only fields that are needed for a contact.

Here is a table that shows the current usage of these fields in CKAN: http://richard.cyganiak.de/2010/ckan/ckan-ppl.html

We note several problems:

Author and Maintainer are often the same
Author and Maintainer are often used interchangeably
People really want to specify URLs for the contacts and stick them into random places because there is no field for it
Multiple comma-separated names in a single field
I'm not sure what to do about this, but a redesign is necessary in my opinion.

Some ideas:

Remove the maintainer field?
Make really clear that Author doesn't refer to the metadata on CKAN, but to the original data
Add an “author URL” field?

#837: CKAN integration with freebase gridworks / google refine

rgrp — Tue, 30 Nov 2010 18:10:09 GMT

Thread: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

Scenario 1

User installs Refine and CKAN extension for refine
On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
They edit the dataset on Refine
On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)
NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required)

Scenario 2

User visits a package on CKAN.net (or another CKAN instance)
There is a button on the page "View and edit this dataset in Google Refine"
Click button -- ask them if they have Google refine installed
Yes: instructions for loading dataset into refine
No: load dataset in hosted version of google refine (we could run this)
User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset.

#895: Add version number (or simular) to css/js includes query string

memespring — Thu, 06 Jan 2011 19:49:42 GMT

Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g.

mycssfile.css?v=12345678

#909: DCat importer for CKAN

pudo — Mon, 17 Jan 2011 12:05:58 GMT

Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported:

CKANrdf generated exports
opengov.se RDF (not really DCat)
Sunlight Nationdatacatalog as harvested by dcat-tools.

#1101: Integrate googlanalytics into site nav

sebbacon — Thu, 21 Apr 2011 13:52:06 GMT

There's a stats plugin (e.g. at http://trac.ckan.org/ticket/832).

Output from the googleanalytics plugin should append to that page, if the stats plugin is present.

Possibly the stats plugin and the googleanalytics plugin should be merged?

Finally, if the stats plugin is active, then a link to the stats page should be added to the main site footer.

#1120: Atom feeds of each tag

tsm — Mon, 02 May 2011 04:08:05 GMT

Tags could/should have an Atom feed. This would mean that every edit to relevant packages could be easily monitored. See [1].

[1] http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001162.html

#1144: Support DSPL

timmcnamara — Thu, 19 May 2011 00:06:49 GMT

DSPL, the Dataset Publishing Language, is being promoted by Google for its "Google Public Data Explorer" system. It is an XML format with metadata.

The format is described on the developer docs ofthe Google Code site.

Google provides a Python script which reads CSV data and generates DSPL

Sample from http://code.google.com/apis/publicdata/docs/dspl_sample.html:

<?xml version="1.0" encoding="UTF-8"?> <dspl xmlns="http://schemas.google.com/dspl/2010" xmlns:geo="http://www.google.com/publicdata/dataset/google/geo" xmlns:geo_usa="http://www.google.com/publicdata/dataset/google/geo/us" xmlns:time="http://www.google.com/publicdata/dataset/google/time" xmlns:quantity="http://www.google.com/publicdata/dataset/google/quantity" xmlns:entity="http://www.google.com/publicdata/dataset/google/entity"> <import namespace="http://www.google.com/publicdata/dataset/google/time"/> <import namespace="http://www.google.com/publicdata/dataset/google/quantity"/> <import namespace="http://www.google.com/publicdata/dataset/google/entity"/> <import namespace="http://www.google.com/publicdata/dataset/google/geo"/> <info> <name> <value>My statistics</value> </name> <description> <value>Some very interesting statistics about countries</value> </description> <url> <value>http://www.stats-bureau.com/mystats/info.html</value> </url> </info> <provider> <name> <value>Bureau of Statistics</value> </name> <url> <value>http://www.stats-bureau.com</value> </url> </provider> <topics> <topic id="geography"> <info> <name><value>Geography</value></name> </info> </topic> <topic id="social_indicators"> <info> <name><value>Social indicators</value></name> </info> <topic id="population_indicators"> <info> <name><value>Population indicators</value></name> </info> </topic> <topic id="poverty_and_income"> <info> <name><value>Poverty & income</value></name> </info> </topic> <topic id="health"> <info> <name><value>Health</value></name> </info> </topic> </topic> </topics> <concepts>  <concept id="population"> <info> <name> <value>Population</value> </name> <description> <value>Size of the resident population.</value> </description> </info> <topic ref="population_indicators"/> <type ref="integer"/> </concept>  <concept id="country" extends="geo:location"> <info> <name> <value>Country</value> </name> <description> <value>My list of countries</value> </description> </info> <type ref="string"/> <property id="name"> <info> <name><value xml:lang="en">Country name</value></name> <description> <value xml:lang="en">The official name of the country</value> </description> </info> <type ref="string"/> </property> <table ref="countries_table"/> </concept>  <concept id="state" extends="geo:location"> <info> <name> <value>State</value> </name> <description> <value>US states</value> </description> </info> <type ref="string"/> <property concept="country" isParent="true"/> <table ref="states_table"/> </concept> <concept id="gender" extends="entity:entity"> <info> <name> <value>Gender</value> </name> <description> <value>Gender, Male or Female</value> </description> <pluralName><value>Genders</value></pluralName> <totalName><value>Both genders</value></totalName> </info> <type ref="string"/> <table ref="genders_table"/> </concept> <concept id="unemployment_rate" extends="quantity:rate"> <info> <name> <value>unemployment rate</value> </name> <description> <value>The percent of the labor force that is unemployed, not seasonally adjusted.</value> </description> <url><value>http://www.bls.gov/cps/cps_htgm.htm</value></url> </info> <topic ref="social_indicators"/> <type ref="float"/> <attribute id="is_percentage"> <type ref="boolean"/> <value>true</value> </attribute> </concept> </concepts> <slices> <slice id="countries_slice"> <dimension concept="country"/> <dimension concept="time:year"/> <metric concept="population"/> <table ref="countries_slice_table"/> </slice> <slice id="states_slice"> <dimension concept="state"/> <dimension concept="time:year"/> <metric concept="population"/> <metric concept="unemployment_rate"/> <table ref="states_slice_table"/> </slice> <slice id="countries_gender_slice"> <dimension concept="country"/> <dimension concept="gender"/> <dimension concept="time:year"/> <metric concept="population"/> <table ref="countries_gender_slice_table"/> </slice> </slices> <tables> <table id="countries_table"> <column id="country" type="string"/> <column id="name" type="string"/> <column id="latitude" type="float"/> <column id="longitude" type="float"/> <data> <file format="csv" encoding="utf-8">countries.csv</file> </data> </table> <table id="countries_slice_table"> <column id="country" type="string"/> <column id="year" type="date" format="yyyy"/> <column id="population" type="integer"/> <data> <file format="csv" encoding="utf-8">country_slice.csv</file> </data> </table> <table id="states_table"> <column id="state" type="string"/> <column id="name" type="string"/> <column id="country" type="string"> <value>US</value> </column> <column id="latitude" type="float"/> <column id="longitude" type="float"/> <data> <file format="csv" encoding="utf-8">states.csv</file> </data> </table> <table id="states_slice_table"> <column id="state" type="string"/> <column id="year" type="date" format="yyyy"/> <column id="population" type="integer"/> <column id="unemployment_rate" type="float"/> <data> <file format="csv" encoding="utf-8">state_slice.csv</file> </data> </table> <table id="genders_table"> <column id="gender" type="string"/> <column id="name" type="string"/> <data> <file format="csv" encoding="utf-8">genders.csv</file> </data> </table> <table id="countries_gender_slice_table"> <column id="country" type="string"/> <column id="gender" type="string"/> <column id="year" type="date" format="yyyy"/> <column id="population" type="integer"/> <data> <file format="csv" encoding="utf-8">gender_country_slice.csv</file> </data> </table> </tables> </dspl>

#1145: Support the Handle System

timmcnamara — Thu, 19 May 2011 00:26:15 GMT

The Handle System is an initiative to provide persistent references for resources. That is, it's basically a proxy system for preventing link rot.

Its documentation is here: http://www.handle.net/. Servers running CKAN could host a "Local Handle Service", which redirects a hash of a resource to an actual URL.

Some suggested use cases:

Researcher would like to cite where data came from
Agencies would like to have a way to prevent vendor lock-in from CKAN if they decide to move to another platform

#1179: Support tag aliases

timmcnamara — Tue, 07 Jun 2011 06:47:01 GMT

A small number of tags are near-duplicates of each other.

Perhaps we could support word stemming from NLTK and/or manual tag aliases:

statistics statistik ... survey surveying surveys

#1182: Comments from deleted packages appear in "Recent Comments" feed

timmcnamara — Thu, 09 Jun 2011 22:24:11 GMT

When a package has been deleted, say for spam moderation, comments still appear in the recent comments section.

This is a problem because non-admin users will be shown a warning that they're not authorised to view the package if they click on the link.

At CKAN.net currently, this affects the most recent comment.

#1184: Support Wuala as CKAN storage option

timmcnamara — Tue, 14 Jun 2011 06:59:11 GMT

Most of CKANs storage options are tied to the USA. This brings concerns of data security for some organisations who may wish to adopt the system. Wuala is a distributed file system that stores data in a peer-to-peer manner. The company behind it, LaCie? sells storage for a fee. However, they also enable clients to have 'free' storage space when machines act as a storage node.

In order to be a storage node, a machine needs to be online for more than 14% of the time - roughly 4h per day. Most CKAN servers are likely to have a far greater uptime than this.

Supporting Wuala would go some way to enabling CKAN to be used in a secure manner. That is, CKAN could be promoted for organisational use where there is lots of data to be stored and large geographic distances to be managed. There is a Python client available and a fairly long Google Tech Talk that overviews the system.

#1185: Administrators can't delete packages from web UI

timmcnamara — Wed, 15 Jun 2011 04:23:38 GMT

Administrators have "View", "Edit" and "History" tabs. However, I can't see a way to delete a package from the web UI.

Version: CKAN.net as of today

#1188: Allow diffing against initial (blank) package version

nickstenning — Wed, 15 Jun 2011 15:52:40 GMT

Currently the history page only allows diffing between different versions of a package, but there doesn't appear to be any easy way to see the changes introduced by the first version of a package.

I'm requesting the ability to diff against a "blank slate" initial state of a project, so I can see the content of the first project commit.

Not sure if this is a vdm feature, so I'm putting this ticket in against ckan.

#1201: seperate out logic in atom feeds to logic layer.

kindly — Fri, 24 Jun 2011 15:21:32 GMT

Simplify the logic in the atom feed an make all feeds use logic layer to return lists.

#1227: Display packages' tags in search results

timmcnamara — Tue, 19 Jul 2011 00:11:02 GMT

In when displaying search results, it would be useful to also display the tags of a package. Sometimes it's difficult to infer the scope of what the package does from the title and the first sentence of the description. Tags are quite concise way to display rich information.

ENV=datacatalos.org, with CKAN 1.4.2a

#1232: [super] Interface improvements

thejimmyg — Wed, 20 Jul 2011 16:11:36 GMT

Child tickets:

#1194 "Welcome back" message for newly registered user
#1202 Links to datapkg utility don't lead to info about it
#925 Change the search box icon to remove the down arrow
#923 Search box doesn't work in leaderboard page in stats extension
#1034 Flash message cached
#737 Markdown syntax summary page
#811 Extra field editing form layout breaks when there are long field names

#1233: [super] Improve wiki-style functionality for history

thejimmyg — Wed, 20 Jul 2011 16:29:52 GMT

At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas:

#191 Searching by modification date
#193 Searching by time-related field
#301 Package discussion pages
#1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages
#1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page
Other improvements as per my word doc.

#1273: Create docs for API v3

amercader — Mon, 15 Aug 2011 12:41:23 GMT

#1278: Refactor authorized_query calls

amercader — Mon, 15 Aug 2011 13:32:57 GMT

There are some functions that still use the Auhtorizer().authorized_query method:

./ckan/controllers/authorization_group.py:24: query = ckan.authz.Authorizer().authorized_query(c.user, model.AuthorizationGroup) ./ckan/lib/base.py:237: groups = ckan.authz.Authorizer.authorized_query(c.user, model.Group, ./ckan/lib/search/sql.py:55: q = authz.Authorizer().authorized_query(username, model.Group) ./ckan/lib/search/sql.py:118: q = authz.Authorizer().authorized_query(self.options.get('username'), model.Package) ./ckan/logic/action/get.py:154: query = Authorizer().authorized_query(user, model.Group, model.Action.EDIT) ./ckan/tests/test_authz.py:158: q = self.authorizer.authorized_query(self.notadmin.name, model.Package) ./ckan/tests/test_authz.py:353: q = self.authorizer.authorized_query(self.notmember.name, model.Package) ./ckan/tests/test_authz.py:357: q = self.authorizer.authorized_query(self.member.name, model.Package) ./ckan/tests/functional/test_authorization_group.py:44: group_count = Authorizer.authorized_query(u'russianfan', model.AuthorizationGroup).count()

#1355: Package extras property does not include the newly created ones

amercader — Mon, 26 Sep 2011 10:48:46 GMT

The extras in the package object sent to the extensions after editing (https://bitbucket.org/okfn/ckan/src/01efd5649c10/ckan/logic/action/update.py#cl-226) do not include the newly added.

#1406: Re-enable RSS subscriptions

zephod — Mon, 17 Oct 2011 14:22:07 GMT

RSS 'subscribe' buttons appeared in many places on the site but were not very helpful. They took (confused) users pointed to the raw feed code, and Google Reader could not understand the feed. Safari, however, could interpret it correctly.

Their presentation needs to be clear and consistent; the RSS feed really needs testing in a variety of readers; and we need to decide exactly which items should get a feed. (Package updates? Groups?)

#1414: track user log-ins on thedatahub.org

shevski — Mon, 24 Oct 2011 11:13:02 GMT

Set up tracking for user logins so that we have stats about how many active users of thedatahub exist want to be able to see who logged in the the last x months

#1423: Edit resources suggestions

markbrough — Wed, 26 Oct 2011 15:11:46 GMT

Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: http://thedatahub.org/dataset/edit/iati-registry
Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in.

#1432: [super] Data processing system for CKAN and Webstore

rgrp — Tue, 01 Nov 2011 10:19:07 GMT

Super ticket: #1190

A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)

#1535: Plump for auth header of: X-CKAN-API-KEY

dread — Wed, 07 Dec 2011 17:31:22 GMT

When using the API, the apikey needs to be supplied in a header called 'Authorization'. Because some proxys / deployments use this header for other things, a configurable header was provided as an alternative, with default "X-CKAN-API-KEY".

Rufus suggests having *one* way for this. a) making this not configurable any more b) making X-CKAN-API-KEY the default

(keep Authorization allowed, but not documented, for backwards compatibility)

#1550: Allow simple auth via the API

ross — Fri, 16 Dec 2011 15:29:29 GMT

It should be possible to pass userid/username and api key and obtain a response from CKAN for external services that use CKAN auth. Those services shouldn't be talking to the DB directly.

#1578: [super] Re-enable and refactor ratings

rgrp — Mon, 19 Dec 2011 19:20:43 GMT

#1598: Reinstate Ratings

rgrp — Thu, 29 Dec 2011 16:52:04 GMT

Ratings were disabled approximately a year ago because:

Unclear purpose and UX. What did ratings tell you? How useful were they?
Spamming (esp by bots: you could submit an anonymous rating via a GET request which caused problems)
Both problems are solvable and it would be nice to have this feature reinstated.

Purpose: can make this more purposable by limiting to logged in users (or at least distinguishing logged in from non-logged in users)
Even better we could allow ratings to be made public (I'm interested in what someone else I respect finds important)
Spamming: limit to logged in users and / or use AJAX over an API to submit ...

#1604: Get ckanext-moderatededits working with CKAN 1.5+ templates

dread — Sat, 31 Dec 2011 17:27:09 GMT

ckanext-moderatededits requires an old and possibly development version of CKAN. It would be good to update it for later CKAN versions.

According to the README, you need CKAN from branch feature-1141-moderated-edits-ajax but the changelog suggests this branch went into version 1.4.2. So it possibly works with 1.4.2 and 1.4.3(.1). But CKAN 1.5 has revamped templates, so the genshi stream filters definitely don't work.

BTW history_ajax/read_ajax calls have been deprecated in CKAN since 1.5.2a and will need fixing up to use the Action API too as part of this.

#1606: metadata license config option

dread — Mon, 02 Jan 2012 10:45:30 GMT

Add a config option to choose the metadata licence. Set default to Open Database License.

Currently the dataset edit form says "Important: By submitting content, you agree to release your contributions under the Open Database License." This is hard-coded, but not suitable for when DGU uses the CKAN form - they use the OGL.

#1642: Extra link generators generate garbled HTML

pudo — Thu, 12 Jan 2012 15:29:31 GMT

I had a package descriptions with URLs that contain "group:foo". This produces garbled output as the system tries to generate two sets of links: the outer link and an inner link.

Need to fix the parser.

Text:

Webdienst basierende Bereitstellung von Geobasisdaten der Freien und Hansestadt Hamburg. Folgende Geobasisdaten werden als WebMapTileService? (WMT-S) für die Dauer des Wettbewerbs netzbasiert unter der Creative Commons Lizenz zur Verfügung gestellt: Digitale Orthophotos 40 cm Auflösung (Layer: apps4d_DOP40), Digitale Stadtkarte (Layer: apps4d_DISK), Digitale Regionalkarte (Layer: apps4d_DIRK), Digitale Karte 1:5000 (Layer: apps4d_DK5).

Metadateneinträge zu den Daten im PortalU:

[apps4d_DOP40](http://www.portalu.de/trefferanzeige?docuuid=D6082285-6CAC-49C5-B7ED-B5A72BC17DEC&plugid=/kug-group:kug-iplug-udk-db_hh)
[apps4d_DISK](http://www.portalu.de/trefferanzeige?docuuid=D2DA3359-3DC9-4CA4-AE9B-7C63D53B220F&plugid=/kug-group:kug-iplug-udk-db_hh)
[apps4d_DIRK](http://www.portalu.de/trefferanzeige?docuuid=B7DDD599-F677-4C80-BED5-F11D82D5D1DD&plugid=/kug-group:kug-iplug-udk-db_hh)
[apps4d_DK5](http://www.portalu.de/trefferanzeige?docuuid=2AE6D23E-48A5-4D85-BC0A-160737E0C8D2&plugid=/kug-group:kug-iplug-udk-db_hh)
One fix is quoting the URLs

#1643: Add fixed tags to thedatahub for better browsing

shevski — Thu, 12 Jan 2012 18:34:53 GMT

Similar to publicdata.eu, want to have themed areas such as finance, environment, census, etc and country tags

#1644: Order default dataset page by most downloaded resources on thedatahub

shevski — Thu, 12 Jan 2012 18:39:02 GMT

Instead of alphabetically as we do currently, alternatively by most viewed datasets

for http://thedatahub.org/dataset

#1668: repoze version discrepency

dread — Tue, 17 Jan 2012 12:02:26 GMT

There's a discrepency in repoze.who versions between the source and package installs:

repoze.who - package 1.0.18 vs source 1.0.19
repoze.who-friendlyform - package 1.0b3 vs source 1.0.8
We get a test failure [1] with the 1.0b3 version (from the ubuntu 10.04 python-repoze.who-plugins package). But we've not noticed any problems on s057 instances (br, no, ie etc) which have the package versions of repoze.who.

The reason the package install uses the earlier packaged versions rather than the ones we'd like is that repoze uses all sorts of horrendous import hacks, making it too difficult to put into our 'ckan-conflict' source package.

James suggests we 'do something horrible like dynamically patch repoze on CKAN import'.

[1] http://buildbot.okfn.org/builders/builder-ckan/builds/1371/steps/shell/logs/stdio ERROR: ckan.tests.functional.test_user.TestUserController?.test_user_create_unicode

#1679: Default roles problem

dread — Tue, 17 Jan 2012 17:57:22 GMT

The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py).

The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different.

We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?

#1684: Remove all config from ckanext-archiver

ross — Thu, 19 Jan 2012 14:37:01 GMT

ckanext-archiver currently has a settings file (and a default) and it should be passed in all relevant information from the context.

Remove all settings (ARCHIVE_DIR and MAX_CONTENT_LENGTH and others) and pass them in from CKAN.

#1697: A Configurable list of states for a Dataset

rgrp — Tue, 24 Jan 2012 10:23:50 GMT

Currently have 'active' and 'deleted' suggest also:

'draft'
'hidden'
(Do we need both). Also write out workflows related to these.

#1717: [super] Search UX improvements

shevski — Thu, 26 Jan 2012 18:53:01 GMT

Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search)
Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page).
Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it.
More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu
Blue search button should be displayed in line with the search bar, not underneath
Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name

#1747: Expire old activities

seanh — Sun, 05 Feb 2012 12:56:29 GMT

Currently the activity streams database tables just get longer and longer over time. Do we want to eventually delete the oldest activities, to keep the length of the table within limits?

#1750: Move ckan/lib/activity.py into the model

seanh — Sun, 05 Feb 2012 18:18:08 GMT

Move ckan/lib/activity.py moved to into the model - say ckan/model/activity_extension.py, because it's so tightly knit with the model code, whereas most of the lib code is used in the controllers.

#1789: Implement a tag_update() logic action function

seanh — Thu, 09 Feb 2012 16:36:53 GMT

So users can rename a tag and/or move it between vocabularies using the API.

Currently we have create_tag() and delete_tag(), but if you were to 'update' a tag by deleting it and then recreating it all the datasets that had that tag will have lost it and you'll have to re-add it to them all.

What should happen to datasets that have the tag, if the tag gets moved between vocabularies? All the datasets just keep the tag with the new vocabulary? This will become a problem if/when we support 'radio button'-style vocabularies (where each dataset can only have one tag from the vocabulary).

#1831: Login with email address

ross — Fri, 24 Feb 2012 08:58:26 GMT

Users should be able to log in using either their username, or their email address, both of which are unique.

Will require a change to UsernamePasswordAuthenticator? in ckan.lib.authenticator.py and possible a useful User.by_email in the user model if it doesn't already exist.

#2197: Storage Metadata API: add/update not working with local file storage (Pairtree)

zydio — Tue, 28 Feb 2012 09:29:37 GMT

If OFS is configured with Pairtree to use a local file storage, all POST requests to add/update metadata ( /api/storage/metadata/{label} ) will fail.

This is due to the use of BotoOFS specific private methods in StorageAPIController.set_metadata(), eg: self.ofs._require_bucket(bucket), self.ofs._get_key(b, label), self.ofs._update_key_metadata(k, metadata) ... those methods can't be found in POTFS and this causes errors. The API should use only OFSInterface methods, or should conditionally make calls based off the actual type of self.ofs.

PS: I did set "ckan" as "Component" in the ticket because storage has been integrated back into the core in CKAN 1.6

#2284: Local queries/views/viz in Related stuff.

ross — Thu, 12 Apr 2012 08:18:20 GMT

From #2204

not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ...

### Comments

In the simplest form, right now all I need to store is store the url to the resource plus hashbang config for recline. That would be super easy ...

#2486: Should be able to use . in dataset names

seanh — Sat, 02 Jun 2012 16:46:23 GMT

#2508: Make it possible to run CKAN tests for each language

seanh — Mon, 11 Jun 2012 10:42:15 GMT

Mistakes in translated strings can cause CKAN to crash or otherwise not work, but it's not practical to manually test every page and function of CKAN in every language that we have new translations for before a CKAN release. It'd be great if the tests could automatically be run for each language.

This is probably a big job, we would have to get the tests to respect a language setting in the ini file, check for any individual test cases that specify the language (e.g. in the URL), and also fix test cases that look for specific English words in HTML output, etc.

In the meantime, a good stop-gap solution might be a script that tests for common mistakes in the po files.

#2513: Dataproxy should not default to utf8

ross — Wed, 13 Jun 2012 08:23:40 GMT

Unless explicitly told by the source web server the dataproxy should not assume that the content it has can be encoded as UTF-8. Even though the chars from 128 - 255 overlap an attempt to decode some byte array as utf8 will fail whenever a latin1 char whose bitpattern has the MSB set.

This will mean that the UTF8Recoder can be more rigid in its acceptance of data, Postel aside.

#2531: New state option: archived / deprecated

rgrp — Fri, 15 Jun 2012 08:59:47 GMT

Deleted means things will get purged at some point.

Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated.

@richard cyganiak ...

#2546: ODS Managing homepage content

ross — Mon, 18 Jun 2012 10:54:02 GMT

Requirements

Require the ability for users to control some level of content that is visible on the home page of their ODS installation. This may be through RSS/Atom feeds (see #2234) or another mechanism but should result in admins being able to change blocks of text on their homepage.

This should not be configuration, but accessible through WUI.

Interface

None

User Stories

As a system administrator I want to have control over content displayed on the front page beyond featured/popular items.

As a system administrator I don't want to manage content through having to write an extension.

Tasks

[ ] Analysis

Estimates

#2554: Research Virtuoso cartridges

ross — Mon, 18 Jun 2012 13:37:25 GMT

Look into writing a cartridge for importing CKAN data into a Virtuoso quadstore

http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger#How Does It Work?

#2607: 'Upload a file' appears on resource form when storage not enabled

seanh — Wed, 27 Jun 2012 13:30:08 GMT

if the user tries to upload a file they will get "Failed to get credentials for storage upload. Upload cannot proceed"

Maybe add a test for it this time, this bug has appeared and reappeared before

#2732: New file upload functionality

ross — Mon, 23 Jul 2012 15:53:09 GMT

We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is http://ckan.okfnpad.org/uploads

Simplifying uploads

Currently uploads are too painful/difficult/fiddly to use and/or configure. We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces.

We want to fix:

File uploads themselves
Storage of uploaded files
Notification of the upload to other components
File uploads

Things file upload should do:

Allow sysadmin to disable
Allow auth'ed users to upload
Store whatever they send on disk, and store DB entry linking the file to the person
When creating the resource, the user should be able to choose from all of the files they have uploaded but not yet associated with a resource. This will allow for bulk upload and then a delayed association. Whenver a user creates a resource they either upload a file now, or see previously uploaded files.
Can we do the upload asynchronously and then associate the uploaded key with the resource before the save ? What happens if the user tries to submit before asymc upload finishes ? Should we delay them?
The upload workflow should look like...

File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data.

When called via ajax then the ID of the newly created file should be returned,
When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param.
The resource save should check whether it has a file id and in that case updates the file object to point to the resource.
This should enable:

Separate file upload into a user's temporary store, either individually or as a batch.
Creating resources and simply choosing from previously uploaded, unassigned files
Adding files/data to a resource after the fact.

File storage

File storage should be local to the CKAN install, and not a remote service. Any archiving to remove storage providers should be outside of the main request.

File storage should:

allow moving data, a sysadmin should be able to move the storage root and change configuration and have the system continue running (i.e. don't store absolute paths).
provide maintainability, it should be easy to determine which old files are not associated with resources and thus can be cleaned up.
allow for collection of information (i.e. estimate of storate space used)
check whether there is enough space and handling the conequences cleanly
ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage.
Have a configurable maximum accepted blob size during upload.
Should store what meta-data was provided with the upload, such as mimetype.
Somewhere in the DB we should store ...

Column Notes
id An identifier
owner The owning user, who uploaded the file
path The path (from the 'storage root') to the file
size The size in bytes of the file on disk
mimetype The mimetype of the file, as provided by the uploader
upload_date When the data was uploaded
resource The ID of the resource it belongs to. A unidirectional relationship.
archived_url The URL where this file has been archived

Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files.

Notifications

We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified. The primary use case for this is to notify the component that will translate/upload certain formats to the data store.

We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file).

#2960: Private datasets are not private

dominik — Mon, 08 Oct 2012 11:04:15 GMT

At the moment everyone can read all resources. There is no support for hiding resources that are in private datasets.

#2962: Search across multiple ckan instances

dominik — Mon, 08 Oct 2012 22:50:22 GMT

Could be done by:

using the solr distributed search
difficult set up
merging result sets from apis
make sure that results can be merged properly (score, facets, ...)

#1165: Add multi-site support to ckan

nils.toedtmann — Thu, 26 May 2011 12:41:07 GMT

Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3).

That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's "<VirtualHost?>", or tracd). Depending on the "Host:" header in the HTTP1.1 request, it would choose which local ckan ini file to obey.

I see two ways to constitute the map hostname-to-ini-file map:

ckan reads a set of ini files, and each ini file declares which servers names it is responsible for
In a global ini file, there are directives mapping servernames to ini files.
In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc.

#1457: Bug with DataNL instance

jilly.mathews — Thu, 10 Nov 2011 11:37:21 GMT

"when logging into http://register.data.overheid.nl/ with OpenID, the /user/me page gives a 404. CKAN version 1.3.4."

n the manual it says an API key kan be created via http://test.ckan.net/user/me /. However, when I try the corresponding http://register.data.overheid.nl/user/me, I get a 404 error (not found).

#1596: Refactor authz roles

dread — Thu, 22 Dec 2011 10:31:28 GMT

Suggestions from rgrp:

Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)
(This ticket is split off from #1065)

#1740: Get rid of `from module import ...`

seanh — Wed, 01 Feb 2012 11:06:09 GMT

It's really bad to do from module import * and CKAN has a lot of them. I suggest a three-pronged approach:

Don't add any more of them.
When you're programming if you see an easy opportunity to remove one then do so.
At some point we should task someone to go through the code and remove them all (which is what this ticket is for), but this will be a big job and may break things.
We should also get rid of most or all of the from module import foo and from module import foo, bar statements.

I think the right thing to do is just import module and then use module.foo in your code. But if you find yourself doing module.foo.bar then you may have a code smell.

See: http://docs.python.org/dev/howto/doanddont.html

#1749: Allow creating activity details through API

seanh — Sun, 05 Feb 2012 18:10:06 GMT

Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts.

#1787: [super] Improve RESTful API

dread — Wed, 08 Feb 2012 11:54:42 GMT

Lists of entities should be full URLs, rather than just the names
Discoverability - /api/v3/rest should list the entity types that can be listed
This could be v3 of the RESTful interface.

#1790: Click to delete tags, rather than have all existing tags in the tag text box

dread — Fri, 10 Feb 2012 15:44:34 GMT

From Pablo:

Editing the tags field is clumsy when there are too many tags. Could show existing effectively as tags (like delicious), then allow clicks to delete. New tags added via text box.

#1800: Tidy up *_list() and *_search() functions in ckan/logic/action/get.py

seanh — Thu, 16 Feb 2012 15:12:09 GMT

For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed.

Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality.

#1824: Add vocabulary pages

seanh — Tue, 21 Feb 2012 17:24:49 GMT

For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page.

We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar.

#2202: Display page view count on dataset and resource pages

rgrp — Sat, 03 Mar 2012 09:04:15 GMT

Just like we display download counts we should display view counts.

#2234: Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN templates

seanh — Fri, 16 Mar 2012 12:59:15 GMT

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds
Feed items should probably be cached somewhere
Suggested

The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.
For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference
Caching is pretty crucial and should probably obey the ttl of the feed.

#2235: Group drop down on dataset edit should use chosen and sort groups by name

rgrp — Fri, 16 Mar 2012 14:15:57 GMT

#2258: Customizable contributor agreement

rgrp — Mon, 26 Mar 2012 08:45:49 GMT

Customize text at bottom of forms
Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...)

#2265: 'More Like This' for a dataset

dread — Tue, 27 Mar 2012 16:20:20 GMT

When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar.

SOLR has a feature for finding documents similar to a particular document, called 'More Like This'.

We would like this for DGU.

#2287: Consistent datetime and time period strings localization

seanh — Thu, 12 Apr 2012 18:08:43 GMT

Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support.

For example, on the user index page right now there are time period strings like "less than 1 month" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly).

ISO date format everywhere might be a good idea.

Need to consider local timezone issues.

Python standard library and Babel may have useful helpers for this.

Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page?

#2378: Extract metadata directly from resources that contain it.

ross — Thu, 10 May 2012 11:58:08 GMT

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

#2404: Auto-screenshot for related items

rgrp — Mon, 21 May 2012 06:55:47 GMT

As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere

Implementation: (?) Use phantomjs or something like http://snapito.com/ Investigate for 1.8, will be needed for TDH Oct.

#2552: Controlling access to features

ross — Mon, 18 Jun 2012 11:26:10 GMT

Requirements

To provide a freemium service it is necessary to be able to provide differing levels of functionality based on the type of user (see #2550). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks.

Initial implementation should focus on limiting access to datastore disk space.

Interface

These changes are currently only for the data hub and should be kept as much as possible within the data hub extension.

User Stories

As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited).
As a system administrator I don't expect to need to manage the levels of users or the features that this applies to.

Tasks

[ ] Clarification of requirements/analysis

[ ] Tests

[ ] Code

[ ] Model

[ ] API

[ ] UI

[ ] Documentation

Estimates

#2573: package_search does not allow solr's per-field facet parameters

icmurray — Tue, 19 Jun 2012 13:27:19 GMT

Solr allows its facet parameters to be specified on a per-field basis, eg. facet.limit applies to all facet fields, but solr allows it to be overriden for a specific field, eg. facet.tags.limit.

We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept. See ckan.lib.search.query.VALID_SOLR_PARAMETERS.

#2810: heroku ckan support

kindly — Tue, 07 Aug 2012 18:40:58 GMT

Get ckan working on heroku

#2761: Document all the errors you can get when setting up filestore, and how to fix them

seanh — Thu, 26 Jul 2012 11:36:06 GMT

Add it to a 'Troubleshooting' section on the filestore page: http://docs.ckan.org/en/ckan-1.7.1/filestore.html

For the error messages and their solutions, see various threads on ckan-dev

#2762: test_related.py crashes

seanh — Thu, 26 Jul 2012 11:55:53 GMT

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

#2919: Remove IController from docs

seanh — Wed, 12 Sep 2012 14:47:33 GMT

Still in docs but seems gone from code

#2936: Updating a group via the API clears its packages

seanh — Wed, 19 Sep 2012 09:47:46 GMT

If the group dict that you post to the API does not have any 'packages' key then all the group's packages get removed. I think it would be better if you could just update e.g. the group's description without having to also post the list of packages, and apparently this is how other update API actions work.

Might be worth checking all the update API actions for this behaviour, and making sure they're all consistent

#2958: Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8

seanh — Thu, 04 Oct 2012 09:29:36 GMT

e.g. 2012_08(주요국가).xls

I tested in CKAN 2.0 and it seemed to work fine there.

#2965: Stats extension broken on 1.8

amercader — Mon, 15 Oct 2012 10:15:41 GMT

Graphs not showing (looks like a flot related file is missing)
Wrong groups counts (e.g. Data Explorer Examples show 1800 datasets when it onlu has 8)

#3003: New pg databases should be created with UTF8 encoding rather than system default

amercader — Fri, 19 Oct 2012 09:33:19 GMT

Just to track @tauberer patch on Github. Probably going to 1.8.1

#2763: Multilingual tests failing

seanh — Thu, 26 Jul 2012 11:56:59 GMT

test_multilingual_plugin.TestDatasetTermTranslation?.test_dataset_index_translation, test_multilingual_plugin.TestDatasetTermTranslation?.test_group_read_translation both failing for me on master

#2846: Organizations allows you to set an organization as the parent organization of itself

seanh — Mon, 13 Aug 2012 12:22:16 GMT

that doesn't make any sense

#2881: Remove or improve 'Domain Model' docs page

seanh — Tue, 21 Aug 2012 09:05:17 GMT

http://docs.ckan.org/en/ckan-1.8/domain-model.html

Currently not very useful. Also these pages:

http://docs.ckan.org/en/latest/domain-model-dataset.html

http://docs.ckan.org/en/latest/domain-model-resource.html

http://docs.ckan.org/en/latest/domain-model-task-status.html

This just seems very incomplete to me (surely there is more to the domain model the dataset, resource and task status?) and very likely to not get updated as changes are mode to the code. Would anyone seriously refer to this rather than just looking at the source code?

If we want to have something along these lines in the docs, I suggest adding docstrings to packages, modules, classes and methods and then using autodoc to pull them into sphinx.

#2882: Remove User Stories Overview from docs?

seanh — Tue, 21 Aug 2012 09:18:43 GMT

Why is this here? Is it up to date? Is anyone keeping it up to date? Does this belong in some sort of CKAN planning or roadmap document rather than in the documentation?

http://docs.ckan.org/en/latest/user-stories.html

#2883: Add high level dev overview to 'For CKAN Developers' section of docs

seanh — Tue, 21 Aug 2012 10:52:35 GMT

Here's a draft: https://gist.github.com/3414107

#2884: Lock icon shown next to unknown licenses

seanh — Tue, 21 Aug 2012 16:01:13 GMT

Set the license of a dataset to blank (the first option) or to 'License Not Specified' or 'Other (Non-Commercial)' and a lock icon appears next to the license on the dataset read page. Shouldn't this be a question mark or something?

#2890: Collect data previews and data store docs in one chapter

seanh — Tue, 28 Aug 2012 10:20:36 GMT

Currently there is this page:

http://docs.ckan.org/en/latest/data-viewer.html

which covers Recline Data Explorer and other kinds of data preview in CKAN. It is under the Publishing Datasets section in the documentation. I had to to a search for 'recline' to find it.

Separately there is this page: http://docs.ckan.org/en/ckan-1.7.1/datastore.html which covers datastore, datastorer, and the data api.

I suggest collecting this together in one chapter called 'Data Previews'. If I understand it right the general gist would be:

CKAN has builtin previews of data resources on resource pages, enabled by default.

Images, Google Documents, and web page resources will be loaded into embedded iframes for preview.

Text-like files will be displayed raw.

CSV or Excel files uploaded to CKAN will be previewed using Recline Data Explorer.

Additionally, you can enable CKAN's DataStore?, requires you to install ElasticSearch? and nginx and put datastore.enabled=1 in your ini file. Lets you use the Data API to query data.

Does having DataStore? enabled mean you get preview of more types of resources? Any resource that's available via the Data API will be previewed using Recline,

You can install ckanext-datastorer, and then CSV and Excel files _linked to_ as CKAN resources will be previewed using Recline also. Requires celeryd.

#2891: Update or remove DataStore google presentation

seanh — Tue, 28 Aug 2012 10:23:43 GMT

Currently this page:

http://docs.ckan.org/en/ckan-1.8/datastore.html

has an embedded Google docs presentation about data store which contains some outdated information, e.g. links to datastoreclient which was merged into ckanclient.

Suggest moving the google presentation into the documentation proper, so it works with grep etc.

#2892: IDatasetForm's form_to_db_schema() breaks API?

seanh — Tue, 28 Aug 2012 11:42:05 GMT

It looks like when an IDatasetForm plugin with a form_to_db_schema() method is active, then the posted data dict is validated against the schema even if the user is posting to an API endpoint like /api/action/package_update etc. This breaks the API, for example 'name' and 'title' are optional keys for package_update (you can supply only the 'id'), but the schema validation will fail if either are not present in the posted dict.

It may be we need different schema for package create and package update, currently IDatasetForm has just the one form_to_db_schema() method.

#2893: Tags missing from form_to_db_package_schema()

seanh — Tue, 28 Aug 2012 12:11:04 GMT

With an IDatasetForm plugin active with a form_to_db_package_schema() method that returns a schema based on ckan/logic/schema.py:form_to_db_package_schema() I'm getting validation errors like this:

[ckan.controllers.api] Validation error: "{'type': 'Validation Error', 'junk': [u'The input field junk was not expected.']}"

They seem to be caused when a data dict with key 'tags' is passed to e.g. the package_update API, for example in ckan/tests/functional/api/model/test_vocabulary.py there are various tests that get this error when they call package_update (if you run the tests with a suitable IDatasetForm plugin active).

In ckan/logic/schema.py:form_to_db_package_schema() 'tags' is for some reason deliberately removed from the schema.

#2894: Empty lists get stripped from package dicst in package_show()

seanh — Tue, 28 Aug 2012 12:40:34 GMT

If there's an IDatasetForm plugin active with a form_to_db_schema(), then packages with e.g. no extras get the 'extras' key entirely stripped by the validation in package_show, which causes some tests to crash, e.g. in ckan/tests/functional/api/test_activity.py (if the tests are run with a suitable IDatasetForm plugin active).

If a package has no extras the dict returned by package_show should still have an 'extras' key with an empty list as the value.

Suspect this also effects packages with no tags, no groups, etc.

#2896: resource_create crashes on ValidationErrors with KeyError: 'resources'

seanh — Wed, 29 Aug 2012 16:29:14 GMT

ckan/logic/action/create.py:resource_create():

try:

pkg_dict = _get_action('package_update')(context, pkg_dict)

except ValidationError?, e:

errors = e.error_dictresources?[-1] raise ValidationError?(errors)

This assumes that package_update always returns an error dict with 'resources' in it. This assumption doesn't hold.

#2897: db_to_form_package_schema() crashes historical dataset page

seanh — Thu, 30 Aug 2012 12:41:35 GMT

If there's an IDatasetForm plugin active with a db_to_form_schema() method based on db_to_form_package_schema(), then viewing historical versions of datasets (e.g. http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191) crashes.

Reason is that revison_id and revision_timestamp need to be added to db_to_form_package_schema(), something like:

schema.update({'revision_id': [validators.ignore_missing, unicode]}) schema.update({'revision_timestamp':

[validators.ignore_missing, unicode]})

#2900: snippets/package_list.html template crashes on dataset that has no 'notes'

seanh — Thu, 30 Aug 2012 14:00:07 GMT

Sometimes a dataset dict comes through package_show() with 'notes': None and then the 'notes' key gets stripped from the dict when validation happens (validation only happens if there's an IDatasetForm plugin with a db_to_form_schema() method) and then package_list.html template crashes, e.g. on user profile pages.

Fix in the template? Or in the schema?

The template also crashes on packages with no tags (empty list gets stripped from dict) and probably other empty fields as well

#2912: ckanext/organizations: Neither public or private initially selected when adding dataset

seanh — Mon, 10 Sep 2012 09:57:56 GMT

When adding a dataset with the organizations extension on, neither the Public or the Private radio button is initially selected. Public should be selected.

#2914: Add form_to_db_schema_options() to IDatasetForm and IGroupForm

seanh — Wed, 12 Sep 2012 10:29:55 GMT

This optional method is supported when creating or updating groups and packages and is used in DefaultGroupForm? and DefaultPackageForm?, but seems to be missing from the interfaces. Should be added with docstring.

#3006: Update translations from Transifex

seanh — Mon, 29 Oct 2012 10:22:24 GMT

There are some new translations already on the CKAN 1.8 resource on Transifex already (e.g. 100% Norwegian translation I think). We should at least pull these from Transifex for 1.8.1, and may as well do a call for translations too.

#1745: Dataset search UX improvements as of Jan 2012

rgrp — Thu, 02 Feb 2012 23:22:21 GMT

Changes to make search both more exploratory and more satisfying to use

Search query build - #1603
Ability to add new facet fields "live"
That is add fields which then contain faceted options (a bit like data.hri.fi)
(??) Autocomplete / drop down on search (i.e. search while you type)
Dubious about value / cost ratio here
Probably would involve to pure JS and HTML implementation.

Implementation

Probably require

API changes to expose solr style API directly #1737

#2320: Introduce new state: archived

rgrp — Mon, 23 Apr 2012 20:12:29 GMT

Some datasets become obsolete: e.g. http://datahub.io/dataset/uncdb Do not want to delete them but want them marked as archived or deprecated or something.

#2321: Use resource names/titles in URLs

rgrp — Tue, 24 Apr 2012 03:00:12 GMT

Would make them much nicer but requires:

Ensure uniqueness (or deal with ambiguity)
(?) Slug/names versus titles
Aside: what is currently called name should probably be title

#2324: Link API docs to relevant version not just latest

amercader — Wed, 25 Apr 2012 10:58:52 GMT

The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version.

Also, remove the 'API' link (to /api/1) as pointless and misleading

#2328: Link to Catalog API for each Dataset, Group and Tag

rgrp — Thu, 26 Apr 2012 12:15:01 GMT

At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page?

#2333: Introduce new resource type datastore

rgrp — Sat, 28 Apr 2012 22:16:00 GMT

This is a resource that is *just* in the datastore. At the moment there are hacky ways to do this (create resource with meaningless URL etc) but we should do this properly:

New Option: DataStore? Table in add a resource form
URL field is disabled (or points at /api/data or something rather pointless)
Download button is disabled
Better would be to find a way to serialize an ES table or stream it ...

#2479: Meanings of Author and Maintainer fields are unclear

markw — Thu, 31 May 2012 08:28:18 GMT

CKAN's default schema has fields for Author ('The name of the main contact, for enquiries about this particular dataset') and 'Maintainer ('another important contact person'). The distinction is not clear. Also the fields are often left blank.

The roles that seems most important are those of the original owner/publisher of the data, and the person maintaining the CKAN record/copy of it. So I suggest:

(1) Rename the fields 'Owner' and 'Maintainer'.

(2) Change the explanatory text for the relevant fields:

Owner: 'The person or organisation who create/collect/publish the data in this dataset.' Owner e-mail: 'E-mail address for enquiries to the Owner named above.' Maintainer: 'The person maintaining this dataset on [name of CKAN instance], if different from the above.' Maintainer e-mail: 'E-mail address for enquiries to the Maintainer named above.'

(3) When a logged-in user creates a new dataset, the main form should have a checkbox, checked by default, marked 'I am the maintainer of this dataset'. If checked, the Maintainer name and e-mail fields are populated from the user's profile.

#2480: Better message when dataset has no resources

markw — Thu, 31 May 2012 08:31:33 GMT

If a dataset has no resources the resources list currently says '(none)'.

Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'

#2483: Non-local resources should not have Download links

markw — Thu, 31 May 2012 12:52:14 GMT

At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally.

This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table.

Suggested fix:

Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource)
Disable the Download button unless the resource is stored locally.

#2529: DataHub (or CKAN) widgets

rgrp — Fri, 15 Jun 2012 08:47:29 GMT

Simple widgets in pure JS. For example:

Count of datasets in a group (could generalise to a query but not sure how useful that is ...)
Embeddable list of top X (5) datasets for a given query
Embeddable list of *my* datasets
Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query).

Or: we have this attached to areas of site where relevant.

Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page).

Cf. http://okfnlabs.org/ckanjs/

#2543: facet.sort is not available in the package_search action

icmurray — Mon, 18 Jun 2012 09:55:35 GMT

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

#2585: Escape solr control characters in search queries, add advanced search screen

seanh — Fri, 22 Jun 2012 10:26:13 GMT

Suggestion from David Read:

We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for "Spend over £25,000 - NHS Leeds" then it would not come up with the dataset with that exact name. It was excluding datasets with the word "NHS" due to the dash/minus sign. It works fine if you escape the minus sign: "Spend over £25,000 \- NHS Leeds".

So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine:

http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/

Perhaps you would consider providing this in CKAN core in future?

I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?

#2625: Add i18n strings from non-core but supported extensions to ckan.pot file

seanh — Mon, 02 Jul 2012 13:48:23 GMT

Have to decide which non-core extensions are going to be supported first.

#2838: Context variables accepted by action functions need to be documented

seanh — Fri, 10 Aug 2012 20:35:11 GMT

I was doing this:

context = {'model': base.model, 'session': base.model.Session, 'user': toolkit.c.user or toolkit.c.author, 'extras_as_string': True} group_dict = logic.get_action('group_show')(context, {'id': group_id})
in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass 'extras_as_string': True in the context. I don't think this or other context variables are documented anywhere.

#2885: Labels (editor, admin) not translated in authorization pages for datasets or groups

seanh — Wed, 22 Aug 2012 08:49:11 GMT

#2889: Support JSON values in extras, when returning JSON from API

seanh — Fri, 24 Aug 2012 12:35:37 GMT

For example, edit the warandpeace dataset and add an extra with key "comments" and value "[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]", now visit API URLs such as:

/api/search/dataset?sort=metadata_modified+desc&all_fields=1&limit=10

/api/rest/dataset/warandpeace

/api/action/package_show?id=warandpeace

The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects.

#2869: user profile not linked on disqus until page refresh

shevski — Thu, 16 Aug 2012 10:58:34 GMT

Reported by Daniel:

Problem: DISQUS comments don't appear with profile details (inc. username). How to reproduce: 1) Go to a dataset, e.g. I went to http://s031.okserver.org:2375/dataset/ff 2) I logged into DISQUS using the Google Authentication, e.g. in my case that is okfn.daniel.lewis@… 3) I made a comment and submitted 4) It comes up with the comment, but no linked profile until the page gets refreshed

#2748: add 'add new resource' button to sidebar

shevski — Wed, 25 Jul 2012 10:41:18 GMT

When editing a resource you see the current and any other existing resources in sidebar

(e.g. see http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True)

Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page

#2786: target blank HTML downloads

shevski — Wed, 01 Aug 2012 11:16:54 GMT

e.g. if I click on download here: http://s031.okserver.org:2375/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b can it take me to a new tab instead of navigating away from ckan?

#2795: Check validation of HTML, CSS, JS

toby — Thu, 02 Aug 2012 10:25:28 GMT

Ensure that we are being standards compliant

#2820: English Language: Visualization -> Visualisation

danieljohnlewis — Thu, 09 Aug 2012 09:27:35 GMT

Problem: In the English version (which has a UK flag, indicating British English), the word "Visualization" is used. For an example see the "Filter by type" drop down on the /apps page. Expected: This should be "Visualisation" in British English. Any instances of "Visualize" should be changed to "Visualise" too.

#2821: Featured Items on Filter

danieljohnlewis — Thu, 09 Aug 2012 09:28:24 GMT

Problem: On /apps page in the Filter Results box there is a "Only show featured items" checkbox, on selection it comes up with 0 solutions. Expected: Presumably an admin can create "featured items" so that they can be randomly selected on front page (is this correct)? If there are no "featured items" in the whole database can this check box be hidden? Bug is: no UI or obvious way to create featured items. Also the checkbox looks un-styled

#2854: no way to see which dataset an app is related to on /apps page

shevski — Wed, 15 Aug 2012 16:38:48 GMT

e.g. from http://s031.okserver.org:2375/apps?page=1 you cannot edit a related item as an admin or view what dataset the item is related to

#2898: Looses data entered on step 1 of create dataset after login

johnmartin — Thu, 30 Aug 2012 13:08:14 GMT

To reproduce:

Enter details for new dataset without being logged in
Hit login page for just before step 2
Once logged in you end up back at the empty add dataset form (instead of expected 2nd step of dataset edit)

#2899: Step 2 > Save & add another bug

johnmartin — Thu, 30 Aug 2012 13:15:19 GMT

Goto:

Add dataset
Get to step 2
Click "Save & add another" without adding any data
Ends on step 3
Should: fail to validate and remain on step 2

#2904: Show more formats/groups/tags shouldn't display if there aren't more

johnmartin — Wed, 05 Sep 2012 14:14:04 GMT

On the search results page when there are no more formats, groups or tags to display it should not display the show more links.

#2905: Add dataset URL key behaviour

johnmartin — Wed, 05 Sep 2012 14:28:03 GMT

On step 1 for adding a dataset can we add a ajax ping that checks if the URL that is generated by the JS is taken? Behaviour should be as follows:

User types title in box (as current)
URL key is regexed by JS (as current)
The JS does a ajax call to check if the generated URL key is valid. The ajax callback returns a JSON dump which contains what the URL key should be (e.g. if 'test' was taken it returns 'test-1')
If the URL key returned in the ajax is different than the one generated by the JS then:
4a. If the user hasn't clicked the edit button then: it automatically updates the URL key field with the URL key supplied in the ajax

4b. If the user has clicked the edit button then: a form error next to the input appears saying that the URL key is taken and within that error is suggests the URL key returned by the ajax as alternative (the user can click said URL key within the error to automatically choose that one)

The URL key isn't reserved by the ajax and upon form submission the usual validation should still apply.

#2908: Dataset related with image should retain aspect ratio

johnmartin — Thu, 06 Sep 2012 11:32:42 GMT

At the moment if you have an related item with an image associated to a dataset that appears in the side column it's resized but doesn't keep it's original aspect ratio.

#2909: User profile headers are a bit weird

johnmartin — Thu, 06 Sep 2012 11:35:48 GMT

They seem to be not exactly well thought out. Needs some thought on better display of the information that they contain.

#2910: User listings are not very compelling

johnmartin — Thu, 06 Sep 2012 11:39:47 GMT

They are currently are just a small image / name. They should be: Larger image, Full name (username), description, edits/dataset

#2867: 1.8 url_for error

ross — Thu, 16 Aug 2012 10:57:49 GMT

From: <thedatahub.org@…> Date: Thu, Aug 16, 2012 at 10:30 AM Subject: WebApp? Error: <class 'routes.util.GenerationException?'>: url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6 To: ckan-sysadmin@…, kindly@…

URL: http://ckan.net/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:155 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:131 in call << r'\1', oldpath)

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:239 in call << try:

res = WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

res = WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.storage:190 in file << return fapp(request.environ, self.start_response)

else:

h.redirect_to(file_url)

h.redirect_to(file_url)

Module ckan.lib.helpers:57 in redirect_to << if are_there_flash_messages():

kw__no_cache__? = True

return _redirect_to(url_for(*args, kw))

def url(*args, kw):

return _redirect_to(url_for(*args, kw))

Module ckan.lib.helpers:79 in url_for << # fix ver to include the slash

kwver? = '/%s' % ver

my_url = _routes_default_url_for(*args, kw) kw__ckan_no_root? = no_root return _add_i18n_to_url(my_url, locale=locale, kw)

my_url = _routes_default_url_for(*args, kw)

Module routes.util:265 in url_for << if not isinstance(url, str) and url is not None:

raise GenerationException?("url_for can only return a string, got "

"unicode instead: %s" % url)

if url is None:

raise GenerationException?(

"unicode instead: %s" % url)

GenerationException?: url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6 CGI Variables CKAN_CURRENT_URL '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'utf-8;q=0.7,iso-8859-1;q=0.2,*;q=0.1' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@…)' PATH_INFO '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '37236' REQUEST_METHOD 'GET' REQUEST_URI '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '<address>Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7f5312a51650> beaker.cache <beaker.cache.CacheManager? object at 0x7f5312a51790> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7f5312a516d0>> beaker.session {'_accessed_time': 1345109433.273328, '_creation_time': 1345109433.273328} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: >, ) paste.registry <paste.registry.Registry object at 0x7f5303b05610> paste.throw_errors True pylons.action_method <bound method StorageController?.file of <ckan.controllers.storage.StorageController? object at 0x7f530891fa10>> pylons.controller <ckan.controllers.storage.StorageController? object at 0x7f530891fa10> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7f52f864f850> pylons.routes_dict {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'} repoze.who.logger <logging.Logger object at 0x7f53127b2b50> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 139994769488336>, 'friendlyform': <FriendlyFormPlugin? 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin? 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>} routes.route <routes.route.Route object at 0x7f531296e690> routes.url <routes.util.URLGenerator object at 0x7f52f82cd450> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52f5870cd8> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7f52f82cd450>, {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'})

Column	Notes
id	An identifier
owner	The owning user, who uploaded the file
path	The path (from the 'storage root') to the file
size	The size in bytes of the file on disk
mimetype	The mimetype of the file, as provided by the uploader
upload_date	When the data was uploaded
resource	The ID of the resource it belongs to. A unidirectional relationship.
archived_url	The URL where this file has been archived

CKAN: {9} Agile Backlog

#2412: More than one resource invalidatiing breaks dataset edit form

#277: Set some config options / settings in WUI (extension)

Use case

Implementation

Settings to be in DB

Settings / Options / KeyValues? Table

Loading settings from DB

WUI

Depends

#1358: Generate configuration documentation automatically from the deployment_ini_tmpl file

#1403: Refactor groups index page

#1411: Force resource format to be lower case (also mimetype)

#1507: Minor fixes to dataset add on Group edit form - 0.5d

#1782: Improvements to datasets page UX (Feb 2012)

#2751: check translations for full demo site

#2363: Documentation of best caching practice.

#2326: dataset permalinks

#2433: API uses name not id for some version 3 calls

#2461: Create demo revisions page

#2698: markdown preview

#2775: Add bin/less to paster serve command

#2822: Resource additional info titles format/i18n

#2416: Normalise resource/data types

#2555: Demo site needs a breadcrumb helper

#2823: resource additional info title order

#2938: Explain bundling in doc/resources.rst

#2939: Orgs are groups

#3012: data.gov auth

#2244: clean up auth

#2282: log api calls for users

#2411: .ini file is a mess

#2499: Documentation for the internal analytics

Setup

Related links

#2257: cleanup template vars

#2572: clean up stats plugin

#2579: move sort_by functions into lib.helpers

#2663: h.resource_display_name needs love

#2758: file storage gives error if config not available but no useful user information

#2718: can't add dataset to more than one group

#2721: deleted groups should not show on 'Add to Groups' dropdown

#2728: deleted group shows on search index - for admins

#2780: way for admins to undelete datasets

#2828: Draft datasets are confusing - tickets need creating

#2830: Need method to undelete groups

#2857: no UI to follow another user although option on dashboard

#2861: Add qa breadcrumb

#2862: Revised revised groups description

#2863: non-admin users can edit other's draft datasets

#2873: ckanext-qa: dataset summary incorrect

#2583: make sure that we implement authentication where needed

#2725: Case sensitivity on tags

#2726: confusing logic on data preview formats

#2735: Dataset order on user page

#2768: normalise excel to xls

#2813: Confusing sidebar on demo dataset page

#2362: Improve plugin documentaion, including examples.

#1069: Stub datasets (request for datasets)

Implementation

Related

#235: Resource format normalization and detection

#1041: Start Using the CKAN Wiki for Tutorial-style documentation

#1168: Test system for deb packaging

#2268: UKLP Use same schema as form for harvested records

#143: Most active users listed on homepage

#1382: Deleted resources are present for harvested package

#1287: NAVL validation errors - Junk fields should be listed explicitly

#2644: user dashboard for demo theme

#2814: Demo: upload file behaves oddly

#2956: Allow for resource editing in CKAN 2.0

#2590: Publisher dashboard

#2699: workflow for associating datasets with groups

#2702: Future Javascript wishlist for demo

#2547: ODS Initial data sets

Requirements

Interface

User Stories

Tasks

Estimates