id	type	owner	reporter	milestone	status	resolution	summary	description	PosixTime	ModifiedTime
30	enhancement	zool	rgrp	v0.3	closed	fixed	Tags are added to a package by package owner	"== As A ==

Site User

== I Want To ==

Create/Edit/Update/Delete tags associated with my package

== Notes ==

  * Tags are service-wide and '''not''' package specific
  * Tags should not allow spaces (but allow everything else e.g. ':')
  * web interface details
    * should provide a text box where users can add space separated tags (a la delicious).
    * tag creation should be seemless (so if a tag does not already tag exists it should automatically be created in the background)
    * would like to provide a list of relevant existing tags. For time being could just list all existing tags below entry box but in long run would like auto-updating/auto-suggesting listing a la delicious 

"	1155905753000000	1185472559000000
277	enhancement	zephod	dread	ckan-backlog	assigned		Set some config options / settings in WUI (extension)	"== Use case ==

As a ckan administrator I want to easily change options about the CKAN install.

== Implementation ==

=== Settings to be in DB ===

Suggested:

{{{
## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN

## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png

## Site tagline / description (used on front page)
ckan.site_description = 

## Used in creating some absolute urls (such as rss feeds, css files) and 
## dump filenames
ckan.site_url =

## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico


## An 'id' for the site (using, for example, when creating entries in a common search index) 
## If not specified derived from the site_url
# ckan.site_id = ckan.net

## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net

## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end = 
}}}


NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

=== Settings / Options / KeyValues Table ===

Columns:

 * [namespace]: ? only if KeyValues (for settings this would then always be settings)
 * key
 * label
 * value (json)
 * type (e.g. date and to specify in advance what type should be)
 * description
 * tags: ?? (for grouping ...)

=== Loading settings from DB ===

Do this in ckan/config/environment.py

=== WUI ===

 * /ckan-admin/settings
 * Show label, plus description plus text field

== Depends ==

  * Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)
"	1269274861000000	1318247121000000
978	enhancement	zephod	dread	ckan-sprint-2012-03-05	closed	fixed	Edit Resource including Extras in Web UI	"Part of super-ticket #1506.

Follows on from #826.

We can configure some extra fields in resources and can edit them in the Web UI, but we can't create new columns in the Web UI. Update the WUI to handle tis.
"	1297429619000000	1330547181000000
1004	defect	zephod	dread	ckan-backlog	closed	fixed	Group creation instructions missing	Need instructions on the group page to tell people they need to login to create a group. Someone must have deleted this.	1298559144000000	1323195485000000
1065	enhancement	zephod	johnlawrenceaspden	ckan-v1.6	closed	fixed	[super] Change Authorization System	"Child tickets

 * #1198 Publisher hierarchy
 * #1050 Authz lib improvement and refactor of ckan/lib/authztool.py
 * #1004 Group creation instructions missing
 * #1099 Strange interactions between two browsers while playing with authz groups
 * #1115 can have two authzgroups with the same name
 * #1133 command line rights manipulation doesn't work
 * #1138 minor navigations behave inconsistently

Old ticket description:

 1. Change name of AuthzGroup to UserGroup to reflect what it is for

 2. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
  * Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
  * Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

 3. Change UserGroups so that they can have a hierarchical structure, 

== More info on Hierarchy change ==

e.g. UserGroup NHS contains the User nhsysadmin, as well as the 
UserGroups SURREY and BERKS, which themselves contain users.

One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not.

Each Group has permissions over various objects.

A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively.


Algorithm:

possible(user, action, package):
   if user has permission for action on package
      or any of have that permission
         or any of his groups group-children (but not user-children), and so on recursively have the permission.






"	1301508331000000	1324550041000000
1108	enhancement	zephod	pudo	ckan-sprint-2011-09-12	closed	fixed	Create a more modern theme for CKAN	"CKAN looks a bit aged, it should be styled more modernly and some elements could be re-arranged: 

 * Collect user info in top bar
 * re-add the logo to ckan.net
 * Remove tags from main menu, replace with /sitemap.xml

Inpiration: 

quora.com, github.com, Google Projects, Google Refine, etc.

== CKAN.net or CKAN general theme? ==

To be decided. Suggest we start with ckan.net specific and then backwards integrate (?). Existing ckan.net theme repo:

https://bitbucket.org/okfn/ckan-ckan.net"	1303830790000000	1315140879000000
1264	enhancement	zephod	rgrp	ckan-sprint-2011-10-10	closed	fixed	Merge ckanext-admin into core	"This extension should be in core now -- functionality is basic and very useful.

Copy across from here: http://bitbucket.org/okfn/ckanext-admin

Suggest this no longer even be an extension."	1312480783000000	1318245716000000
1293	enhancement	zephod	rgrp	ckan-sprint-2011-09-12	closed	fixed	Rename Package to Dataset throughout WUI, API	"== Changes ==

 * All templates
 * [DONE] Routing: /package/ -> /dataset/ (including API)
 * [DONE] Tests: self.app.get('/dataset/...') rather than package ...
  * One test to check redirect works?
 * Docs

== Leave ==

 * Stay with Package in code e.g. stay with 'PackageController'
  * also leave stuff like pkg = .... (gradual migration)
 * Leave extensions

== Notes ==

 * Package -> Dataset
 * Data Package -> Dataset
 * respect capitalization"	1314405700000000	1317649968000000
1296	enhancement	zephod	rgrp	ckan-sprint-2011-09-12	closed	fixed	Improved resource adding/editing on dataset/page page	"Subticket of #1294.

 * Nice way to edit all resource attributes
 * Nice way to add new resources
  * Including file upload (cf #878).
   * Cf work in ckanjs (this can be a straight integration).

"	1314406093000000	1315948224000000
1297	enhancement	zephod	rgrp	ckan-sprint-2011-09-12	closed	fixed	In-place preview of edited description markdown on edit dataset form	"Subticket of #1294.

Can integrate from working system on ckanjs."	1314406183000000	1317649974000000
1307	enhancement	zephod	rgrp	ckan-sprint-2011-09-12	closed	fixed	Implement captcha for signup	" * Major issue with spam signup on thedatahub.org
 * Suggest using recaptcha

Est cost: 1d"	1315153860000000	1317649963000000
1309	enhancement	zephod	zephod	ckan-sprint-2011-09-12	closed	duplicate	Registering new user requires immediate login	"Registering a new user directs you to that user's profile page but does not log you in. There is an empty set of ""Recent changes"" and no ability to modify the page. Users must follow the login link in the top-right corner to continue.

Registering a new user should immediately log you in."	1315225510000000	1315586335000000
1313	enhancement	zephod	dread		closed	duplicate	Tweaks to captcha configuration	"Awesome to have the captcha implementation and running on thedatahub.org! Here are some further
suggestions concerning the config options, before this goes into a release:

 * Document ""ckan.recaptcha.publickey"" and ""ckan.recaptcha.privatekey"" in doc/configuration.rst
 * Which one config option defines whether recaptcha is enabled? There is disagreement between the template and the lib. Maybe better to have a third option for this. When setting the syntax for this, I can see us wanting to use captcha for other operations, like package edit, package create, group create etc. so perhaps we'd want to be able to turn each one on/off separately, using the name of the logic action. e.g. {{{ckan.recaptcha.enable = user_create, group_create}}}
 * We got this exception last night from the template:
{{{
Module ?:37 in <Expression u'g.recaptcha_publickey'>
<<          <dd><input type=""password"" name=""password2"" value="""" /></dd>
       
               <dd py:if=""g.recaptcha_publickey"">
                 <script type=""text/javascript""
                   src=""http://www.google.com/recaptcha/api/challenge?k=${g.recaptcha_publickey}"">
>>  <dd py:if=""g.recaptcha_publickey"">
Module genshi.template.eval:324 in lookup_attr
<<                      val = obj[key]
                       except (KeyError, TypeError):
                           val = cls.undefined(key, owner=obj)
               return val
>>  val = cls.undefined(key, owner=obj)
Module genshi.template.eval:410 in undefined
<<          """"""Raise an ``UndefinedError`` immediately.""""""
               __traceback_hide__ = True
               raise UndefinedError(key, owner=owner)
>>  raise UndefinedError(key, owner=owner)
UndefinedError: <ckan.lib.app_globals.Globals object at 0xb8f1182c> has no member named ""recaptcha_publickey""
}}}
I'm not sure why - might be something to do with the maintenance on the server, but worth double checking, since we don't want exceptions, say if one or other config variable is not defined."	1315395185000000	1317076432000000
1320	enhancement	zephod	rgrp	ckan-sprint-2011-10-10	closed	fixed	Extend basic markdown to support normal http / https links	"It is very annoying that you have to wrap links in <...> and most people don't bother. Extend h.markdown in code and markdown in js to fix this.

(We should also support 'package' links etc in javascript?)

*ADDITIONAL*
  * Note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
  * Move the markdown api to /util/markdown


"	1315422000000000	1318164603000000
1348	enhancement	zephod	zephod	ckan-sprint-2011-09-26	closed	fixed	UX polishing (dataset create/view/edit experience)	"Tweaking UIs and polishing the work done in #1294.

## Edit Page ##
  * [Done; cset:49e0b059b6f5] Hide upload if  storage not enabled
  * [Done; cset:81ff13b42dad] Make 'Add resource' links on edit page clearer and more button-like.
  * [Done; cset:03904fc50cbf] Move edit form tabs to the left
  * [Done; cset:fde26b87d6ce] API placeholder text should not link to a CSV file
  * [Done] We used to autocomplete the format field - restore this functionality
    * Works on live. My local db has an empty 'resource_revision' table; maybe 'paster create-test-data' should insert something here?
  * [Done; cset:1aab499d73db] Summary should contain 'name' (or, if unavailable, use description. Long term: description is to be deprecated)
  * [Done] Remove/move instructions where possible (a lot could just be placeholder text on the elements) (other stuff could only show on hover)
    * NB: reveal_instructions.js already disabled on edit page
    * [RP] have now done this on create -- think we can leave on edit
  * [Done; cset:755b1a622199] We have broken 'error' reporting for resource issues. Either need to move that into error summary or somehow not destroy it when we rewrite table on page load ...
  * [Done; cset:7ad090cb7855] Add slideDown to Resource expanded view 
  * [Done; cset:d3595143e506] Need ability to delete resources
    * No thought yet given to uploaded resources. Long term re-think probable.
  * [Done; cset:2430a68a59fc] Empty resource table to resemble view page (""no resources"" text)

## General ##
  * [Done; cset:cc0ad47293e5] Highlight the package tab I'm on (view/edit/history)
    * More generally, the presence of the #minornavigation bar is very inconsistent throughout the site.
  * [Done; cset:d5babc3629cf] Remove 'license' filtered search from backend
  * [Done] Move script stuff to footer of page
"	1316170410000000	1317650043000000
1349	enhancement	zephod	zephod	ckan-sprint-2011-09-26	closed	fixed	Template tidying	"Various things in templates can be tidied up after work has finished on #1294:
  * remove styling in page in group/index.html (put in css main)
    * ditto for package/search.html
    * ditto for home/index.html 
  * Forms.css - to merge into style.css; remove explicit imports elsewhere
    * @import at the top of style.css @import url(forms.css)
  * Make a note: Success/failure controllers clash CSS with success/failure flash banners
"	1316171025000000	1317650050000000
1350	enhancement	zephod	zephod	ckan-sprint-2011-09-26	closed	fixed	i18n of new javascript	"New js code running produced in the work following #1294 contains various string literals, and needs proper internationalisation.

See files:
  * application.js
  * templates.js

Furthermore, efforts can be made to tidy up strings as described in #1329."	1316172576000000	1317650056000000
1351	enhancement	zephod	zephod	ckan-sprint-2011-09-26	closed	fixed	Tidy edit-user form; strip preview functionality	"The template for editing a user's profile seems to output the same form twice in one page.
Furthermore, the page is made needlessly complex by the preview functionality. The neat inline preview used in package will make this page a lot neater.

  * Fix the duplicate form
  * Strip the preview functionality from the front-end, replacing it with js editor
  * Strip the preview functionality from the back-end and test"	1316534753000000	1317650062000000
1358	enhancement	zephod	rgrp	ckan-backlog	assigned		Generate configuration documentation automatically from the deployment_ini_tmpl file	"At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs.

Suggest we write a script that automatedly generates reference documentation for the config from the source.

May be obsoleted by #277 (some config in db)"	1317076350000000	1318257823000000
1360	enhancement	zephod	rgrp	ckan-sprint-2011-09-26	closed	fixed	Remove support for filter by download and filter by openness in search	"Deliver little user benefit and easier just to search and then see what is listed as open and with resources.

== Implementation Details ==

Areas to fix:

 * UI
 * lib/search/...
 * tests
 * ...?

== Cost ==

Est: 2h"	1317077958000000	1317650018000000
1368	enhancement	zephod	zephod	ckan-sprint-2011-10-24	closed	fixed	UX polishing and bugfixes	"From Ira:
PDF of bugs and suggestions is attached.

This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in #1404.

[[ DONE ]]
  * [dataset/edit]: Deletions are confirmed with a dialog box
  * [dataset/edit]: Long URLs break the edit resources tab. Show name (& 'no name') with a pencil icon. 
  * [dataset/edit]: Rearrange the delete resources mechanism; 
    * Use a little minus icon on the far left, followed by a pencil icon for editing.
    * The 'delete resource' button can become 'hide'
    * Deletions are confirmed with a dialog box
  * [dataset/edit]: Rearrange the 'you have unsaved changes' warning; 
    * Delete the little green plus sign on the right;
    * Try a flash banner instead (more general purpose & beautiful)
  * [dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset.
  * [dataset/view]: Subscribe button (& subscribe to history) doesn't work as expected. Modal dialog carrying instructions?
  * [dataset/view]: Drop the 'add to a group>>' and 'add tag>>'. Needs inline tag-adder.
  * [markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out.
  * [dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels.
  * [groups/add]: Hide unimportant fields
  * [navigation] Add 3D bevel (@see github) to current tab highlight.
  * [dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see http://groupspaces.com/a/group/
  * [groups/view]: Show description and owner. To match dataset section.
  * [groups/authz]: Tidy to match admin/authz (pretty buttons & spacing). Add navigation bar.
  * [groups]: Entire section to use pretty buttons please.
  * [about]: Get the about page working on test.ckan.org. (Works locally and on live).


[[ WONTFIX ]]
  * ~~[dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold.~~ (wontfix)
  * ~~[groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'.~~ (ticket #1403)
  * ~~([groups]: (Investigate) Is it easy to implement a little search-groups feature?)~~ (ticket #1403)
  * ~~[*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...)~~ (wontfix: we need to rethink this mechanism; authorization refactor is coming)
  * ~~[general]: 'Authorization' refactor to 'Permissions' on nav-bars.~~ (wontfix, further discussion. authorization might be less techy...)
  * ~~[search]: (Investigate) Can we quickly & easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets?~~ (wontfix: high cost and maintenance overhead; need compelling user story)
"	1317391960000000	1318847910000000
1369	enhancement	zephod	zephod	ckan-sprint-2011-10-10	closed	fixed	Remove showdown.js from the page download	ckan.js can be rearchitected to avoid instantiating showdown.js in the root namespace. That way, layout_base.html doesn't need to import the script (because we don't use the showdown-dependant parts of ckan.js).	1317392516000000	1318165086000000
1377	defect	zephod	zephod	ckan-sprint-2011-10-10	closed	fixed	Ckan admin repair	"Integrating ckanext-admin into core has thrown up a number of problems:
  * Look & feel does not match the rest of the site
  * Tests are not passing
  * On the trash page, clicking 'undelete' triggers a purge
  * Using the purge functionality is dangerous; deleting and purging the latest revision will corrupt a dataset (& several corrupt datasets have been found on thedatahub.org)
  * Trash page can contain nested form tags in certain cases (breaking test harness & form redirection)

"	1318240018000000	1318245795000000
1378	enhancement	zephod	zephod	ckan-sprint-2011-10-24	closed	fixed	New to TheDataHub? - StackOverflow style banner	Implement a banner across the top of the screen giving newbies a link to the about page, and making them feel welcome. See stackoverflow for reference?	1318247329000000	1324047835000000
1379	defect	zephod	zephod	ckan-sprint-2011-11-07	closed	fixed	Markdown conversion of linked references broken	"eg. http://thedatahub.org/dataset/us-gov-tarp

[This Paper][my-paper] is very interesting.

[my-paper]: http://somelink.com/paper

Write a test to check this markdown feature works - if not, use external markdown lib rather than webhelpers.markdown. Probably need to add markdown lib to list of requirements."	1318247719000000	1320161439000000
1380	enhancement	zephod	zephod	ckan-sprint-2011-11-07	closed	fixed	Making customizing CSS easier	"While integrating the new theme and making a large number of UX tweaks, I've allowed the CSS file to bloat out of control with some very specific rules and subtle hacks, as well as a couple of clearfix <div> objects in the markup which isn't really good practice.

In general we want a logical CSS structure which is as easy to modify as possible. You should not find yourself being overwritten by highly specific rules all the time!

== Actions ==

 1. Clean up css
 2. Add a config option ckan.template_head_end - arbitrary string inserted at end of <head> tag so you can add custom css etc (may also want to update theming docs to reflect possibility of using this)
 3. (While we're at it): Add ckan.template_footer_end config option to deployment.ini_template. (This already exists, we just haven't told anybody)
 4. The /users index page isn't linked anywhere. Clean it up and link it in the footer next to Groups etc.
 5. package/new, package/edit, group/new and group/edit all include their forms differently. Use h.literal in all cases.
 6. Fix the padding on minornavigation.

== Background ==

David Read sent me the following as background:

V. basic theming is described here: http://docs.ckan.org/en/latest/theming.html

Deep customisation of the templates is done by creating an extension, with this as a template: https://bitbucket.org/okfn/ckanext-exampletheme/overview

That's what Augusto and the Brazilians have done http://alpha.dados.gov.br/dados/ and their code is here:
hg clone http://dev.dados.gov.br/codigo/dev/tema-ckan
<snip>
Another thing of interest was us becoming compatible with Wordpress themes last December. Richard Pope worked with Rufus on this. I believe the base theme is 'twentyten' and I'm guessing that one would swap that css file with another to change theme, but I don't know. I don't know if this feature has been dropped since then.
http://lists.okfn.org/pipermail/ckan-dev/2010-December/000073.html

Dave"	1318263956000000	1324048324000000
1403	enhancement	zephod	zephod	ckan-backlog	new		Refactor groups index page	"Groups are listed alphabetically with paging - not an ideal user experience. We would like to list groups in order of 'popularity': The number of datasets they contain.

Following this chain of thought, then, it would be nice to rearrange the groups table by clicking on column headers and having it sort by that column.

Furthermore, then, we'd like to implement a full-fledged groups search feature (if this is at all feasible).

The forthcoming groups refactor will probably have some bearing on this task.

"	1318847512000000	1318847566000000
1404	enhancement	zephod	zephod	ckan-v1.7	closed	wontfix	Search Page UI improvements	"[Refactored] :: Follows on from #1506 UX changes.

  * Declutter the sidebar. No yellow box.
  * Facets to go on the left, rather than the right. More logical flow.
  * Did you know you can search by author? Probably not. Find a nice way of presenting extended search options.
  * Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results).
    * Update that look-and-feel to include the new resource icons created in #1506
"	1318847818000000	1338203639000000
1409	enhancement	zephod	zephod	ckan-sprint-2011-11-07	closed	fixed	Add Gravatars to user profiles	"Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)


[Reopened: Additional]
Use gravatars in the HTML generated by helpers.py to create lists of users. Eg. on user/list and dataset/history."	1319200773000000	1324047722000000
1410	enhancement	zephod	zephod	ckan-v1.5	closed	duplicate	Add Gravatars to user profiles	Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)	1319200774000000	1319213205000000
1411	enhancement	zephod	rgrp	ckan-backlog	new		Force resource format to be lower case (also mimetype)	"Format should be lowercase. Automatically lower case (for extra points have a bit of javascript to force lower case when entering).

 * Even more points: do a update on thedatahub repo to make all format lower case (or script this as an update?)"	1319319604000000	1319319604000000
1425	enhancement	zephod	dread		closed	fixed	Dataset delete button	"From Jonathan gray: 

At the moment to delete a package you have to go to 'state' and change from 'active' to 'deleted'.

For me this was not very obvious, and someone recently wrote to me about this too.

Its a small issue - but unless you know what 'state: active' refers to, its not obvious what this means, nor that this is where to go if you want to delete a package.

From UI perspective this could be replaced by either a trash can or some text that says 'delete', perhaps with some prompt that explains what happens when you delete a package - explaining that non-admins won't be able to see it, and how you purge, etc.
"	1319649860000000	1330085282000000
1445	enhancement	zephod	rgrp	ckan-sprint-2011-12-05	closed	fixed	Resource View page in WUI	"Super ticket: #1032

 * Locate at:  /dataset/{dataset}/resource/{id}

See: http://wiki.ckan.org/Dataset_View_Page


Implemented in branch feature-1445-resource-view.

Still to do:

 * Add inline data explorer to page"	1320665049000000	1330019916000000
1450	enhancement	zephod	johnglover	ckan-sprint-2011-12-05	closed	fixed	Simplify dataset view page	"See http://wiki.ckan.org/Dataset_and_Resource_Pages_Proposal

 * Create a collapsible section for the README
 * Reduce the number of things on the page
 * Tidy up groups and tags in the sidebar
 * Resource view becomes a list of search results

(additional)
 * Resource navbar item links to first resource
 * Resource navbar has a dropdown menu of all resources in the dataset
 * Edit and Authz navbar items should be in a separate logical group to the right

Very rough summary: create more info section and move stuff out (relayout page e.g. README, tags etc)"	1320677788000000	1330019974000000
1506	enhancement	zephod	rgrp	ckan-sprint-2012-03-05	closed	fixed	[super] UX improvements to dataset edit, create and search + group + user page (Dec 2011)	"== Historical Parent Tickets ==
  * ~~Parent-ticket: #1450~~
  * ~~Parent-ticket: #1032~~

== Step Zero ==
  * ~~COMPLETELY REDESIGN resource edit process~~

== Dataset Edit-Resources ==
  * ~~Force format field to be lowercase~~
     * ~~Modify backend to serve it up in that format~~
  * ~~Subticket #978: Add extra fields to resources in Web UI~~
  * ~~Fix display of large datasets (http://test.ckan.net/dataset/europeana-lod)~~
  * ~~When submitting forms with errors, open the correct resource pane.~~
  * ~~No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10~~
  * ~~Resource description field to become the markdown editor we use everywhere else.~~

== Dataset Settings == 
  * ~~Subticket #1648: Clarify edit terminology~~
  * ~~Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).~~
  * ~~Fix (i.e. allow / show) name editing on dataset edit.~~
    * ~~Fix the slugify so it does not overwrite if i change title after fixing name.~~
    * ~~Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'~~
  * ~~Autocomplete the author/maintainer box (requires API extension?)~~ [WONTFIX what? it's not like that field is linked to users of the site or anything]

== Dataset View ==
  * ~~Fix animation glitch in expanding description~~
  * ~~Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?~~
  * ~~Verify that there are no more (none) placeholder <td> objects (i believe they've been refactored out)~~


== Dataset Create ==

  * ~~""Upload or link data files, APIs and other materials related to your dataset.""~~
  * ~~Include the description markdown editor again (keep it small)~~
  * ~~Licence should maybe be restored~~
    * ~~""(Don't worry if you don't know what your licence is)""~~
  * ~~Show the log message field~~
    * ~~Beautify~~
  * ~~Pre-populate the log message field?~~
  * ~~? Show Home Page field when creating dataset~~ [WONTFIX don't push it]
  * ~~?  Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation?~~ [WONTFIX doesnt make sense with new resource editor awesomeness]

== User pages ==

 * ~~Improvements to user page UX - 1d - #1396~~
 * ~~Broken links for non-Gravatar use icons - 0.25d - #1719~~

== Etc ==
  * Take before and after screenshots"	1323088306000000	1330702859000000
1507	enhancement	zephod	rgrp	ckan-backlog	assigned		Minor fixes to dataset add on Group edit form - 0.5d	"Group edit dataset add form needs some work

  * Dataset name is not cleared when you add
  * No way to remove item from list of datasets to be added if I make a mistake
  * (2nd Apr 2012) It now seems that option to add multiple datasets at once has disappeared (perhaps during the CSS/HTML refactor ...)
"	1323088429000000	1338205220000000
1517	defect	zephod	zephod	ckan-sprint-2011-12-05	closed	fixed	Accessing resource view sometimes auto-downloads the resource	"Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file. 

For example, being here: http://test.ckan.net/dataset/europeana-lod
and clicking on any of the links to the first resource."	1323109647000000	1323175957000000
1521	enhancement	zephod	shevski	ckan-sprint-2012-01-23	closed	fixed	[super] Groups UI and UX improvements in v1.6 from Cygri	"Various UX and UI improvements to groups.

Etherpad: http://ckan.okfnpad.org/feature-1521-groups-ux

== Improved groups ==

From Richard Cyganiak.

This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as http://ie.ckan.net/ could be turned into a simple group on the [http://thedatahub.org Data Hub] without losing their separate identity.

 * ~~'''Group homepages:''' Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.~~
 * '''Subdomains for groups:''' Enable subdomains such as {{{ie.thedatahub.org}}} as aliases for {{{thedatahub.org/group/ireland}}}
 * ~~'''Search within the group:''' Have a big search box that searches just within the group on the group page, with a look like on http://thedatahub.org/dataset~~
 * ~~Have an '''Add a dataset to this group''' link on the group page, perhaps in the sidebar.~~
 * ~~'''Navigation within the group:''' Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from http://thedatahub.org/dataset~~
 * ~~'''Group documentation:''' Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.~~
  * ~~The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.~~
  * Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines ''About'', ''Frequently Asked Questions'' and ''How to get involved'', we'd then get three links in the sidebar that point to {{{datahub.org/group/ireland/about}}}, {{{datahub.org/group/ireland/frequently-asked-questions}}}, {{{datahub.org/group/ireland/how-to-get-involved}}}
    * RP: WONTFIX this is being considered for future (generally useful) but  cost/benefit here was low particularly as it wasn't clear what optimal implementation was.

== Additional ==

 * ~~Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)~~
 * Better group index page (sort by number of dataset, allow searching ...) - #1403
  * Add search to groups: http://thedatahub.org/group
  * Way to filter by activity, number of datasets
 * #1597 '''Tag search, filtered by group'''
  * RP: not worth it in terms of benefit/cost IMO

== Steps ==
 * ~~""Add Dataset"" button on action bar~~
  * ~~Make it a dropdown? -> New Dataset / Existing Dataset~~
  * ~~Redirect Add Existing -> Edit~~
  * ~~Check permissions~~
 * ~~Expandable ""about this group"" box~~
 * ~~Implement simplest search-this-group box~~
  * ~~turn the list of datasets into a dynamic list of search results~~
     * ~~pagination~~
  * ~~Matching set of facets in the sidebar (matches 'real' search page)~~
    * ~~Deleteable, displayed facets~~

== Nice-To-Have ==
 * Get rid of the old group_package_show action (confirm with team)
 * Tabbed ""about this group"" box (extract headers?)
 * ~~Scrollable group members widget in the sidebar~~"	1323169961000000	1330675930000000
1528	enhancement	zephod	zephod	ckan-backlog	closed	fixed	Gravatar next to My Account link on all pages	"Rather than the anonymous 'user' icon in the top-right corner, we could display your gravatar on all pages when you are logged in. 

This was suggested by Richard Cyganiak:
http://lists.okfn.org/pipermail/ckan-discuss/2011-November/001825.html"	1323197931000000	1330020444000000
1530	defect	zephod	lucychambers	ckan-v1.7	closed	fixed	URL field help text in wrong place	"Confusion between entering values in the title and name/url fields in datacatalogs.org.

The help text ""Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"" applies to the name/url field, yet in Lucy's browser the text that is actually hovering over the title field in my browser.
"	1323261984000000	1338202731000000
1583	enhancement	zephod	johnglover	ckan-sprint-2012-02-06	closed	fixed	Integrate QA into frontend - 1d	"Tasks:
 * ~~On dataset view page, add star rating after each resource view url. Probably similar to current format label after url.~~
 * ~~Add star rating to resource view page, along with a way of displaying the reason for the rating and tips to improve. Should be unobtrusive.~~
 * ~~Fix the table at http://thedatahub.org/qa/dataset/broken_resource_links - URLs should be truncated to a fixed not-too-long length and the other fields line up (eg by using a table)~
 * ~~Handle 404s by showing that 'we think this resource is 404ing'.~~
 * ~~Handle the absence of QA data on a resource (currently nothing is displayed. Maybe that's fine.~~
 * ~~Do we need to integrate the 'reason for rating' string into the UI?~~

Next Steps:
 * ~~Use db dump of thedatahub.org for final testing.~~"	1324459327000000	1327582702000000
1603	enhancement	zephod	rgrp	ckan-v1.7	closed	duplicate	Search query builder	"Super ticket: #1745

Ability to build up search query using a nice javascript-y interface.

 * Add facets by selecting attribute and adding -> search facet options in dropdown -> added to search (with 'x' to remove -- as we currently do).
  * (a bit like the data.hri.fi)
 * Some improvements to css
  * See http://eutr.ep.io/
 * Improvements to faceting
  * Ability to configure faceting and number of items to show (?)
 * Pure JS search implementation to make it easy to reuse across site"	1325268364000000	1338202654000000
1646	defect	zephod	dread	ckan-sprint-2012-01-23	closed	worksforme	Resource navigator options display spuriously	"When viewing a dataset, the ""Resources"" navigation button contained the Resource titles on the Resource navigator button, instead of in a drop-down mouse-hover menu.

http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f

Chrome/Linux"	1326452700000000	1327407044000000
1716	enhancement	zephod	shevski	ckan-sprint-2012-02-06	closed	fixed	move or change save warning on edit dataset page [a:0.1d]	To allow navigation between sections to still work, also potentially rephrase to avoid confusion that you have to save every time to add a tag or resource and can't say add description, tag and resource in one go	1327599893000000	1328530370000000
1735	enhancement	zephod	dread	ckan-sprint-2012-02-20	closed	wontfix	"""Unsaved changes"" warning briefly when you click ""Save"""	"Super ticket #1744

If you edit a dataset, simply add a tag and click on ""Save"" then you get the message ""You have unsaved changed. Make sure to click 'Save Changes' below before leaving this page."" inserted whilst the submission takes place. (This is because the click moves focus away from the tag field). This (reasonably common) edge-case could usefully be squashed.

Also note typo: ""unsaved changed"""	1328009420000000	1328609804000000
1744	enhancement	zephod	rgrp	ckan-sprint-2012-03-05	closed	fixed	UX improvements to dataset edit as of Dec 2011	"Super ticket: #1506

  * ~~Show edit summary form again, at least on edit pages (and prefill it for create to something sensible like: Creating dataset {title} {name})~~ [Done #1506]
  * ~~Fix drop down for resource types in resource add/edit to limit to fixed set of options (as per http://wiki.ckan.org/Domain_Model/Resource)~~
    * ~~See also http://lists.okfn.org/pipermail/ckan-discuss/2011-December/001860.html~~
  * ~~Allow reordering of resources (or: and this is somewhat more complex but possibly simpler: allow editing of a resource_order field and we use that to sort resources (this is more like wordpress page_order -- these two options could become the same if we just hijack the existing position field on the join table and get rid of ordereredlist extension))~~ [DONE #1506]
  * ~~Radically simplify set of licenses~~ [DONE #1359]
  * ~~""Unsaved changes"" warning briefly when you click ""Save"" #1735~~
  * ~~""Unsaved changed"" typo on dataset edit page~~
"	1328224628000000	1330349472000000
1782	enhancement	zephod	rgrp	ckan-future	new		Improvements to datasets page UX (Feb 2012)	" * Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?]
 * ~~Icons for *types* of resources in resource list~~
 * List of 'Owners' (administrators) at top right of sidebar (a la group page)
  * Much prefer owner to administrator ..."	1328609441000000	1338205232000000
1788	defect	zephod	zephod	ckan-sprint-2012-03-05	closed	fixed	IE7 compatibility problems	"From Antonia Acuno, via Ira:

It appears that the current layout for the datahub does not work in
IE7, which would mean a problem for us as most gov uses it and is one
of those we have to support...any views?

Tested in several gov departments computers, so a common enough issue.

http://imgur.com/ip4xa

From Rufus:
We haven't officially supported IE7 so far but can put engineering
effort into this if absolutely needed. I note that IE 7 is now 3y out
of date and is likely a security risk. Obviously this is something to
ask Antonio about."	1328799250000000	1333133365000000
2223	enhancement	zephod	rgrp	ckan-sprint-2012-03-19	closed	fixed	Switch to bootstrap from blueprint	"Why?

 * We already use quite a bit of boostrap
 * We already plan to use bootstrap JS
 * Bootstrap is excellent and pretty light"	1331384974000000	1332842692000000
2286	enhancement	zephod	zephod	ckan-sprint-2012-04-16	closed	fixed	Upgrade to boostrap v2.0.2	"I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from TheDataHub core, for example.)

 * Switch to the latest minifed CSS + JS
 * QA-sweep the site, especially dataset/edit"	1334251818000000	1338205222000000
2412	defect	zephod	icmurray	ckan-sprint-2012-06-25	assigned		More than one resource invalidatiing breaks dataset edit form	"When attempting to add more than one resource at once, if more than one resource invalidates, this results in a js error, leaving the form in an inconsistent state.

Repro:

 1. Go to /dataset/new
 2. Add a new resource.  Fill in one of the fields with an invalid value. eg - last_modified, or size...
 3. Add another resource, doing the same thing: make one of the fields invalid.
 4. Try to save the dataset.
 5. The entered resource information will be lost, and a js error ""Uncaught Error: Can't add the same model to a set twice,: backbone.js:586"" will be thrown."	1337717148000000	1338203234000000
986	defect	wwitzel3	thejimmyg	ckan-v1.4-sprint-2	closed	fixed	Broken link report from the ckanext-qa code	"Should have the following features:

* A list of all packages with broken links on the resource
* Under each packages a description of any resources with broken links to include:
  * Link
  * Description
  * Format
  * First found to be broken date

[If you can't do the last one yet, don't worry]"	1297683994000000	1297812401000000
164	enhancement	wwaites	dread	v1.0	closed	fixed	Creating RDF data from CKAN	"Related to:
ticket:90 Link to RDF version of CKAN data and
ticket:163 (regularly convert CKAN data to RDF).

Convert ckan data to RDF with suitable ontology and put on semantic.ckan.net."	1256052643000000	1271251422000000
449	task	wwaites	johnbywater		closed	invalid	Start email thread about RDF in CKAN with JF and WW		1282299482000000	1311182945000000
489	story	wwaites	johnbywater		closed	fixed	Notify RDF service of model events		1282426051000000	1294416189000000
496	requirement	wwaites	johnbywater	ckan-v1.4-sprint-2	closed	fixed	Provide a mechanism for exporting GEMINII metadata via a CSW interface		1282427307000000	1299164106000000
537	task	wwaites	wwaites		closed	duplicate	Caching and Performance improvement	"There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests.

Introducing caching has to be done carefully and should be done in a ''standards compliant manner''.

== General strategy ==

 * Where possible, cache output within the pylons app (beaker).
 * Facilitate external caching in an end-user's web browser or a caching proxy
 * Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes).
 * When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data.

== Tasks ==

These tasks should be broken into sub-tickets:

  * caching of parts of templates that are expensive to render (package list, tag list, group list)
  * caching of entire output using beaker particularly for API read operations.
  * need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches.
  * cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: http://www.mnot.net/cache_docs/#CACHE-CONTROL
    * Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well
    * Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner

== Future Work ==

  * Investigate ckanclient library maintaining a local cache as a web browser would
  * Investigate using a CDN like Google Storage or Amazon for serving cached data."	1283184362000000	1311178929000000
539	requirement	wwaites	rgrp	ckan-v1.4	closed	invalid	The system shall improve performance		1283242447000000	1303118486000000
540	story	wwaites	rgrp		closed	fixed	Implement caching in a systematic manner		1283242705000000	1302694845000000
541	enhancement	wwaites	rgrp		closed	fixed	Full-page caching even for logged in users	"The simplest way to improve performance on specific pages is to implement full page caching.

This, however, has challenges on pages with user-specific content. For example, on CKAN when a user is logged at the top of the page we have ""You are logged in as: {username}"". Similarly we change the visibility of some material depending on the status of users:

  * Visibility of Edit or Authz link on packages and groups"	1283243139000000	1294924872000000
542	task	wwaites	rgrp		closed	fixed	'Show and Tell' page customization using cookie + js	"To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: http://www.ibm.com/developerworks/web/library/wa-rails2/

In the first place we just need to apply this to:

  * You are logged in as
  * Package/group edit/authz tabs"	1283244432000000	1283278942000000
543	task	wwaites	rgrp		closed	duplicate	Investigate partial page caching and edge-side includes	"Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page.

Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish.

Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat."	1283244784000000	1311178918000000
694	bug	wwaites	dread		closed	fixed	No postgres tools for current version	Database for all CKAN instances upgraded to Postgres 8.4, but none of the eu machines were upgraded with the tools necessary to administer them.	1286977052000000	1287087916000000
754	enhancement	wwaites	johnbywater	ckan-v1.4-sprint-5	closed	wontfix	Filtering for harvesting sysem	This has once again been ruled out for UKLP for the timebeing.	1288040469000000	1303118020000000
766	requirement	wwaites	johnbywater	ckan-v1.3	closed	duplicate	The system shall support harvesting metadata from ArcGIS	"Perhaps 80% of the users of the UKLII may be using
ArcGIS."	1288177933000000	1296593519000000
786	requirement	wwaites	johnbywater	ckan-v1.3	closed	duplicate	The system shall support deleting a registered harvest source		1289210982000000	1294410004000000
789	requirement	wwaites	johnbywater	ckan-v1.3	closed	duplicate	The harvest source form shall validate location field by positive identification of source type		1289212106000000	1296593257000000
801	enhancement	wwaites	johnbywater	ckan-v1.4-sprint-3	closed	duplicate	Get/Set last harvested time for a given harvest source	Implies that last harvested time must be stored on source by harvesting job (see #802).	1289483546000000	1300196714000000
844	defect	wwaites	wwaites		closed	fixed	bad mail config ckan.net/ckan.org	"ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example:

{{{
B30E6D9F       2873 Sat Dec  4 14:05:40  MAILER-DAEMON
(delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out)
                                         no-reply@ckan.net
}}}

both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null"	1291556753000000	1296340486000000
851	defect	wwaites	wwaites	ckan-v1.3-sprint-1	closed	fixed	Link Checking	"revisit link checker from http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py

revisit ollyc's parallel work

look at how this gets used in practice (maybe deprecate in favour of curate tool (suitably wrapped to hide details from the user)"	1291722860000000	1293025112000000
853	enhancement	wwaites	wwaites	ckan-v1.3-sprint-1	closed	fixed	Client upload to storage without having primary storage keys	"Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys.

This would lead to an extension to OFS.

This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN).

 * master ticket #852"	1291723063000000	1294594581000000
856	defect	wwaites	dread		closed	invalid	Tests for caching	Test coverage for ckan.lib.cache is only 40%. Can that be usefully improved?	1291724064000000	1311177085000000
860	defect	wwaites	wwaites	ckan-v1.3	closed	fixed	full text indexing for semantic.ckan.net	...	1291726029000000	1291726067000000
861	defect	wwaites	wwaites		closed	invalid	unicode errors creating rdf output	http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d	1291726125000000	1311168845000000
862	defect	wwaites	wwaites	ckan-v1.3	closed	fixed	deploy rdf on catalogue.data.gov.uk	handle various dgu extras, configure apache for autonegotiation, etc.	1291726237000000	1291726261000000
879	enhancement	wwaites	rgrp	ckan-v1.3	closed	fixed	Storage auth API	"API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension."	1292868509000000	1316030482000000
883	enhancement	wwaites	wwaites	ckan-v1.4	closed	fixed	uklii harvesting refactor	"== master ticket ==

 * #884 implement harvest job delete command for an easier life
 * #885 move csw code into dedicated ckanext plugin
 * #886 move forms api into dgu extension module
 * #887 move harvesting code to use generic harvesting module
 "	1293277713000000	1300196622000000
885	defect	wwaites	wwaites	ckan-v1.3	closed	fixed	reimplement csw module to use owslib		1293278099000000	1294253508000000
886	defect	wwaites	wwaites	ckan-v1.3	closed	duplicate	move uklii forms api to dgu module	doesn't belong in ckan core	1293278145000000	1294916538000000
887	enhancement	wwaites	wwaites	ckan-v1.4	closed	fixed	move harvesting to generic module	in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat	1293278215000000	1300196600000000
894	defect	wwaites	wwaites	ckan-v1.4	closed	invalid	harvest from http://locationmetadataeditor.data.gov.uk/geonetwork/srv/csw		1294253876000000	1300196388000000
930	defect	wwaites	dread		closed	fixed	call_timing files created for every request	"On a production server we produce literally millions of little files saying how long every single request took to process.

They are here:
{{{
hmg.ckan.net.2/pylonsdata/call_timing
}}}
They were added by ckan/lib/base.py in cset:da438a9085d3.

I don't believe anyone uses these stats."	1296061721000000	1297066292000000
952	task	wwaites	pudo	ckan-backlog	closed	invalid	CKAN should run under nginx/uswgi	"
second part of #908"	1296730498000000	1310125204000000
988	task	wwaites	thejimmyg	ckan-v1.4-sprint-2	closed	fixed	Get a better HTML view of a GEMINI document using XSL		1297685975000000	1299493598000000
1084	task	wwaites	wwaites	ckan-v1.4-sprint-6	closed	fixed	ckan.net RDF links changed	"need to make some changes for the links to semantic.ckan.net. it should use http://semantic.ckan.net/record/<package_id> now

append .rdf, .ttl, .nt, .dot, .json (even .html for an ugly table)  to taste (or just leave off the suffix and let content negotiation take care of it)

the base url is changed, but it now uses id not name.

see for example:
 * http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33
 * http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33.ttl"	1302616717000000	1304934534000000
1206	defect	wwaites	dread		closed	fixed	"""Content-Type json"" header scuppers package POST"	"Compare these two requests to create a package:
{{{
curl http://test.ckan.net/api/rest/package -d '{name:""test""}' -H 'Content-Type: application/json' -H 'X-CKAN-API-KEY: tester'
curl http://test.ckan.net/api/rest/package -d '{name:""test""}' -H 'X-CKAN-API-KEY: tester'
}}}
The second one gets the payload through (ckan log):
{{{
Retrieving request params: UnicodeMultiDict([('{name:""test""}', u'')])
}}}
But the first one causes a ServerError because the payload (name:""test"") doesn't make it to request.POST or request.params:
{{{
Retrieving request params: UnicodeMultiDict([])
}}}
The only difference is the ""ContentType: application/json"" header, which seems a reasonable thing to include. Javascript lib backbone.js (for example) inserts this automatically.

So why does this header cause the payload to not get through to the request object?"	1309344348000000	1309450216000000
202	enhancement	tonfa	dread	v0.11	closed	fixed	Internationalization (i18n) try-out	"Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment:
http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

== Steps ==

 1. i18n templates
   * Extract content using Babel
   * Get genshi translation working
   * Do the translation!
 2. i18n all strings in controllers (if any)
 3. i18n forms"	1260179192000000	1265892368000000
1370	enhancement	toby	rgrp	ckan-sprint-2012-04-30	closed	fixed	[super] Social sharing for datasets (and resources)	"Add a way to share datasets and see mentions of datasets on twitter (and elsewhere)

As a visitor I want to share a link to a dataset I have found. I also want to see how many others have shared this (or mentioned it).

 * Sharing platform: simplest option is probably twitter but could generalize to e.g. sharethis system which supports, facebook, twitter etc etc.
  * Will need a config option for relevant API key e.g. ckan.sharethis.apikey
 * Location: Put this at top of sidebar on dataset view (a better suggsetion?)
 * Should show how many mentions / shares there are. (very important!)

== Discussion ==

 * (Tom and Ira and ...): do we really want this, versus e.g. proper follow extension?
 * What about bringing conversation back in to CKAN. E.g. show all times this dataset (i.e. its url) was mentioned on twitter. If we do this I think this should be definite +1.
 * Also, given our users, I think just doing twitter (identica) may be sufficient (how many people want to share links to datasets on facebook?)
"	1317422686000000	1336045983000000
1387	defect	toby	johnglover	ckan-v1.8	closed	fixed	Not specifying a protocol in dataset url causes broken link to be served on dataset view page	"If a user enters a url with no protocol (eg: www.data.gov.uk) in the 'url' field of a dataset, a broken link is created on the view page. 

CKAN should probably either put in a http:// if no protocol is given, or validation should fail."	1318428893000000	1343042311000000
1586	enhancement	toby	dread	ckan-v1.7	closed	duplicate	'Sort by' field for package search	"We default to sort packages search results by ""search rank"", but we can change it to be ""name"", ""title"", ""metadata_modified"" date, ""metadata_created"" date. It works in the API now, so let's have (say) a drop-down in the Web UI for this.

Follows on from #191."	1324479588000000	1337079419000000
1653	enhancement	toby	ross	ckan-sprint-2012-03-05	closed	fixed	Change URLs for multilingual site	"To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session.

* Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs.

* Write Middleware + update url_for to take account of the language. [2d]
 

* Document the language setup, and how to replicate it. [1d]

"	1326710590000000	1329845387000000
1792	enhancement	toby	kindly	ckan-sprint-2012-03-05	closed	fixed	[super] api to use same validation and logic as wui forms.	The api currently uses the default validation schemas regardless of the form you specified on the front end.  There should be a way to post through the api using the same validation rules.	1329146487000000	1333037710000000
1803	enhancement	toby	toby	ckan-sprint-2012-03-19	closed	fixed	clean up routes		1329483497000000	1332333091000000
1804	defect	toby	dread	ckan-sprint-2012-03-05	closed	fixed	CKAN mounted at URL - changing language problem	"e.g. http://189.9.137.65/dados/ clicking on Deutsch link is {{{http://189.9.137.65/dados/locale?locale=de&return_to=%2Fdados%2F&hash=1dc17c315c419df850da0dd3599eefa9da76fbeb}}} and redirect goes to {{{http://189.9.137.65/dados/dados/?__cache=97995106}}} so /dados/dados/ when it should be /dados.

Affects CKAN 1.6b only (not yet released)."	1329484956000000	1330347185000000
1806	task	toby	toby	ckan-sprint-2012-03-05	closed	fixed	make drop down language selector	language chooser should look like the ec one	1329741393000000	1331295310000000
2220	enhancement	toby	toby	ckan-sprint-2012-03-19	closed	fixed	make ckan cacheable	"need to sort cookies etc

should work with nginx and varnish"	1331228643000000	1338205222000000
2227	enhancement	toby	toby	ckan-sprint-2012-03-19	closed	wontfix	Tests are slow and broken	not all tests run and they take too long :(	1331556570000000	1332843126000000
2244	enhancement	toby	toby	ckan-future	new		clean up auth	"* cache admins etc
* aim to move check_access so independent of logic function
* general spruce up"	1332177771000000	1338205402000000
2251	enhancement	toby	kindly	ckan-sprint-2012-04-16	closed	fixed	Internal analytics for ckan.	"Page views and Resources clicks need to be tracked.

== User Stories ==

US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.

US1a Next to a resource or a dataset see how often it has been downloaded / viewed

US1b I want to see datasets or resources ranked by most downloaded or viewed

US1c See a trend graph for a dataset (and resources)

Adminstrative Dashboard (?)

    * I want to see the traffic breakdown by country to my site ...
    * Ditto for browser type, language, etc etc
    * I want to see it graphed over time ...

== Implementation Details ==

1. How do we store this data in CKAN?
2. How do we track (and store)?
3. How do we display

 * Config option ckan.status.enabled = False (by default)

=== Storing Data ===

How does ckanext-googleanalytics do this? Current table:

package_id | count_recent | count_total

Move to a new stats_summary table

id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value

Do we store this data into the search (solr) so we can search by it?

=== Displaying Data ===

 * Helper functions / dictize:
  * Helper function: h.stats_get(object_type, id, stats_type)
   * h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
  * Change to dictize
 * Location in the default theme (do we show for example in search results too!)
 * Support for ranking by most popular in search?

=== Tracking Data ===

 * Our own solution (just write to site_tracking)
 * Google analytics (plus extension for retrieving data) <- would need a refactor
 * Piwiki

==== Own Solution ====

(For later: not as part of this ticket probably)

site_tracking table

id | url | timestamp | action (page_view, resource_download) |

 * Make javascript to make request to ckan to store clicks and page views.
 * Add middleware so these requests do not go through pylons and just store data quickly.
"	1332331029000000	1336046155000000
2253	enhancement	toby	toby		closed	wontfix	CMAP [super]	"Somewhere for CMAP stuff not in other tickets

need to create some general tickets

* template changes
* general demo server setup
"	1332341133000000	1340038490000000
2254	enhancement	toby	toby		closed	fixed	better hooks for templates	"TD look at issues and then email ckan-dev

meeting notes from 21-3-2012
general ideas non specific

Problem:
* We want to remove the Transformer usage due to performance and its close tie to genshi
* Want a standard way for core code and extensions to put code into the templates where *templates request it* (templates control insertion rather than extensions ...)
* Stacking versus overwriting on entry points
    * Relationship to h.requires('jquery', version) - http://codex.wordpress.org/Function_Reference/wp_enqueue_script
Options:
* placeholders or helper methods
* {{user _info}} {{sub_menu}} is this equivalent to <span class=""insert-user-info""></span>
    * [xyz arguments] -> h.load_snippet('xyz', ...)
* ${h.snipit()}
"	1332341333000000	1338210316000000
2256	enhancement	toby	toby		closed	fixed	remove TDH from core	make core independent of TDH and make core as minimal app	1332513199000000	1336046098000000
2257	enhancement	toby	toby	ckan-v1.9	new		cleanup template vars	"look at reducing what is pulled into templates eg ckan.lib.helpers
ensure that these changes don't break existing extensions etc"	1332513307000000	1340097071000000
2261	enhancement	toby	kindly	ckan-sprint-2012-04-02	closed	fixed	All links need to option to change where language is placed in url.	"For the ecportal case we need links to be like the following

(%host)/open-data/en/data/(.*)[[BR]]
(%host)/open-data/fr/data/(.*)



"	1332762702000000	1332840112000000
2282	enhancement	toby	toby	ckan-future	new		log api calls for users	"This was part of #2251 but is now it's own ticket.

we need clarity on this.  The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some?  do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me."	1334074725000000	1338210829000000
2289	defect	toby	seanh	ckan-v1.8	closed	fixed	Flash messages are not being translated?	"There are a couple of flash messages in ckan/controllers/user.py that show up in English, even if another language is in use, and even though the strings are marked for translation with gettext and have translations in the po and mo files:

""[USER NAME] is now logged in""

""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"""	1334255836000000	1341314151000000
2302	enhancement	toby	ross	demo phase 3	closed	fixed	Simple theming	"Needs re-testing after the ODS re-theme.
====

Simple image upload, text changes and css updates for a single instance.  All done on a single configuration/admin page

See http://ckan.okfnpad.org/simple-themes for user stories

[x] Initial version will only take an image_url rather than allowing an upload.

[x] New page with form for title, tagline, image loader etc.

[x] Allow the user to paste in values for the CSS, do we really want a colour swatch?

[x] Generate local CSS override which should *somehow* override colors in the original CSS.

[x] Allow user to provide text for an about page.
"	1334582628000000	1344012491000000
2311	enhancement	toby	toby	ckan-sprint-2012-04-30	closed	fixed	disqus - improvements	"get working on TDH

look at removing stream filters"	1334598853000000	1336046186000000
2326	enhancement	toby	toby	ckan-future	new		dataset permalinks	"create permalinks

eg \dataset\~a5-DQ\nice-name

~XXXXX defines the permalink the nice-name is ignored

* packages table needs a new field for this data

* existing datasets need updating

* new datasets need the identifier creating

* routing needs updating

* needs to work for all urls like edit

Once this is implemented we can decide when we will use permalinks vs the normal urls."	1335371946000000	1338212799000000
2355	enhancement	toby	toby		closed	fixed	[super] make extensions cleaner	"* Do an analysis of why upgrades are problematic
* What can we do better
* Decide what to implement"	1335885051000000	1338212946000000
2362	enhancement	toby	kindly	ckan-v1.9	accepted		Improve plugin documentaion, including examples.	Improve documentation for plugins and add examples. Could use ckanext-example as a base.	1335888791000000	1342085420000000
2363	enhancement	toby	kindly	ckan-v1.8	new		Documentation of best caching practice.	Need better documentation on best practices in making page loads faster for non logged in users.	1335889017000000	1340099794000000
2371	enhancement	toby	toby	ckan-v1.9	closed	fixed	fanstatic integration	"aims

* allow templates to specify needed css/js

* allow split js files to be bundled (including plugin supplied if possible - stage 2)

* serve minified in production/ non-minified in dev

* possibly look at doing the minification via paster - maybe as different ticket"	1336046411000000	1345209675000000
2375	enhancement	toby	aron.carroll	demo phase 5	closed	fixed	Create new base theme	"As designed by Sam Smith -> https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/
Also to be used as the demo theme although I think they will diverge at some point. At the moment they should be implemented as one and the same."	1336469009000000	1343317873000000
2411	enhancement	toby	markw	ckan-future	new		.ini file is a mess	The .ini file needs cleaning up in assorted ways which Toby will expand upon.	1337686900000000	1338210872000000
2416	enhancement	toby	aron.carroll	demo phase 5	accepted		Normalise resource/data types	"Currently we have far too many types that are essentially the same format. The new demo theme is using icons for common types. So we need to normalise the common formats into pretty strings. e.g. application/json, JSON, .json and json all should be output as “json”

See the following Basecamp thread for UI examples and discussion.

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056"	1337792946000000	1342617802000000
2417	enhancement	toby	aron.carroll		closed	fixed	Clean up output for dataset search results	"Currently due to the data structure the search result filters are output incorrectly. Repeating the facet with each tag.

See http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa

It outputs:

Tags: fibre Tags: terrestrial

It should be:

Tags: fibre terrestrial"	1337793213000000	1337864155000000
2420	enhancement	toby	toby		closed	fixed	@depreciated decorator	"create a depreciated decorator to wrap functions we intend to remove

logs a warning
maybe check that 'depreciated' is included in the docstring too"	1337858144000000	1337950422000000
2433	defect	toby	toby	ckan-v1.8	new		API uses name not id for some version 3 calls	see https://github.com/okfn/ckan/pull/5	1337957648000000	1340099820000000
2440	enhancement	toby	aron.carroll	demo phase 1	closed	fixed	Create demo homepage	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments

Implementation:

http://s031.okserver.org:2375"	1338206405000000	1340373461000000
2445	enhancement	toby	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo dataset add related page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments

Implementation:

http://s031.okserver.org:2375/dataset/adur_district_spending/related/new

Needs a new endpoint at /dataset/DATASET/related/new this should have a form containing the following fields.

 * Title (required)
 * Type of item (Application|Visualisation)
 * Description
 * URL (required)
 * Image URL

When submitted if  an item is created it should redirect (303) back to /dataset/DATASET/related with a flash message saying ""Related item was successfully created"".

If failed to create it should redirect (303) back to the form and populate the error messages, see add dataset for examples (https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/package/new_package_form.html)

"	1338210652000000	1338982754000000
2454	enhancement	toby	aron.carroll	demo phase 5	closed	fixed	Create demo login page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments

Implementation:

http://s031.okserver.org:2375/en/user/login"	1338211420000000	1343317795000000
2461	enhancement	toby	aron.carroll	demo phase 4	assigned		Create demo revisions page	"I believe this is to be merged in with site activity.

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts

Implementation:

http://s031.okserver.org:2375/en/revision"	1338212055000000	1344543838000000
2482	defect	toby	toby		closed	fixed	related items dataset page 500 error		1338468524000000	1338472932000000
2484	enhancement	toby	toby	ckan-v1.8	closed	fixed	move follower functionality into helper functions		1338547770000000	1340285964000000
2491	enhancement	toby	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Demo related page is missing breadcrumb	"The following pages are missing complete breadcrumbs.

Add related page:

http://s031.okserver.org:2375/dataset/adur_district_spending/related/new

Needs to look like.

Dataset > {dataset_title} > Related > Create Related

Template here: https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/pages/form_page.html#L6

Same deal with Edit related.

http://s031.okserver.org:2375/dataset/adur_district_spending/related/edit/5f9e665b-3e3c-428f-88b1-8554189cf14a

Needs to look like.

Dataset > {dataset_title} > Related > Edit Related

https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/edit.html#L5
"	1338977634000000	1338979614000000
2492	enhancement	toby	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Demo create related form has broken error output	"See: http://s031.okserver.org:2375/dataset/adur_district_spending/related/new

Submit the form to see broken formatting.

eg. [u'Missing value']

[[Image(ckan-broken-error.png)]]"	1338977874000000	1338982678000000
2494	enhancement	toby	aron.carroll	ckan-sprint-2012-06-25	closed	invalid	Demo site has server error for a 404	"Example

http://s031.okserver.org:2375/not-here

Should show the 404 template."	1338988913000000	1338994890000000
2496	enhancement	toby	toby		closed	fixed	get debug fixed in demo branch	stacktrace	1339067709000000	1339418832000000
2497	enhancement	toby	toby		closed	fixed	abort loses language settings	"if not logged in then go to add dataset url
the language is lost

check if this is a ckan 1.7 issue"	1339067855000000	1339421305000000
2498	enhancement	toby	toby		closed	fixed	on loggin redirect to requested page		1339067929000000	1339660986000000
2499	enhancement	toby	icmurray	ckan-v1.8	new		Documentation for the internal analytics	"There's no docs for setting up the internal analytics, nor what it provides.

== Setup ==

Add:  `ckan.tracking_enabled = true` to .ini file

Run: `paster tracking -c dev.ini`

== Related links ==
http://trac.ckan.org/ticket/2251"	1339070297000000	1339070297000000
2501	enhancement	toby	aron.carroll	demo phase 1	closed	fixed	Implement three step dataset creation in demo site	"New form should allow three steps.

1. Add basic fields to create a dataset
2. Add a data resource to the dataset (this can be repeated multiple times using ""Save and add another""
3. Add final metadata to the dataset.

It should be possible to jump backward through the steps by clicking the arrows at the top.
Errors in fields should be validated for each step when the next button is clicked.

Implementation:

http://s031.okserver.org:2375/dataset/new"	1339084829000000	1340373057000000
2502	enhancement	toby	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create Jinja block for common helpers	Most important is {% snippet %} but {% url_for %} would be great too.	1339084936000000	1339494853000000
2503	enhancement	toby	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Look into allowing theme extensions to extend base templates	"For example:

Creating a template in ckanext-demo/templates/base.html containing something like

{{{
{% ckan_extends ""base.html"" %}
}}}

Will extend ckan/templates/base.html
"	1339085096000000	1339580675000000
2512	enhancement	toby	ross	ckan-sprint-2012-06-25	closed	fixed	Bad link on demo site.	On the group read page http://s031.okserver.org:2375/group/nhs the add dataset button goes to the edit group page instead of the add dataset url.	1339518597000000	1339576809000000
2523	enhancement	toby	toby	demo phase 1	closed	fixed	New package metadata form needs creating	"http://localhost:5000/dataset/new_metadata/dataset

Controller : package
Action : new_metadata

template package/snippets/package_metadata_form.html"	1339605176000000	1340368036000000
2532	enhancement	toby	toby	demo phase 1	closed	fixed	show activity steam for user no js	"We should allow the user activity stream to be accessible with no js by passing a param or something.  Need to agree how we hide/show basically agree what classes we use.


This should be a generalised solution that we can use in other places as needed.

I think this just needs discussion"	1339754158000000	1340874456000000
2536	enhancement	toby	toby	ckan-v1.9	closed	fixed	cleanup activity streams		1339765323000000	1347901915000000
2553	enhancement	toby	toby	demo phase 2	closed	fixed	Create a get snippet api action	"just passes template name


?template_name=..."	1340020613000000	1342090078000000
2555	enhancement	toby	aron.carroll	demo phase 5	new		Demo site needs a breadcrumb helper	Something to make building breadcrumbs a bit nicer	1340026983000000	1342618384000000
2559	enhancement	toby	aron.carroll	demo phase 2	closed	fixed	Demo theme add resource has missing functionality	" * Show resources in the sidebar
 * Allow people to go backward and forward
 * Have a back button on each form
 * Lose resource type
 * Add extras as optional metadata
 * Remove save
 * Add back/Previous
 * Add Another"	1340035336000000	1342084940000000
2560	enhancement	toby	aron.carroll	demo phase 2	closed	fixed	Demo theme add metadata is missing functionality	" * Hide remove button when adding new field
 * Finish takes you to the newly created dataset"	1340035384000000	1342084988000000
2566	enhancement	toby	shevski	demo phase 1	closed	fixed	add back buttons to create dataset process		1340039366000000	1340373308000000
2569	enhancement	toby	shevski	demo phase 2	closed	fixed	add other resources to sidebar of resource pages	Allow user to navigate directly from one resource page to another via resource box in sidebar	1340039608000000	1342085235000000
2572	enhancement	toby	toby	ckan-v1.9	new		clean up stats plugin	attempt to disengage the stats plugin from core as much as possible	1340105054000000	1340899524000000
2575	enhancement	toby	amercader	ckan-v1.9	closed	fixed	Licenses have i18n issues (was Inconsistent language on license dropdown)	"License titles at least for some of them need translating.  This wants to be done in a more efficient and simple manner than currently exists.

the original problem was:

The license dropdown on the dataset form shows titles in Swedish by default. Changing the language seems to default to English, but after refreshing the page a couple of times it shows a wrong language again.

Thanks to pudo for reporting."	1340125845000000	1343216544000000
2578	defect	toby	toby	demo phase 4	closed	fixed	demo theme breaks the tests	"The demo branch fails huge numbers of tests.

many are going to be template based - these need to be sorted - ideally we need to get them to work with legacy/new templates but see how we go on that one

anyhow this is not a priority whilst we are branched but needs resolving before we merge into master"	1340276924000000	1347901892000000
2579	enhancement	toby	toby	ckan-v1.9	new		move sort_by functions into lib.helpers	"make these more available but keep existing functionality so not to break any users

remove_field()

drill_down_url()

etc"	1340281798000000	1340899337000000
2581	enhancement	toby	toby	demo phase 2	closed	fixed	allow state = 'draft'	pass some stuff through the context	1340287298000000	1342074635000000
2583	enhancement	toby	toby	demo phase 5	new		make sure that we implement authentication where needed	in development many auth checks may have been lost we need to check they are still working etc	1340359478000000	1342086274000000
2586	enhancement	toby	toby	demo phase 1	closed	fixed	edit dataset - tags broken	tags is list not string	1340361630000000	1340380561000000
2588	enhancement	toby	toby	demo phase 1	closed	fixed	search index for new package	when package is added it does not seem to get indexed	1340373145000000	1340817441000000
2589	enhancement	toby	toby	demo phase 2	closed	fixed	users can view draft datasets	need to add some logic for this	1340373194000000	1342172694000000
2591	enhancement	toby	toby	ckan-v1.8	closed	fixed	postgres unicode problems postgres v9.1.4	"sqlalchemy fails on postgres 9.1.4 when unicode is returned

see http://docs.sqlalchemy.org/en/rel_0_7/dialects/postgresql.html

"	1340627776000000	1340813161000000
2595	enhancement	toby	aron.carroll	demo phase 2	closed	fixed	Demo add dataset needs to remember which stages have been visited	"For example.

 1. Start on stage one
 2. Add dataset click next
 3. Click previous

Expect Stage 1 to be ""active"", stage two to be ""complete""

 1. Start on stage one
 2. Add dataset click next
 3. Click next
 4. Click previous

Expect Stage 2 to be ""active"", stage one and three to be ""complete""

 1. Start on stage one
 2. Add dataset click next
 3. Click next
 4. Click previous
 5. Click previous

Expect Stage 1 to be ""active"", stage two and three to be ""complete"""	1340642246000000	1342086346000000
2596	enhancement	toby	aron.carroll	opendatasuite 1	closed	fixed	Create demo extension	Should have all supported extensions enabled, those that don't work should be disabled and ticketed. Also needs a customised homepage.	1340642315000000	1345209485000000
2597	enhancement	toby	aron.carroll	demo phase 1	closed	fixed	Demo add dataset throws error if tags are empty	" 1. Go to new dataset
 2. Add a slug
 3. Click next

Get error.

{{{
URL: http://localhost:5000/dataset/new
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond
  app_iter = self.application(environ, detect_start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
  return request.get_response(self.app)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/injector.py', line 52 in __call__
  response = request.get_response(self.app)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 152 in __call__
  return self.wrap_app(environ, session_start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/base.py', line 258 in __call__
  res = WSGIController.__call__(self, environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 422 in new
  return self._save_new(context)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 703 in _save_new
  return self.new(data_dict, errors, error_summary)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 430 in new
  data['tag_string'] = ', '.join(h.dict_list_reduce(c.pkg_dict['tags'], 'name'))
TypeError: string indices must be integers, not str
}}}"	1340642399000000	1340644591000000
2606	enhancement	toby	toby	demo phase 1	closed	fixed	make fanstatic use full path	ckan/main not main	1340795985000000	1340799139000000
2609	enhancement	toby	aron.carroll	opendatasuite 1	closed	fixed	Demo theme errors when creating a dataset without a resource	"1. Complete step one of add dataset
2. Click next without adding any resource data.

Page errors.

{{{
URL: http://localhost:5000/dataset/new_resource/asdasd
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond
  app_iter = self.application(environ, detect_start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
  return request.get_response(self.app)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/injector.py', line 52 in __call__
  response = request.get_response(self.app)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 152 in __call__
  return self.wrap_app(environ, session_start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/base.py', line 258 in __call__
  res = WSGIController.__call__(self, environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 495 in new_resource
  get_action('resource_create')(context, data)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/logic/action/create.py', line 240 in resource_create
  raise ValidationError(errors, _error_summary(errors))
ValidationError: {u'URL': u'Missing value'}
}}}

http://localhost:5000/dataset/new_resource/asdasd"	1340810211000000	1340900271000000
2618	enhancement	toby	toby	demo phase 4	closed	fixed	allow nested js resources		1340878063000000	1347901864000000
2620	enhancement	toby	aron.carroll	demo phase 2	closed	fixed	Pass a  related item into the dataset tempate	"See the related sidebar box

http://s031.okserver.org:2375/dataset/adur_district_spending

If a package has related items the first (or a random) related item should be passed into the snippet."	1340897809000000	1342074737000000
2632	enhancement	toby	toby	ckan-v1.9	closed	fixed	check allowed helpers exist	in lib helpers make sure the allowed function lists functions exist or throw an error.	1341827811000000	1342087027000000
2633	enhancement	toby	toby	demo phase 2	closed	fixed	for dev always look for templates	so we find newly created templates	1341828058000000	1341830278000000
2634	enhancement	toby	toby	ckan-v1.8	closed	fixed	change log - helper function limited in 1.8	make sure documented	1341828248000000	1342085656000000
2638	enhancement	toby	toby	ckan-v1.8	closed	fixed	add a get from request param helper function		1341844597000000	1342112772000000
2646	enhancement	toby	toby	demo phase 2	closed	fixed	delete dataset	make it so 	1341920466000000	1342014170000000
2652	enhancement	toby	toby	demo phase 3	closed	fixed	get qa working with the demo		1341937315000000	1345106904000000
2653	enhancement	toby	aron.carroll	demo phase 2	closed	fixed	Requesting file upload authentication fails on demo	"Try: http://localhost:5000/api/storage/auth/form/photo-of-a-duck

Gets:

{{{
File '<string>', line 2 in auth_form
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/jsonp.py', line 26 in jsonpify
  data = func(*args, **kwargs)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 407 in auth_form
  authorize(method, bucket, label, c.userobj, self.ofs)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 200 in ofs
  StorageAPIController._ofs_impl = get_ofs()
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 71 in get_ofs
  storage_backend = config['ofs.impl']
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/paste/registry.py', line 146 in __getitem__
  return self._current_obj()[key]
KeyError: 'ofs.impl'
}}}"	1341938467000000	1342005167000000
2655	defect	toby	shevski		closed	fixed	updating datasets loses data	Editing and saving a resource wipes the data from the dataset. Happened with http://demo.ckan.org/dataset/my-tes I clicked on edit, changed a tag, clicked update and the data was gone!	1341999482000000	1342001885000000
2657	enhancement	toby	shevski	demo phase 2	closed	fixed	hide history button on group pages until retheme	History button in top right of http://demo.ckan.org/group/data-explorer goes to http://demo.ckan.org/group/history/data-explorer and needs hiding	1342007213000000	1342008765000000
2662	defect	toby	shevski	demo phase 2	closed	fixed	adding colons in tags results in server error on demo	"changing a tag from ""country-uk"" to ""country:uk"" and updating gives server error

otherwise updating metadata / adding tags seem fine

either we should allow colons or form should return a message indicating what characters are allowed"	1342017594000000	1342018898000000
2663	enhancement	toby	toby	ckan-v1.9	new		h.resource_display_name needs love	"This function is shit and needs cleaning up and a doc string

description is markdown and should be treated properly

either we should truncate all or leave it to the templates but work universally

url if no name / desc this is in demo-theme branch"	1342017746000000	1342017746000000
2664	enhancement	toby	toby	demo phase 2	closed	fixed	tags with : cause crazy error message on add dataset in demo		1342018503000000	1342632478000000
2671	enhancement	toby	toby	ckan-v1.9	closed	fixed	fix allow_partial_update	need to be able to delete stuff explicitly	1342183618000000	1345209579000000
2678	enhancement	toby	shevski	demo phase 2	closed	fixed	No UI for deleting a related item	"Needs to be a way for admins (owners of dataset) to delete related items. I suggest adding a button to the edit page

http://s031.okserver.org:2375/dataset/gold-prices/related/edit/d3249863-040a-4f55-a5d8-c86697d0b7b8"	1342436087000000	1342685932000000
2685	enhancement	toby	toby	demo phase 2	closed	fixed	delete buttons should be form buttons	stop bots/browsers etc following links and auto deleting stuff	1342510565000000	1342514443000000
2691	enhancement	toby	shevski	demo phase 2	closed	fixed	Tag 404 error	"
404 Not Found The resource could not be found. "	1342538490000000	1342604273000000
2694	defect	toby	toby	demo phase 2	closed	fixed	index page 500 error	"we need to have proper fix for this - not sure if there is some caching issue

this around the group search"	1342605699000000	1342680771000000
2698	enhancement	toby	shevski	demo phase 4	assigned		markdown preview	for description / other fields with markdown support	1342620085000000	1344543252000000
2701	enhancement	toby	shevski	demo phase 3	closed	fixed	add comments to datasets	disqus??	1342620406000000	1343949913000000
2703	enhancement	toby	shevski	demo phase 2	closed	fixed	add new resource to existing dataset	with add button on dataset page and in 'data & resource' section when there's no data	1342620885000000	1342687332000000
2707	enhancement	toby	shevski	demo phase 3	closed	fixed	Tidy up additional info boxes on dataset and resource pages	"This currently looks terrible. Some simple things we should do to improve this.

 * Localise and captitalise known (non custom) keys into human titles. eg. ""size"", ""created_at"", ""mime_type"".
 * Apply units to values of known keys where possible eg. ""size in appropiate units""
 * Convert dates into a more human friendly format. We can avoid localisation hell by using an ISO format like YYYY/MM/DD hh:mm:ss which is still better than ISO8061
 * Move the non user fields to the top of the table.
"	1342621471000000	1344504846000000
2712	enhancement	toby	toby	demo phase 2	closed	fixed	delete resource functionality needed	we need a delete_resource logic action and enable for resources	1342680517000000	1342690104000000
2715	enhancement	toby	shevski	demo phase 2	closed	fixed	inactive sidebar widgets	hide trackbacks & subscribe wboxes on dataset pages until we add disqus	1342714073000000	1342716931000000
2716	enhancement	toby	shevski	demo phase 2	closed	fixed	truncate long resource names	http://s031.okserver.org:2375/dataset/hi-people/resource/4558c365-fd7a-4f0e-9f2a-90ad47db6b39	1342714182000000	1342716474000000
2718	enhancement	toby	shevski	demo phase 4	new		can't add dataset to more than one group	"trying to add a dataset to another group means it's no longer part of the first group

http://demo.ckan.org/dataset/edit/afterfibre"	1342780550000000	1344544203000000
2721	defect	toby	shevski	demo phase 4	new		deleted groups should not show on 'Add to Groups' dropdown	"Groups previously deleted still show up in the add dataset process in step 3 'Additional info'

http://s031.okserver.org:2375/dataset/new_metadata/ff"	1342948632000000	1344544214000000
2724	enhancement	toby	shevski	demo phase 3	closed	fixed	Inconsistent labelling / UX on draft datasets	"1. One of the datasets here: http://s031.okserver.org:2375/en/user/shevski has [DRAFT] after it. It's not clear why this has been added -> user confusion
2. The draft dataset is not clickable in the dataset list
3. The ff dataset (same as the ff [DRAFT] dataset) in the Activity Steam section a) is clickable and b) does not have [DRAFT] after it"	1342949402000000	1343319009000000
2725	enhancement	toby	shevski	demo phase 5	new		Case sensitivity on tags	"My feeling is that 'country-US' and 'country-us' should be the same tag. However currently tags with caps are treated differently 

see http://s031.okserver.org:2375/en/dataset/test-dataset

with TEST and test - there also get indexed twice in the search page"	1342949667000000	1343030773000000
2726	enhancement	toby	shevski	demo phase 5	new		confusing logic on data preview formats	"1. If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this

2. If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28"	1342949927000000	1343030906000000
2727	enhancement	toby	shevski	demo phase 2	closed	fixed	long resource name breaks sidebar navigation box	"see http://s031.okserver.org:2375/en/dataset/test-dataset/resource_edit/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28

& image attached"	1342950093000000	1343053760000000
2728	defect	toby	shevski	demo phase 4	new		deleted group shows on search index - for admins	'test-group', which has been deleted,shows up on main search page under groups - and can be filtered by - see http://s031.okserver.org:2375/dataset?groups=test-group	1342950784000000	1345023944000000
2730	defect	toby	shevski	demo phase 2	closed	fixed	please change pricing footer link	"Pricing <ckan.org/datasuite/solutions> needs to go to <ckan.org/datasuite/services>

http://ckan.org/datasuite/services/"	1342951353000000	1343032612000000
2734	enhancement	toby	shevski	demo phase 2	closed	fixed	once format on resource form has errored, no longer submits form	"Steps to reproduce:
1. go to edit a resource, e.g. http://s031.okserver.org:2375/dataset/ted/resource_edit/a621a6ad-6f1c-46d1-b858-03cdb7a25c07

2. Change the format field to something it doesn't get. e.g. add a comma ""HTML,""

3. Get message: ""The form contains invalid entries:
Last modified: Date format incorrect
Size: Invalid integer""

4. Format field reverts, try to re-save/submit form - i.e. click on ""update resource""

5. nothing happens (same if you even try changing the format field"	1343059061000000	1343127101000000
2735	enhancement	toby	shevski	demo phase 5	assigned		Dataset order on user page	"I think the datasets on user pages http://s031.okserver.org:2375/user/me
should be ordered by latest updated (with most recent at the top) instead of in alphabetical order.

What do you think?"	1343062877000000	1344349245000000
2737	enhancement	toby	shevski	demo phase 2	closed	fixed	change help/prefilled text in format field	"Should be basic format (which is what the datastore wants & what we display to users) e.g. ""CSV, XML, JSON etc""

http://s031.okserver.org:2375/dataset/new_resource/"	1343121262000000	1343136141000000
2738	enhancement	toby	shevski		closed	invalid	Data API	"Why is datastore not enabled for this resource http://s031.okserver.org:2375/dataset/frere/resource/cdeddb39-b0cb-4762-aa5b-f80fc892d4dd ?

Is this going to be a site wide setting? How do new resources get added to datastore? Or are we just using the dataproxy for demo?"	1343123012000000	1343125335000000
2740	enhancement	toby	shevski	demo phase 2	closed	fixed	misleading breadcrumbs	"When adding a resource, the breadcrumb should say ""add data"" or ""add resource"" not ""create dataset"" as you see here:
http://s031.okserver.org:2375/dataset/new_resource/frere

"	1343123274000000	1343137576000000
2743	enhancement	toby	shevski	demo phase 2	closed	fixed	add dataset button within group	"Currently the 'add dataset' button on a group page http://s031.okserver.org:2375/group/another-group-in-the-wall goes to edit group. This is confusing since the interface for this is no way streamlined or helpful.

Can we get this to go to the create dataset process instead (ideally with the group pre-chosen) for now? 

Further on will need to re-think & plan the workflow for publishers which will be a large part of new work"	1343125202000000	1343131092000000
2746	enhancement	toby	shevski	demo phase 2	closed	fixed	help bubble on demo homepage missing	Hey, this is still missing. It's the one with the add dataset text	1343209387000000	1343214031000000
2747	enhancement	toby	shevski	demo phase 2	closed	fixed	Breadcrumb should use title not URL of dataset	"For example here: http://s031.okserver.org:2375/en/dataset/new_resource/gold-prices
The breadcrumb shows ""gold-prices"" which is not the title of the dataset"	1343211742000000	1343215679000000
2750	enhancement	toby	toby	demo phase 3	closed	fixed	javascript translations need ripping	"we need to ensure that translations in .js files are ripper for translation

look at what is available as preference to creating our own"	1343216246000000	1343415279000000
2751	enhancement	toby	toby	demo phase 5	new		check translations for full demo site	need to check everything gets translated - sean did this before so will have info	1343216443000000	1344243046000000
2752	enhancement	toby	ross	demo phase 2	closed	fixed	Navigation highlight missing	When navigating to http://s031.okserver.org:2375/group on the demo server, the Group link in the navbar is not highlighted in the same was as the other links are when visiting those pages.	1343216582000000	1343222618000000
2755	enhancement	toby	shevski	demo phase 3	closed	fixed	user pages very slow to load	navigating to a user page -one's own or http://s031.okserver.org:2375/en/user/ira or http://s031.okserver.org:2375/en/user/shevski, even when signed out is noticeably slower than navigating between other parts of the site; over 2 seconds. Maybe even 3	1343225395000000	1344505304000000
2758	enhancement	toby	toby	ckan-v1.9	new		file storage gives error if config not available but no useful user information	"We get an error which should be improved the actual problem is this but not passed to user

 ⇝ KeyError: 'ofs.impl'
View as:   Interactive (full)  |  Text (full)  |  XML (full)
Module ckan.controllers.storage:2 in auth_form         view
Module ckan.lib.jsonp:26 in jsonpify         view
>>  data = func(*args, **kwargs)
Module ckan.controllers.storage:407 in auth_form         view
>>  authorize(method, bucket, label, c.userobj, self.ofs)
Module ckan.controllers.storage:200 in ofs         view
>>  StorageAPIController._ofs_impl = get_ofs()
Module ckan.controllers.storage:71 in get_ofs         view
>>  storage_backend = config['ofs.impl']
Module paste.registry:146 in __getitem__         view
>>  return self._current_obj()[key]
KeyError: 'ofs.impl'"	1343287709000000	1343287709000000
2768	enhancement	toby	shevski	demo phase 5	new		normalise excel to xls	so that data proxy works	1343319382000000	1344351663000000
2769	enhancement	toby	shevski	demo phase 3	closed	fixed	add site wide related page	"Can we add the related dashboard (http://trac.ckan.org/ticket/2347) to demo - central place to show off all the related stuff added.

"	1343323567000000	1343734083000000
2774	enhancement	toby	aron.carroll	demo phase 3	closed	fixed	Provide a method of passing localised strings into the JavaScript	"We need to be able to do the following:

 * Rip the localised strings from the JavaScript source code. These are strings that look like {{{_('String')}}}, {{{translate('String') }}} and {{{ isPlural('String') }}}.
 * An endpoint that provides the translations in the correct locale for the user.
 * The endpoint should provide the locale as JSON (possibly using http://jsgettext.berlios.de/doc/html/po2json.html or similar tool)

The JavaScript can then read this file on page load."	1343661072000000	1344351626000000
2775	enhancement	toby	aron.carroll	demo phase 4	new		Add bin/less to paster serve command	"Ideally the ./bin/less command would be run when the server is started.

 * Also it would be good to have a paster command to build the production CSS with {{{ ./bin/less --production }}
 * The command could also detect missing node binaries and redirect to the documentation."	1343685686000000	1344543962000000
2780	enhancement	toby	shevski	demo phase 4	new		way for admins to undelete datasets	"Since admins can see deleted datasets - there should be a way for them
a) to know they are currently deleted & not viewable by normal users - ticket: #2779
b) way to undelete such datasets - this ticket

I suggest a button on the edit form instead of the delete button
i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message?"	1343737248000000	1345023811000000
2782	enhancement	toby	shevski	demo phase 3	closed	wontfix	integrate with leaflet for demo background generation	use http://leaflet.cloudmade.com/ properly to generate map backgrounds for demo instances	1343748857000000	1343811113000000
2787	enhancement	toby	aron.carroll	demo phase 5	closed	fixed	Ensure templates_legacy directory is loaded for extensions	Currently the stats plugin has a template and templates_legacy directory. We need to ensure that the templates in templates_legacy are loaded.	1343827782000000	1344543473000000
2793	enhancement	toby	ross	demo phase 3	closed	fixed	Markdown of resource page not rendered as markdown	"As per #2650 (on master/1.8 now)

If the resource has a description it isn't rendered as markdown ..
"	1343896663000000	1343897811000000
2798	enhancement	toby	toby	demo phase 4	closed	fixed	Admin Config reset	"this want to

a) be a link on the left of the form like deletes

b) needs a confirm like delete

c) wants to work with the autoform"	1344012605000000	1345540010000000
2799	enhancement	toby	aron.carroll	demo phase 3	closed	fixed	Translations for demo not picking up {% trans %} blocks	"Also they seem to be missing the odd {{{ _() }}} block. Such in the new package form.

https://github.com/okfn/ckan/blob/2375-demo-theme-development/ckan/templates/package/snippets/package_form.html#L44
"	1344243005000000	1344249253000000
2812	enhancement	toby	markw	demo phase 3	closed	invalid	Groups description needs improving	"The 'What are Groups?' text at <http://demo.ckan.org/group> is a
bit naff. Here is a suggested replacement:

What are groups?

Groups allow you to group users and data together so that they are
easier to manage. For example datasets in a group could be visible
only to group members, or publicly visible but only editable by group
members.

Groups can be used to collect together datasets from a single
publisher, or related to a particular project, for example.
"	1344419953000000	1344524290000000
2813	enhancement	toby	markw	demo phase 5	new		Confusing sidebar on demo dataset page	"On a dataset page on demo.ckan.org, the left sidebar is confusing.

 * It starts with some random links. Actually they are links to groups which the dataset is in, but this isn't clear.
 * The sidebar elements that are actually part of the dataset are 'Datset extent' and 'License', so these should be right at the top (if they belong in the sidebar at all), instead of which they are right at the bottom in the junk part of the page (i.e. probably lower than the bottom of the main page, and hence lower than anyone will scroll)."	1344420206000000	1344445419000000
2816	enhancement	toby	toby	ckan-v1.8	closed	fixed	DomainObjectOperation not in model	"Over-enthusiastic removal of some model items

this breaks qa extension"	1344496621000000	1345040471000000
2822	enhancement	toby	toby	demo phase 4	new		Resource additional info titles format/i18n	"the title for additional info should be translated

capitalised etc"	1344504620000000	1344543985000000
2823	enhancement	toby	toby	demo phase 5	new		resource additional info title order	"Order the items so that none user fields are first from ticket #2707

"	1344504773000000	1344504773000000
2826	defect	toby	danieljohnlewis	demo phase 3	closed	fixed	Null group image	"Problem: Example group ( http://s031.okserver.org:2375/en/group/example-group ) has a broken image, image not found. Link either incorrect, or no image set, if no image set then shouldn't it show a default image?
Expected: There should be an image."	1344505546000000	1344528004000000
2828	enhancement	toby	shevski	demo phase 4	new		Draft datasets are confusing - tickets need creating	"reported as editing datasets incorrect

e.g. clicking on edit here http://s031.okserver.org:2375/dataset/ff
takes you to create dataset page http://s031.okserver.org:2375/dataset/edit/ff

but this is the correct behaviour of a draft dataset


We need to show draft datasets correctly


proper tickets need making for the different issues after review of issues with them - who can see, where, admins and viewing, orgs too etc"	1344506178000000	1344547324000000
2830	enhancement	toby	toby	demo phase 4	new		Need method to undelete groups	need controller action and front-end method	1344509408000000	1344547341000000
2836	enhancement	toby	toby	demo phase 3	closed	fixed	Need extra info for forms	"like we have on the form here

http://s031.okserver.org:2375/harvest/new

you said it partly exists so maybe I just need some training"	1344605931000000	1345023663000000
2847	defect	toby	shevski	demo phase 3	closed	fixed	'show inactive sources' checkbox displaying incorrectly	"On harvest page: http://s031.okserver.org:2375/harvest the checkbox appears above the text and not clear what it refers to/

Beter to relocate to the left (as on publicdata.eu) see screenshots attached.

"	1344864322000000	1345023769000000
2848	defect	toby	shevski	demo phase 3	closed	fixed	no dataset descriptions on front page	http://s031.okserver.org:2375/	1344938505000000	1344947085000000
2849	defect	toby	shevski	demo phase 4	closed	fixed	header tagline misalighned	"should be left alighned, not right aligned. Otherwise tag line disappears left off the page.

See: header here: http://s031.okserver.org:2375/ckan-admin/config"	1344938638000000	1344947130000000
2852	enhancement	toby	shevski	demo phase 3	closed	fixed	better message for trying to refresh inactive sources in harvest	"Currently if you try to refresh (i.e. schedule the harvester to re-run) an inactive source (one who's state is 'withdrawn' - e.g. http://s031.okserver.org:2375/harvest/3feea629-32d8-4431-9957-9f31d32864f1)

you get the error message: ""An error occurred: [Can not create jobs on inactive sources]""

This could be better and not red, e.g. and info box (maybe yellow) saying that ""Cannot re-run harvesting on inactive sources. Please first change the status to 'active'"""	1345048317000000	1345106515000000
2853	enhancement	toby	shevski	demo phase 3	closed	fixed	unclear error message in refreshing harvest source	"Tried to refresh the datagm harvest source to get error message:

""An error occurred: [There already is an unrun job for this source]""

This is unclear and not sure why it created an error. We should have something informative such as ""A harvest has already been scheduled for this source""

Do we have a date/time for when this has been scheduled that we can display?"	1345048482000000	1345104231000000
2856	defect	toby	shevski	demo phase 3	closed	fixed	server error when clicking on deleted app link	"1. go to http://s031.okserver.org:2375/user/okfn
2. Click on activity stream
3. Click on 'Deletion test' related item that's linked to in entries 1 & 2
4. Server error"	1345049684000000	1345107503000000
2857	defect	toby	shevski	demo phase 4	new		no UI to follow another user although option on dashboard	"e.g. second option in left hand sidebar is follow another user' http://s031.okserver.org:2375/user/dashboard but if you click on that and then any user, there's no follow button

to me the dashboard user stuff is highly unfinished and probably should not be included unless someone is very keen for it"	1345049983000000	1345107097000000
2858	enhancement	toby	toby	demo phase 3	closed	fixed	Harvester tweaks	"I've had a look at the new harvesting theme, it looks great, thanks. I
did some minor changes and reworded the errors as per tickets #2852
and #2853, but I kept the messages red, as the only other ones
available (""flash_notice"") are blue and are not very prominent.

Toby, on the old form there was a list of descriptions for all
available harvesters which could be nice to have back. I couldn't find
an easy way with the new form macro, so I don't know how easy would be
to implement

http://i.imgur.com/pzS8z.png"	1345103198000000	1345105842000000
2860	enhancement	toby	shevski	demo phase 3	closed	fixed	target blank link in qa report	can we get the link going to an explanation of the 5 star system on page http://s031.okserver.org:2375/qa/dataset/five_stars to open in new tab? it's leaving demo all together so better to open in new tab than navigate away	1345113826000000	1345120404000000
2861	enhancement	toby	shevski	demo phase 4	new		Add qa breadcrumb	can we add in the breadcrumb navigation to qa pages so that you can go back to http://s031.okserver.org:2375/qa from sub pages such as http://s031.okserver.org:2375/qa/dataset/five_stars	1345114046000000	1345483270000000
2862	enhancement	toby	markw	demo phase 4	new		Revised revised groups description	"Revised text for 'What are groups?' box at demo.ckan.org/group (after discussion with IB re #2812):

What are groups?

Groups allow you to group users and data together so that they are easier to manage. Group owners can assign roles and authorisations, giving each project or department control of its own data publishing.

Users can browse or facet by groups, which could be an organisation (for example, the Department of Health) or topic (e.g. Transport, Health), making it easier to find the data they are looking for."	1345114322000000	1345115072000000
2863	defect	toby	shevski	demo phase 4	new		non-admin users can edit other's draft datasets	edit button shows up for daniel lewis for http://s031.okserver.org:2375/dataset/ff 	1345114479000000	1345201502000000
2864	defect	toby	shevski	demo phase 4	closed	fixed	don't show `create one now` message on other's user pages	e.g. I can see 'You haven't created any datasets. Create one now?. ' http://s031.okserver.org:2375/user/kindly when logged in as shevski	1345114552000000	1345471138000000
2873	defect	toby	danieljohnlewis	demo phase 4	assigned		ckanext-qa: dataset summary incorrect	"

For example see:  http://s031.okserver.org:2375/qa/dataset/five_stars

Which has: Example dataset as a score of: 4 After examination of the dataset (  http://s031.okserver.org:2375/dataset/example-dataset ) it is made up of a CSV and an HTML file.

"	1345126429000000	1345208283000000
2876	enhancement	toby	toby	demo phase 4	closed	fixed	Admin Config changes are not forced	need to make sure these update everywhere when done	1345208451000000	1345470265000000
2938	enhancement	Toby	dominik		new		Explain bundling in doc/resources.rst	"Since it makes sense to bundle resources into one js file that are commonly used together, there should be some explanations on how separate js scripts are merged into one file for production. 

Explanations could include how groups are handled or different resource files."	1348488835000000	1348488835000000
2939	enhancement	toby	toby		new		Orgs are groups	holder for ticket number	1348564034000000	1348564034000000
3012	enhancement	toby	toby		new		data.gov auth		1352393409000000	1352393409000000
235	enhancement	tobes	dread	ckan-v1.9	assigned		Resource format normalization and detection	"Try to gather proper MIME  information for all package resources in CKAN. This is a shared ticket with dcat-tools (https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service. 

Sub-tasks: 

 * Create a Google Spreadsheet with two Worksheets: ""MIME-Mappings"", i.e. ""CSV"" -> ""text/csv"" and ""Name mappings"", i.e. ""text/csv"" -> ""Comma-Separated Spreadsheet"". 
 * Collect and map surface forms from all CKANs
 * Access this via Swiss and apply, store as a PackageResource extra field pending #826 (Resource extras). 
 * Add heuristics for format auto-detections: 
  * Map well-known file extensions 
  * Recognize obvious magic (Zip, Tar)
  * Peek into Zipfile/Tarfiles
 * Define a convention for generic data types (many CKAN packages have only ""Spreadsheet"" defined, either detect specific type or set MIME to */tabular-data or similar)
 * See also: #816 (Autocomplete for the resource format field)"	1263827604000000	1340627624000000
1069	enhancement	tobes	rgrp		assigned		Stub datasets (request for datasets)	"Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki.

We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ...

(Just as we have datasets listed that exist but aren't available ...)

Alternative would be to have a request for datasets subsystem.

I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists.

== Implementation ==

 * Agree a new dedicated tag. e.g. todo.does-not-exist

== Related ==

 * This ticket has clear links to http://getthedata.org/"	1301666919000000	1340632215000000
2710	enhancement	tobes	toby	demo phase 3	closed	fixed	make the api_info.html not look like shit	you know the score	1342624385000000	1343318314000000
1241	task	timmcnamara	timmcnamara		closed	wontfix	Overview video of http://thedatahub.org	Produce a video explaining some of the things which can be done on the datahub.org	1311567813000000	1340632688000000
1107	refactor	tidy-up bitesize	rgrp		closed	fixed	Move package autocomplete from package controller and move to API	"Currently autocomplete method on package controller. This method should be in API (like other autocomplete methods).

Will need to update client code (just forms atm I think)."	1303808480000000	1340632612000000
143	enhancement	thejimmyg	dread	ckan-backlog	assigned		Most active users listed on homepage	Display league of users' recent activity on homepage.	1255010373000000	1312372381000000
282	enhancement	thejimmyg	dread		closed	wontfix	Provide diverts when package name changes	When a package's name is changed, references to the old name (in the WUI and REST) are redirected to the new name (assuming the old name is not being reused by another package).	1270659003000000	1338206417000000
294	enhancement	thejimmyg	dread		closed	duplicate	Add/remove extra fields in Package edit form	Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources.	1271756591000000	1291830960000000
321	enhancement	thejimmyg	johnbywater		closed	duplicate	Delegate authentication to Drupal	"When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model.

The Drupal front-end shall have:

1. Login page - fixed location, can authenticate users, on successful  authentication sets auth cookie and redirects to HTTP_REFERER.

2. Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions.

3. Access denied page - fixed location, static resource, gently 
indicates what has happened, and how to ask for permission.

The CKAN slave edit page shall:

1. Try to detect a Drupal session key (passed as cookie or as request param).

2. Redirect to Drupal login page if no session key.

3. Check authorisation if session key is found.

4. Redirect to access denied page if session key not authorised.

5. Present the Package edit page.

6. Reject unauthenticated or unauthorised edit submissions.

7. Snag invalid edit submissions from authenticated and authorised users.

8. Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end.

"	1274705234000000	1291831399000000
427	task	thejimmyg	dread	ckan-v1.4-sprint-3	closed	fixed	Match licenses from GEMINI document	" * examples of posting a new package
 * example license_id - explain"	1281436611000000	1299164063000000
430	task	thejimmyg	pudo		closed	wontfix	Upgrade CKAN to Pylons 1.0	"Pylons 1.0 has been out for a while and appears to be running stable on a few production sites. It also introduces a few new features, among them the ability to specifiy controllers using the entrypoint mechanism. 

http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py

This might be a bit ugly since the default behaviour of some pylons objects, such as tmpl_context, app_globals and redirect has changed. "	1281962485000000	1298283075000000
435	defect	thejimmyg	dread	ckan-v1.4	closed	wontfix	Switching between ckan instances causes orphaned postgres processes	"Analysis using pg_top reveals orphaed processes:

https://trac.dataco.coi.gov.uk/projects/datagov/ticket/460#comment:6

This followed switching ckan instances on the dgu machine. Do these hang around forever clogging up a db server? Can switching between instances be smoothed to avoid this?"	1282147736000000	1298284084000000
448	story	thejimmyg	dread		closed	invalid	[Assist with] editing packages through Drupal		1282299478000000	1294417061000000
453	task	thejimmyg	johnbywater		closed	invalid	Analysis and planning for UKLII		1282299725000000	1294411374000000
480	requirement	thejimmyg	johnbywater	ckan-v1.4	closed	fixed	Catalogue service shall conform to specification	"Common requirements for running CKAN behind a (e.g Wordpress or Drupal) front-end:

 1. Unrestricted total read-only access to catalogue API for general public (e.g. voluntary organisation).
   - monitored by API key
   - not monitored by API key
 2. Restricted total read-write access to catalogue API for authorized clients (e.g. front-end system, bulk upload clients).
   - restricted by CKAN access controller
   - restricted by HTTP Auth
   - restricted by IP address
 3. Restricted total read-write access to catalogue Web UI for authorized users (e.g. site admins).
   - restricted by CKAN access controller
   - restricted by HTTP Auth
 4. Restricted partial read-write access to catalogue Web UI for authorized users (e.g. group admins).
   - restricted by CKAN access controller
   - restricted by HTTP Auth

[[Image(fig128130.png)]]"	1282422612000000	1300281551000000
482	story	thejimmyg	johnbywater	ckan-backlog	closed	wontfix	API Rate Limiting		1282422756000000	1298284158000000
499	story	thejimmyg	johnbywater		closed	wontfix	Registration and management of UKLII related code lists, controlled vocabularies and other forms of reference/static data		1282432571000000	1296593038000000
563	requirement	thejimmyg	johnbywater		closed	duplicate	Support a minimal CSW server interface or export to GeoNetwork		1284033576000000	1296592472000000
565	requirement	thejimmyg	johnbywater	ckan-v1.3	closed	invalid	The system shall support UKLP code lists		1284033864000000	1294417752000000
569	requirement	thejimmyg	johnbywater	ckan-v1.4	closed	fixed	Harvested GEMINI document metadata validation		1284040221000000	1297347214000000
667	defect	thejimmyg	dread	ckan-v1.3	closed	wontfix	API is slow	"See message and script:
http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html

1500 requests shouldn't take hours."	1285885228000000	1297069121000000
668	defect	thejimmyg	Colin Calnan		closed	invalid	Caching issues on API v1	"It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues.

Steps to test:

1. Modify a dataset on datadotgc.ca, redirects to CKAN 

2. On save, redirects to http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100

3. You can see that the Dataset has not updated correctly. Run a check on the API v1 - http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present

4. Check the v2 of the API - http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present.

5. Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either."	1285953542000000	1311176649000000
676	requirement	thejimmyg	johnbywater	ckan-v1.3	closed	wontfix	The system shall support browsing to correct version of the CKAN documentation for a given service		1286276136000000	1295869107000000
679	task	thejimmyg	dread	ckan-v1.3	closed	fixed	Improve standard package edit form	" * Add help text, drawing from:
   * ca.ckan.net
   * http://wiki.okfn.org/ckan/doc/package
   * http://wiki.okfn.org/ckan/doc/faq
 * Make longer text revealed on click"	1286386050000000	1294166120000000
691	requirement	thejimmyg	johnbywater	ckan-backlog	closed	duplicate	Package Relationships		1286822735000000	1295610145000000
711	requirement	thejimmyg	johnbywater	ckan-v1.3	closed	fixed	Packages shall support both UK government and location information with a common set of attributes	"Previously, a package type attribute was intended to distinguish between package data that came from government depts. and publishers location information (GEMINI). Now there is an intention use a common set of attributes to support data from either source.

Hence, there currently a v.3 of the government attributes, a document about extracting the attributes from Gemini document using XPath, and a comparison documents mapping between governemtn attributes and Gemini attributes.

That means there are potentially four different list of attributes. I'm not sure how to go about reviewing the lists and converging any differences.

But we do want to end up with a list of names, a description of their purpose, and then perhaps the XPath which would reveal the value from a Gemini doc?"	1287580866000000	1296592589000000
739	requirement	thejimmyg	johnbywater	ckan-v1.3	closed	fixed	The system shall support retrieving a copy of the harvested metadata for a given package	...obviously, not all packages will have a harvested metadata document.	1287779710000000	1296592842000000
740	requirement	thejimmyg	johnbywater		closed	duplicate	Get copy of harvested metadata for a given package		1287779799000000	1296592889000000
750	enhancement	thejimmyg	johnbywater		closed	duplicate	Get CSW records modified since given time		1288014402000000	1296592940000000
757	task	thejimmyg	johnbywater	ckan-v1.3	closed	duplicate	Create migration script to add harvest source filter attribute to existing tables		1288040584000000	1296593448000000
764	enhancement	thejimmyg	dread	ckan-v1.5	closed	duplicate	Read-only CKAN Web UI	"Whilse using CKAN web interface, you are not tempted to edit stuff:
 * You know at all times this CKAN is read-only
 * All editing facilities are still seen but greyed-out with an indication why it is."	1288091619000000	1310133334000000
767	task	thejimmyg	johnbywater	ckan-v1.4	closed	wontfix	Harvest metadata records from ArcGIS CSW API		1288178056000000	1304936413000000
770	defect	thejimmyg	johnbywater	ckan-v1.3	closed	wontfix	Improve the paster harvester help command	"Issue 1. Help command without arguments fails:

$paster help --plugin ckan --config development.ini

gives:

EITHER

Command help not recognized

OR

paster help: error: no such option: --plugin

Issue 2: Harvester command with arguments fails.

$ paster harvester
Traceback (most recent call last):
...
    cmd = self.args[0]
IndexError: list index out of range
"	1288190220000000	1296593925000000
778	task	thejimmyg	dread	ckan-v1.3	closed	fixed	Package edit form - name auto-generated from title		1288630777000000	1294416567000000
779	task	thejimmyg	dread	ckan-v1.3	closed	fixed	Package edit form - stylesheet in DGU updated		1288630800000000	1294416581000000
780	task	thejimmyg	dread	ckan-v1.3	closed	fixed	Package edit form - 'preview' button shows preview, with form lower in page	This ticket refers to the form shown in Drupal. This doesn't have a preview function, but we don't have a requirement for it right now, so leaving ticket closed.	1288630829000000	1294660720000000
781	task	thejimmyg	dread	ckan-v1.3	closed	duplicate	Package edit form - 'extra' fields have 'add' and 'delete' buttons	"(Just like resource table, but extra fields don't need up/down buttons)

Nice to have: a blank field is added when you tab from the last filled-in field in the table."	1288630935000000	1294415081000000
784	task	thejimmyg	johnbywater	ckan-v1.4	closed	wontfix	End to end testing for UKLP		1289210251000000	1304936251000000
787	task	thejimmyg	thejimmyg	ckan-sprint-2011-09-12	closed	fixed	Auth API	"Auth Proposal

Use case: We'd like an authenticated and authorized Drupal user to be able to
edit/delete packages using the API

To do this a user would need a CKAN API key so we need some way of Drupal
telling CKAN who a user is so that they can get their API key.

Proposed Implementation
-----------------------

A user visits the CKAN API key page to get their key. Because CKAN is at
catalogue.data.gov.uk (a subdomain of the Drupal site) it can read Drupal
cookies.

If there is no ``DRXtrArgs`` or ``DRXtrArgs2`` cookie, we know the user isn't
signed in so we redirect them to Drupal to sign in.

WISHLIST: It would be really nice if we could pass the URL to redirect back to
Drupal so tha Druapl can send them back to the CKAN API key page

Either way, they get back to CKAN and now the cookies exist. When the first
HTTP request header is sent, CKAN will read the Drupal session ID and then call
a Drupal API, server to server.

TODO: Implement an API on the drupal server which accepts a Drupal session ID
as an argument and returns the username and credentials, but only if the
request if from the CKAN server (perhaps we specify an API key in the request)

If the session is valid CKAN will set its own auth cookie and show them the
page with the API key. CKAN only considers a user signed in if both the CKAN
cookie *and* Drupal session ID are present. If at any time they sign out of
Drupal the Drupal session disappears so they will be signed out of CKAN too.

Now the user has an API key they can use the standard CKAN command line API
tools.

The API key is the only thing the user will need to use the CKAN API. This
poses a problem. What if a user is removed from Drupal but still has a CKAN API
key?

Two solutions:

1. API keys will be set to only be valid for the length of a Drupal session so
   that a user will only be able to use the write API for a little time after
   credentials are revoked

2. CKAN provides an API to allow Drupal to revoke keys

Option 1. seems easier to me. If we choose this we will write a ``get_api_key``
command line tool so that Drupal user can write things like this:

::

    datapkg list_packages --api_key `get_api_key`
 
The ``get_api_key`` function will prompt for username and password and then
perform the steps necessary to get an API key."	1289211035000000	1315821118000000
795	story	thejimmyg	johnbywater	ckan-v1.3	closed	duplicate	"""Match up"" GEMINI2 ""metadata point of contact"" with registered entities"	"TM wants to know how this will be ""matched up"". I don't know whether this always the data provider, or sometimes the publisher."	1289228025000000	1296593361000000
803	story	thejimmyg	johnbywater		closed	invalid	Auto-generate next migration version unit and test stub files	As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually.	1289486678000000	1314031451000000
881	defect	thejimmyg	fccoelho		closed	invalid	http authentication requirement when installing	I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code.	1293021889000000	1296335072000000
884	enhancement	thejimmyg	wwaites	ckan-v1.3	closed	fixed	harvest job delete operation	implement paster subcommand to delete harvest jobs	1293277800000000	1296592140000000
905	defect	thejimmyg	Stiivi	ckan-sprint-2012-02-06	closed	worksforme	Unable to search with accented characters in package names	"I have a package with name: ""Štatistické prehľady agendy cestovných pasov"" and when I try to search for ""Štatistické"" there are no results.

CKAN setup:

- psql DB with UTF-8 tables

- psql search backend

- ckan version: 1.3.1b

Example:

http://verejnedata.sk/package/agenda-cestovnch-pasov

try to search for the first word in the package name.
"	1294939120000000	1328638536000000
906	enhancement	thejimmyg	Stiivi	ckan-sprint-2012-03-05	closed	fixed	Ability to search without accents for accented words	"In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results.

This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents.

For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCII//TRANSLIT).
"	1294939588000000	1330990400000000
920	defect	thejimmyg	hellmann@…		closed	fixed	empty tags are kind of confusing	"http://ckan.net/tag/linguistsic
doesn't have a single entry, but is still kept and displayed.

It should be purged !!!"	1295355280000000	1300319140000000
941	enhancement	thejimmyg	rgrp	ckan-v1.5	closed	wontfix	Submit apps or ideas for apps related to datasets (extension)	The basic purpose of this extension is provide a way to record  'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.	1296403099000000	1310134339000000
963	enhancement	thejimmyg	rgrp	ckan-v1.4-sprint-1	closed	fixed	Package CKAN as a debian package	Package CKAN as a debian package so it is easier to install.	1297076364000000	1298284252000000
993	defect	thejimmyg	jason.kitcat@…	ckan-v1.4-sprint-2	closed	fixed	Link on ckan.net footer needs changing	"The link for About > 'Project Home Page' on the CKAN.net footer should be changed to point to:

http://ckan.org/

The current link is broken.

Thanks,
Jason"	1298154006000000	1298373114000000
1035	enhancement	thejimmyg	dread		closed	wontfix	Form impressions given an ID	"To counter Cross Site Request Forgery attacks, each form generated is assigned a random number in its url, which must be passed when you submit the form.

(Something to consider for the new form mechanism)"	1299857978000000	1338206251000000
1040	defect	thejimmyg	dread	ckan-v1.4	closed	fixed	File system mounted	Public file path code adds a trailing , and thus adds / as a static file app in the Pylons middleware cascade.	1300213855000000	1328806824000000
1041	enhancement	thejimmyg	thejimmyg	ckan-backlog	assigned		Start Using the CKAN Wiki for Tutorial-style documentation	"For example, I will document the following:

 * A specific guide on using mercurial when working with our branching and merging policy
  * DONE - http://wiki.ckan.net/Becoming_a_CKAN_Developer
 * Policy document on how we use tickets, plan sprints and make releases
  * DONE - http://wiki.ckan.net/Becoming_a_CKAN_Developer
 * How to package software as .deb file for project deployment

I'd love if someone else would write:

 * An authorisation tutorial covering the core model, the command line tools and examples of every possible way of using the system
 * A HOWTO guide with screenshots for adding a package
"	1300284715000000	1312372367000000
1050	enhancement	thejimmyg	johnlawrenceaspden		closed	invalid	Authz lib improvement and refactor of ckan/lib/authztool.py	"Refactor ckan/lib/authztool.py so that the relevant methods are independent of the command line interface.

The extracted methods should live in a new file ckan/authz.py. authztool.py should probably move into cli.py and will just do command line parsing and printing and use ckan/authz.py. The updated web gui for authz will also use this code.

Tests should be made. There's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests.

all to go on a branch feature-1050-refactor-authtoolz

== Optional extras ==

 * Rename ckan/authz.py to ckan/lib/authz.py or even ckan/logic/authz.py


"	1300451937000000	1315394117000000
1086	defect	thejimmyg	johnlawrenceaspden		closed	wontfix	no way to delete authorization groups from web interface	as title. 	1302625333000000	1323346552000000
1094	enhancement	thejimmyg	thejimmyg	ckan-v1.5	closed	duplicate	[super] Refactor the Auth System	"Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787.

Two man reasons for the changes are:

 * We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer.

 * Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able.

The first two changes revolve around the is_authorized method, which
is called by the logic layer to ask whether a particular user (e.g.
Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package).

1. The first thing the is_authorized method is a hook to a plugin
which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!)

Reason: authorization can be completely delegated to another system
(or partially)

2. is_authorized method currently takes (username, action, object)
but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model.

3. Rename these two classes to better reflect what they are
 * AuthorizationGroup -> UserGroup
 * Group -> PackageGroup

4. Rename the Editor role to PriveledgeUser since Editors sometimes can't edit.

Although this sounds a bit radical we already have auth extensions.

== Read-only CKAN Web UI ==
(Additional requirement from #764)

Whilse using CKAN web interface, you are not tempted to edit stuff:

 * You know at all times this CKAN is read-only
 * All editing facilities are still seen but greyed-out with an indication why it is."	1303117973000000	1311173649000000
1117	defect	thejimmyg	nils.toedtmann		closed	invalid	"Depend deb package ""ckan"" against ubuntu package ""python-pastescript"""	... otherwise the scripts fails.	1304089619000000	1304277240000000
1124	enhancement	thejimmyg	nils.toedtmann	ckan-sprint-2011-12-05	closed	fixed	push apt package python-ckanext-solr into our debian repository	python-ckanext-solr is already available in http://apt-alpha.ckan.org/datanl-dev, but not yet in http://apt-alpha.ckan.org/debian (that is why we had to [pip-install it for DataGM). Please push into main repo.	1304537793000000	1323168156000000
1168	enhancement	thejimmyg	dread	ckan-backlog	assigned		Test system for deb packaging	"Get buildbot to:
 * build the deb packages
 * install them into a fresh virtual machine
 * run smoke tests on the installed ckan"	1306441994000000	1330990423000000
1176	defect	thejimmyg	dread	ckan-sprint-2011-10-28	closed	fixed	unicode_body exception	"We seem to occasionally get this exception on ckan.net. 

User agents causing this include:

 * Alexa ia_archiver (3/6/11)
 * Maxthon browser (17/5/11)

{{{
Module ckan.controllers.error:32 in document
<<              return original_response.body
               # Otherwise, decorate original response with error template.
               c.content = literal(original_response.unicode_body) or cgi.escape(request.GET.get('message', ''))
               c.prefix=request.environ.get('SCRIPT_NAME', ''),
               c.code=cgi.escape(request.GET.get('code', str(original_response.status_int))),
>>  c.content = literal(original_response.unicode_body) or cgi.escape(request.GET.get('message', ''))
AttributeError: 'NoneType' object has no attribute 'unicode_body'
}}}"	1307351562000000	1314273012000000
1287	enhancement	thejimmyg	dread	ckan-backlog	assigned		NAVL validation errors - Junk fields should be listed explicitly	"When you create a package, but specify a key that is not allowed (e.g. 'relationships') then you get error message:

{{{
{""__junk"": [""The input field __junk was not expected.""]}
}}}

It should mention the actual key which is not expected. e.g.

{{{
{""relationships"": [""The input field 'relationships' was not expected.""]}
}}}

Kindly said that James' version of NAVL was better in this respect, so this might be best solved by moving to that."	1314203102000000	1330990459000000
1382	defect	thejimmyg	thejimmyg	ckan-backlog	new		Deleted resources are present for harvested package	Perhaps the importer deletes them before re-importing. We shouldn't have deleted resources, so let's investigate.	1318337889000000	1318337889000000
1415	enhancement	thejimmyg	nils.toedtmann	ckan-sprint-2011-12-05	closed	fixed	Comments on current status of ckan deb packages	"This is a scratch pad ticket with some comments on the current status of our ckan deb packages. I know that some of it is the deb packaging roadmap anyway, please forgive me if i mention them here again.

Rufus and me re-deployed some community ckan instances onto s022 (see http://trac.okfn.org/ticket/926). We followed the documentation 
http://docs.ckan.org/en/latest/install-from-package.html 

 * Deb package version number: the version of the deb package is ""python-ckan 1309471251~149be76faabc+lucid-1"", and it's hard to guess from there that it contains a ckan 1.4.2a
 * When is 1.4.3/1.5.x expected as deb package?
 * There was a bug in the DB upgrade script /usr/share/pyshared/ckan/migration/versions/029_version_groups.py (line 150) which looks like it was fixed 1.4.1==>1.4.2 but was nevertheless present in this deb package.
 * The current script /usr/bin/ckan-std-install
  * does not set the Apache ServerName according to the $INSTANCE variable
  * automatically configures a ckan extension named after $INSTANCE
  * depends on local postgres
  * could be replaced with ""/usr/bin/ckan-deploy --name=ckan-std --domain=ckan-std.localhost (see next point)
 * (i think this is exactly James' plan): have more generic deployment script /usr/bin/ckan-deploy as part of python-ckan which takes arguments like
  * --domain=cc.ckan.net
  * --aliases=$list-of-domains
  * --name=cc (defaults to ""domain"")
  * --no-db (does not configure a DB)
  * --sql-alchemy=$DB_CONFIG_STRING (also runs ""paster --plugin ckan db upgrade --config"")
  * --extension $list-of-extesions 
  * ..."	1319457069000000	1323167941000000
1467	defect	thejimmyg	thejimmyg	ckan-sprint-2012-01-09	closed	worksforme	CKAN dumps dgu miss certain publisher information	Pawel knows about this so David Read, Pawel and I need to find time to discuss it.	1321376042000000	1326120319000000
1524	enhancement	thejimmyg	thejimmyg	ckan-sprint-2011-12-19	closed	fixed	New Release of Latest MI Reports		1323172688000000	1323191733000000
2268	enhancement	thejimmyg	amercader	ckan-backlog	new		UKLP Use same schema as form for harvested records	Datasets created via the harvesting (UKLP ones) and the ones created via the form use a different schema. That could lead to problems. We need to discuss the best approach.	1333108606000000	1333108606000000
698	task	Stiivi	thejimmyg	ckan-v1.3-sprint-1	closed	fixed	CKAN Data API v1	"This proposal is to discuss adding a new API for proxying certain spreadsheet data via JSON-P to make it possible to build simple browser apps directly off the API.

See the attached proposal for information."	1287073433000000	1293649815000000
1	enhancement	somebody	rgrp	milestone1	closed	fixed	Visitor visits service	"== Title ==

Visitor visits service

== As a ==

Visitor (general web user)

== I want to ==

Visit the website of the service (frontpage)

== So that ==

  * See the front page and discover information about the service
  * Perform various activities related to the service such as registering and finding packages, registering as a user and logging in"	1152549417000000	1183636342000000
2	enhancement	somebody	johnbywater	milestone1	closed	fixed	Register as user on the service	"== As a ==

Visitor

== I want to ==

Be able to register as a user on the service

== So that ==

One can login and be an authenticated user and perform the actions that only authenticated users can perform

== Notes ==

  * Registration involves the provisional of a minimal set of user details, namely:
    1. username (unique in lifetime of the service)
    2. fullname
    3. email
    4. password
"	1152549635000000	1152553157000000
3	enhancement	somebody	johnbywater	milestone1	closed	fixed	Login to the service	"== As a ==

Visitor

== I want to ==

Login to the service

== So that ==

I am now an authenticated user and may perform the actions that only authenticated users may perform

== Notes ==
  * Login by providing username and password
  * Login screen should also include a link or information about registration
  * If username do not match (or the username does not exist) a login failure is noted and the visitor is re-presented with the login screen as well as information on how to register
"	1152549717000000	1152553322000000
4	enhancement	somebody	johnbywater	v0.7	closed	fixed	Find a package	"== As a ==

Visitor

== I want to ==

Locate packages that I am, or might be, interested in

== So that ==

I know what packages exist and can read the information associated with those packages

== Notes ==

Find a package by searching by (or some combination thereof):
    1. tag
    2. string contained in title or description
    3. license

"	1152549820000000	1223908381000000
6	enhancement	somebody	johnbywater	milestone1	closed	fixed	View a package	"== As a ==

Visitor

== I want to ==

View the details of an individual package

== So that ==

I know more about the package. Attributes available are detailed in updating a package (../UseCase12)

"	1152549960000000	1183648175000000
9	enhancement	somebody	johnbywater	milestone1	closed	fixed	Register new package	"== As a ==

Authenticated User

== I want to ==

Create/register a package

== So that ==

The package will be listed in the system and visible to users

== Notes ==
  * Full package registration process may be broken down into:
    1. Creating a package by provision of minimal attributes:
      1. name: unique and web-usable
      2. auto-generation of default values such as:
        1. date-registered (today's date)
        2. first owner (one owner should be created corresponding to the authenticated user performing this action)
      3. (?) auto-generate a default release
    2. updating the created package -> ../UseCase12
"	1152550171000000	1152555104000000
13	enhancement	somebody	rgrp		closed	invalid	Delete a release		1152550611000000	1218545330000000
18	enhancement	somebody	johnbywater		closed	invalid	Purge a user (registered person)		1152550871000000	1199787607000000
19	enhancement	somebody	rgrp		closed	invalid	Undelete a user (registered person)		1152550881000000	1199787631000000
21	enhancement	somebody	johnbywater	v0.11	closed	fixed	Undelete a package		1152550936000000	1265891796000000
22	enhancement	somebody	johnbywater		closed	invalid	Update person attributes		1152551014000000	1199787243000000
23	enhancement	somebody	johnbywater		closed	invalid	Request new password (or a reminder) if old has been lost		1152551256000000	1199786799000000
24	enhancement	somebody	johnbywater	v0.9	closed	fixed	View list of most recently registered or updated packages on front page		1152551291000000	1246441366000000
26	enhancement	somebody	johnbywater		closed	duplicate	A registered person creates their own tags for a package		1152551351000000	1152555283000000
28	enhancement	somebody	johnbywater		closed	wontfix	Provide a dump of all packages listed on the system in XML		1152551416000000	1200902911000000
31	enhancement	somebody	rgrp		closed	wontfix	Add a comment to a package	"== As a ==

Registered user

== I want to ==

Add a comment to a package

== Notes ==

  * The comment should be tagged with the username of the person making the comment
  * Should we allow comments per release as well as per package? '''No''' for the time being.
  * Comments will be 'flat' and '''not''' threaded (at least for the present)
  * Comments should be plain text and so html should be escaped (in the future we might look to use some kind of formatter such as markdown)"	1157374505000000	1185472236000000
34	defect	somebody	zool		closed	worksforme	Page Not Found - returns with 200 rather than 404 as it should		1163008961000000	1253781550000000
1308	defect	shevski	dread		closed	fixed	'create package' user case failure	On the CKAN home page there is no suggestion that you can add or register a dataset. The 'Register a dataset' option has been removed at some point! (only visible to users that have registered - why would they do that when the benefits aren't described?)	1315219500000000	1323174312000000
2547	enhancement	shevski	ross	opendatasuite 2	assigned		ODS Initial data sets	"
== Requirements ==

The ODS demo site will need data adding, initially as fixtures but it would also be useful if we started evaluating datasets that we can ship with ODS installations (at least in the UK) from places such as DGU and ONS.

May wish to create a ticket for making sure the datasets within the system are reset every X hours. Perhaps.


== Interface ==

None

== User Stories ==

 * As a new system administrator for an ODS instance, I don't want to have a site devoid of any data. Geographically relevant datasets would be welcomed.

 * As a bizdev person I would like to be able to demonstrate how ODS works with real datasets.


== Tasks ==

[ ] Identify relevant sources for datasets

[ ] Pick datasets

[ ] Set them up for import


== Estimates ==


"	1340016906000000	1340705614000000
2587	enhancement	shevski	toby	demo phase 1	closed	fixed	author/maintainer not shown if email missing	is this the expected behaviour seems crap to me, should we just see the author but just no link 	1340369450000000	1340376975000000
2590	enhancement	shevski	ross	ckan-backlog	assigned		Publisher dashboard	"Need proper user stories but ...

Publisher admins/editors may need a more useful group read page showing things like:

 * The current search

 * Recent activity

 * People within the group

 * Followers

 * Others? 
"	1340618617000000	1346663416000000
2612	enhancement	shevski	shevski	demo phase 2	closed	fixed	Hide options based on auth	For example, hide the 'edit' button on user profiles http://s031.okserver.org:2375/user/ross2#activity unless I'm logged in as Ross	1340812165000000	1342714613000000
2642	enhancement	shevski	amercader	demo phase 3	closed	fixed	Adapt harvest frontend to new theme		1341846196000000	1345113645000000
2644	enhancement	shevski	toby	demo phase 5	assigned		user dashboard for demo theme	"we now have a user dashboard that needs theming not sure if we need sam to look at it

http://localhost:5000/user/dashboard"	1341910821000000	1344255836000000
2648	enhancement	shevski	aron.carroll	demo phase 2	closed	fixed	Add file upload to the demo site		1341921081000000	1342714659000000
2660	defect	shevski	shevski	demo phase 2	closed	fixed	server error when updating dataset	"same as yesterday: http://demo.ckan.org/dataset/edit/newcastle-city-council-payments-over-500
clicking update/save returns error"	1342010970000000	1342090301000000
2667	enhancement	shevski	toby	demo phase 2	closed	fixed	edit resource needs form	"I thought I'd made this ticket before but can't find it.  When we have a fully created dataset we need to be able to edit the resources.

template: package/resource_edit.html

when you've made a form reassign to me and I'll get it working
 "	1342170103000000	1343058578000000
2680	enhancement	shevski	shevski	demo phase 3	closed	fixed	review group deletion	"Super admins & owners/creators of groups need a way to delete them

Is the auth model in place for all this stuff? May be worth sitting down and going through how this should work

Currently can change state from active to deleted - this is not ery user friendly (want a button with a 2 step process maybe?) & I'd like to know who can view this option? Everyone who can edit a group? Who can edit?"	1342436525000000	1344509493000000
2699	enhancement	shevski	shevski	demo phase 5	assigned		workflow for associating datasets with groups	"needs review & speccing out
e.g. datasets created by a user who belongs to a certain publisher (group) get auto added to this group"	1342620176000000	1344507133000000
2702	enhancement	shevski	shevski	demo phase 5	assigned		Future Javascript wishlist for demo	"tooltip on popular datasets with number of views
facets to update automatically
creating a dataset without reloading page between steps
hover on licences information
autocomplete on search terms
group filtering
social share buttons in lightboxes
dataset counts on homepage"	1342620475000000	1344255984000000
2717	enhancement	shevski	toby	demo phase 3	closed	fixed	French breaks the demo header	"when language is french


a) the menu overlaps the logo

b) much of the logo becomes unclickable

@shevski

a) seems like you  b) is aron"	1342780487000000	1344359846000000
2814	enhancement	shevski	markw		assigned		Demo: upload file behaves oddly	"3. Uploading a file behaves counter-intuitively (I would suggest wrongly).

When adding a new resource by uploading a file, I select a file called
say create-group.png. I expect the following to happen:

 * the pathname of that file is filled in the box;
 * nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc).

Instead of this,

 * the file is immediately and, seemingly, irrevocably uploaded;
 * the box is filled with a mysterious URL (https://commondatastorage.googleapis.com/ckan-demo ...)
 * the mysterious URL is so long I can't even see the final element, which is my file name.
"	1344420360000000	1346670381000000
2855	defect	shevski	shevski	demo phase 3	closed	fixed	language box no longer functional	e.g. tried changing the lang here: http://s031.okserver.org:2375/ to no effect. Doesn't work on any other pages either. Flags have also dissappeared	1345049152000000	1345104178000000
2956	enhancement	shevski	johnmartin	ckan 2.0	assigned		Allow for resource editing in CKAN 2.0	On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually.	1349277100000000	1349346442000000
576	defect	sebbacon	wwaites	ckan-v1.3	closed	fixed	CKAN Requires Old Version of SQLAlchemy	"Requires 0.4. 0.5 is a maintenance branch, 0.6 is current.

VDM appears to work correctly (all the tests pass) with 0.5. CKAN does not -- requires more investigation to determine exactly why.

It would actually be nice to be able to run with 0.6 or 0.5, though 0.6 will require some changes to VDM as well"	1284141573000000	1294753848000000
662	defect	sebbacon	johnbywater	ckan-v1.4	closed	fixed	Can't put entity that is returned by posting to package register	"It's because Package carries several out-of-band values, which are snagged on the way back out. Entity get response also can't be posted.

However, post response can be re-posted (because it isn't the same as the register-post/entity-get responses.

An issue for CKAN too.

Sub-ticket of #961 (form, validation, model sync meta-ticket) and depends on that work."	1285410546000000	1301076463000000
868	enhancement	sebbacon	thejimmyg		closed	fixed	Test improvements	"The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach.

We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!)

Some suggestions for achieving this include:

 * Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database
 * Not setting up and tearing down the database so frequently.
"	1291987811000000	1294753596000000
876	enhancement	sebbacon	rgrp	ckan-v1.3-sprint-2	closed	fixed	Support sqlite as a database backend for CKAN	Among other things this will allow the tests to run much quicker.	1292858711000000	1294753889000000
921	enhancement	sebbacon	thejimmyg		closed	fixed	Refactor DGU Harvesting	"Refactor harvesting code to simplify.
"	1295389237000000	1300197629000000
926	enhancement	sebbacon	sebbacon	ckan-v1.4-sprint-1	closed	fixed	Pick a simpler form framework	"The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

Review existing (and potentially non-existing) frameworks with a view to porting forms over to something more explicit and lightweight.

Implement the current Package forms as an example of how this would work.

Document and circulate.

Sub-ticket of #961 (form, validation, model sync meta-ticket)"	1295869056000000	1298541597000000
937	enhancement	sebbacon	rgrp	ckan-v1.4-sprint-5	closed	fixed	Record download stats for resources (extension)	"As a User (especially as a Package Owner/Maintainer) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

== Implementation ==

 * Use the existing support for this feature in google analytics or piwiki
 * For google analytics see:
  * Integrate to record: http://www.google.com/support/analytics/bin/answer.py?answer=55529
  * Accessing analytics data via API (for showing download counts): http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html
 * Limitations only record downloads by browsers with js turned on.
  * This is OK I think (in any case machine download via e.g. datapkg does not get downloaded)

=== Old Spec (do it in CKAN) ===

 * Record info of form: resource id (or url?), timestamp
 * Do this via javascript capturing of onclick event talking to an api
 * API: /api/resource/{id}/download
  * POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
  * GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

== Questions ==

 * Do we record ip addresses (to handle de-botting etc)?
 * Do we count preview clicks as well?"	1296340273000000	1302513831000000
971	enhancement	sebbacon	sebbacon		closed	duplicate	Support middleware as a plugin	I need to completely munge the rendered data coming from the app.  There's not currently a plugin hook for this, where I get access to the entire environment.  The most generic form of plugin would be WSGI Middleware.  I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system.	1297277473000000	1299245064000000
972	enhancement	sebbacon	rgrp	ckan-v1.5	closed	wontfix	Merge 'extras' into main package dict rather than having separate key	"Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.

== Issues ==

 * Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961"	1297332282000000	1310134129000000
1011	enhancement	sebbacon	sebbacon	ckan-v1.4-sprint-3	closed	fixed	Make pluggable AuthzGroups implementation	"I have a requirement to allow an external source to define the groups of which a user is a member.

I propose to create an IAuthzUserGroups plugin interface that allows an extension to arbitrarily extend the list of AuthzGroups that a user is in."	1298819657000000	1299245206000000
1013	defect	sebbacon	sebbacon	ckan-v1.4-sprint-5	closed	fixed	Refactor & write tests for Wordpresser extension	The Wordpresser extension seems to be working well for DataGM, but needs caching and tests.	1298887392000000	1301909717000000
1014	defect	sebbacon	sebbacon	ckan-v1.4-sprint-3	closed	fixed	Decouple oAuth repoze extension from DataNO, write tests	The current oAuth extension tightly couples repoze.who, ckan, and data.no.  These components should be decoupled; they also need more tests	1298887493000000	1299245293000000
1047	defect	sebbacon	sebbacon	ckan-v1.4-sprint-4	closed	fixed	Package edit form claims you're not logged in at the end when you are	"At the bottom, it says something like

Author: Bob Bumgardner
Since you have not signed in this will just be your IP address. Click here to sign in before saving (opens in new window)."	1300384556000000	1300387309000000
1051	defect	sebbacon	sebbacon	ckan-v1.4-sprint-4	closed	fixed	Ability to set custom favicon	Allow deployers to set a URL pointing to their own favicon	1300703160000000	1301305079000000
1063	defect	sebbacon	sebbacon		closed	fixed	Groups listing widget on package screen shouldn't show group name by default	"I've been asked if we can do something about the overflow of the Group
name in the right hand column on this page:

 http://register.data.overheid.nl/package/europese-aanbestedingen

The reason is that the list display for groups is in the form
""group_tltie (group_name)"", and of course group_name can't have spaces
and so can't wrap nicely.

I was wondering if there's a good reason why we don't only display
group_title (if it exists) and group_name only when there's not a
title?"	1301408459000000	1302514033000000
1299	enhancement	seanh	kindly	ckan-sprint-2012-01-23	closed	fixed	Activity streams table migration	Migrate tables for activity streams	1314696635000000	1327322739000000
1413	enhancement	seanh	shevski	ckan-backlog	closed	fixed	Ask users to 'add email address' when logged in	"I'd like us to display a one-off banner that prompts users who log into thedatahub.org to go and update & their profile & fill in their email address (if we could do it just for those who don't have an email address, then even better)
That is, once they log in, they see a banner which says:
Please update your profile [here] and add your email address, so you can receive notifications, be able to reset your password and get updates

Then when we build notifications or if we decide to survey people, we can actually contact them. Email address should be required as standard

DR: Also, mention their Full Name too - v. useful for selecting correct user in group curation"	1319453108000000	1330020486000000
1434	enhancement	seanh	dread	ckan-v1.8	closed	fixed	Add i18n strings from extensions	Decide which extensions are important.	1320145648000000	1341245889000000
1471	enhancement	seanh	dread	ckan-v1.8	closed	fixed	Be compatible with new ubuntu release in April	"Nils flagged up that Ubuntu launches another LTS in April which we should probably move to. This ticket is to get the ball rolling with what we do. We can always include in our package the exact package versions of deps that we want, but it is good to be in line with the release if possible.

Here's how to see the packages included with it:
http://packages.ubuntu.com/search?keywords=python-pylons&searchon=names&suite=all&section=all

Pylons 0.9.7 -> 1.0-2 Requires a small amount of work
SQLAlchemy 0.7.3-2 OK
Babel 0.9.6-1 OK
Genshi 0.6-2 Perfect

Another alternative is to move to another web framework, especially minding that Pylons is not being further developed."	1321552345000000	1340810270000000
1486	defect	seanh	seanh		closed	fixed	"""allow_partial_update"" option does not work with resource lists of packages"	"1. Call ckan.logic.action.update:package_update()
2. Pass in a context dict with ""allow_partial_update"":True
3. Pass in a partial data dict containing only the package's ID and those fields that you want to update (e.g. title).

=> All of the package's resource will have their statuses changed to ""deleted"".

The ""allow_partial_update"" option should apply to the resource lists of packages, but it does not."	1322061318000000	1323172027000000
1491	defect	seanh	dread		closed	fixed	Visible strings need internationalisation	"Sean spotted that some strings need internationalising, such as flash messages. Generally all strings that appear on the web front-end should be internationalised.

Particular places that need i18n:

* ckan/controllers/admin
* ckan/lib/base

Note: there are some exceptions, such as in i18n.py, very obscure error messages and stuff that only appears on the API.

NB: there is a cost in making a string internationalisable (all our volunteers have to translate it), so we should not be too zealous. We should also look at the i18n/ckan.pot to see if any of the existing strings can be reused."	1322481000000000	1330020630000000
1494	enhancement	seanh	seanh	ckan-sprint-2012-01-09	closed	wontfix	API call for getting a user's public activity stream as rendered rtext	"This could be implemented as a separate API call, or the rendered text versions of the activities could be added into the JSON returned by the existing API call.

This requires setting up templates for rendering activity streams items and detail items as nice, human-readable text.

There are some open questions, e.g.: Do we want the entire activity stream rendered as a block of plain text? As HTML? Or do we want a list of JSON objects, where each object contains its textual and/or HTML representations as fields?

Activity stream items and their related detail items are separate objects that each have their own textual representations.

For a mockup of the kind of text messages we want, see:

http://datahub.pudo.org/pudo

but note that this ticket is just for creating the text snippets themselves, not rendering then in an HTML page or RSS feed. Also the mockup only show activity items and not their detail items."	1322495447000000	1326109757000000
1496	enhancement	seanh	seanh	ckan-v1.7	closed	fixed	Render a user's activity stream on her user page		1322495588000000	1334582338000000
1510	enhancement	seanh	seanh	ckan-v1.7	closed	wontfix	Migrate existing data into activity streams database tables	"See '### Task 2' in this etherpad: http://ckan.okfnpad.org/notifications

I have no idea yet about how to do this or how long it might take me."	1323093901000000	1339677620000000
1515	enhancement	seanh	seanh	ckan-v1.7	closed	fixed	[super] Activity Streams	"We want to add ""activity streams"" (chronological lists of activities) to CKAN.

Activity streams standard: http://activitystrea.ms/

Super branch where the feature was initially developed (this has now been merged into master): https://github.com/okfn/ckan/tree/feature-1515-activity-streams

Etherpad: http://ckan.okfnpad.org/notifications (most relevant parts inlined here)

Friedrich's mockup: http://datahub.pudo.org/pudo

== Tasks ==

~~Strikethrough~~ means the task is done.

 * ~~New domain objects Activity and ActivityStream [2d] #1298~~

 * Migrate existing data into activity streams model [3.5d] #1510 

 * ~~Generate activities when objects are saved [4d] #1298~~

 * ~~Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] #1511~~

 * ~~API call for getting a user's public activity stream as JSON [0.5d] #1495~~

 * ~~API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] #1494~~

 * ~~Render a user's activity stream on her user page [3d] #1496~~ (needs review/page design work)

 * ~~Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.~~

 * ~~Logic functions for getting activity streams for packages, groups, and users.~~

 * ~~API calls for getting activity streams of packages, groups, and users~~ as JSON.

 * ~~Rendering activity streams for packages and groups etc. into their pages.~~ (done but disabled pending page design work)

 * ~~API calls for inserting events into activity streams.~~ (Still need to handle activity details in the API, ticketed).

 * Conform to activitystrea.ms standard

 * Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page.

 * Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams

 * Email notifications

 * Extension point for extensions to register themselves as renderers for different types of activity

 * Deploy to UAT [0.5d]

 * Docs and announce [1d]"	1323105048000000	1338204149000000
1621	defect	seanh	seanh		closed	fixed	UnicodeDecodeError when validating user password	"A test case is currently failing for me on master:


{{{
ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode
----------------------------------------------------------------------
Traceback (most recent call last):
  File ""/home/seanh/pyenv/lib/python2.6/site-packages/nose/case.py"", line 197, in runTest
    self.test(*self.arg)
  File ""/home/seanh/pyenv/src/ckan/ckan/tests/functional/test_user.py"", line 342, in test_user_create_unicode
    res = res.follow()
  File ""/usr/lib/pymodules/python2.6/paste/fixture.py"", line 603, in follow
    return self.test_app.get(location, **kw)
  File ""/usr/lib/pymodules/python2.6/paste/fixture.py"", line 208, in get
    return self.do_request(req, status=status)
  File ""/usr/lib/pymodules/python2.6/paste/fixture.py"", line 389, in do_request
    **req.environ)
  File ""/usr/lib/pymodules/python2.6/paste/wsgilib.py"", line 343, in raw_interactive
    app_iter = application(basic_environ, start_response)
  File ""/usr/lib/pymodules/python2.6/paste/lint.py"", line 170, in lint_app
    iterator = application(environ, start_response_wrapper)
  File ""/usr/lib/pymodules/python2.6/paste/cascade.py"", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File ""/usr/lib/pymodules/python2.6/paste/registry.py"", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File ""/usr/lib/pymodules/python2.6/repoze/who/middleware.py"", line 69, in __call__
    auth_ids = self.authenticate(environ, classification, ids)
  File ""/usr/lib/pymodules/python2.6/repoze/who/middleware.py"", line 201, in authenticate
    userid = plugin.authenticate(environ, identity)
  File ""/home/seanh/pyenv/src/ckan/ckan/lib/authenticator.py"", line 29, in authenticate
    if user.validate_password(identity.get('password')):
  File ""/home/seanh/pyenv/src/ckan/ckan/model/user.py"", line 113, in validate_password
    hashed_pass = sha1(password_8bit + self.password[:40])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 12: ordinal not in range(128)

}}}
"	1326105806000000	1335878161000000
1625	enhancement	seanh	seanh	ckan-sprint-2012-01-23	closed	fixed	Add activity stream events for new/changed users	This requires adding a logic function for emitting an activity stream event, and then editing the logic functions for creating or updating users and making them call the new emit event function. This same emit event function can later be used to emit activity stream events for other types of object as well.	1326187794000000	1326736328000000
1631	enhancement	seanh	seanh	ckan-sprint-2012-01-23	closed	fixed	Add activity stream events for new/changed groups		1326304020000000	1326736381000000
1632	enhancement	seanh	seanh	ckan-v1.7	closed	wontfix	Render a group's activity stream on its' page	Add logic functions for getting the activity stream for a group in JSON and rendered HTML formats, add HTML activity stream into group page.	1326304199000000	1335877871000000
1633	enhancement	seanh	seanh	ckan-v1.7	closed	wontfix	Render a dataset's activity stream on its' page	Add logic functions for getting the activity stream for a dataset in JSON and rendered HTML formats, add HTML activity stream into dataset page.	1326304298000000	1335877927000000
1634	enhancement	seanh	seanh	ckan-v1.7	closed	duplicate	Allow users to follow/unfollow activity streams of other users, datasets and groups	"This is a fairly big feature to add. Analysis: http://ckan.okfnpad.org/27

This ticket is related to #1635 (email notifications for activity streams) which is about managing a list of activity streams for which the user receives email notifications. There are decisions to be made about how the two will work together."	1326304542000000	1334587390000000
1635	enhancement	seanh	seanh	ckan-backlog	new		Email notifications (e.g. for activity streams)	"CKAN should be able to send email notifications to users.

Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users.

One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well.

Need to implement (at least some of) #1634 before this can be implemented, in order to have something to send notifications about.

Analysis here: http://ckan.okfnpad.org/27"	1326304587000000	1355141157000000
1636	enhancement	seanh	seanh	ckan-v1.7	closed	wontfix	Purge items from activity streams when object (package etc.) purged from CKAN		1326304754000000	1338204258000000
1637	enhancement	seanh	seanh	ckan-v1.6	closed	fixed	API call for getting the list of activity detail items for a given activty stream item	(and add test cases for it)	1326304817000000	1326737169000000
1638	enhancement	seanh	seanh	ckan-sprint-2012-01-23	closed	wontfix	Don't use JsonType in activity streams	Dump and load JSON explicitly instead.	1326304935000000	1326305570000000
1639	enhancement	seanh	seanh	ckan-sprint-2012-02-06	closed	fixed	Move activity streams CSS into external file	"The HTML and CSS for the activity streams templates needs to be tidied up. Currently the CSS is inlined in the templates themselves, and there is much unnecessary repetition of CSS.

Perhaps put this one in hold until there has been some design discussion about how the activity streams and the user, dataset and group pages with activity streams should look?"	1326305019000000	1328526779000000
1663	enhancement	seanh	seanh	ckan-sprint-2012-02-06	closed	fixed	API call for creating an activity streams event	The create_activity() logic function already exists, but validation and authorization need to be added before it's hooked up to an API call.	1326794844000000	1328013153000000
1664	enhancement	seanh	seanh	ckan-future	closed	fixed	Smarter group activity streams	"We already have simple activity streams for groups that have new group or group updated events (see #1631). Smarter/more useful group activity streams would also contain the events for things related to the group: users, datasets, other groups, etc.

This should just mean using a more complex SQL query in the group_activity_list() logic function.

Things might get tricky when a user or dataset joins a group, this may show up as two events (the user or dataset was modified, and the group was modified). We may want to have two separate events for this (one to go in the user or dataset activity stream and one for the group), or maybe we just want one 'user/dataset joined group' event."	1326795654000000	1351863437000000
1665	task	seanh	seanh		closed	fixed	Begin doing research into eurovoc	How big is it? How are we going to store it? etc.	1326795828000000	1329742600000000
1666	enhancement	seanh	seanh	ckan-sprint-2012-02-06	closed	fixed	Add activity stream events for tags	"...and show them on the tag pages.

This means adding tag_activity_list() and tag_activity_list_html() logic functions, adding the templating to render tags as HTML, and adding the tag HTML into the tag pages.

What exactly should go into a tag's activity stream? As with datasets we may want a more complex SQL query, where events from objects related to the tag (datasets, users, etc.) appear in the tag's stream."	1326795985000000	1328527113000000
1667	enhancement	seanh	seanh	ckan-future	new		Add an extension point for rendering activity streams	"Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML.

There needs to be an extension point where extensions can register their own rendering functions for particular activity types."	1326796151000000	1338204295000000
1686	enhancement	seanh	seanh	ckan-sprint-2012-02-06	closed	fixed	Add activity events for when a dataset is added to or removed from a group		1326993692000000	1328527370000000
1694	enhancement	seanh	seanh	ckan-sprint-2012-02-06	closed	fixed	Add quick functional test(s) for activity streams HTML rendering	The activity streams feature is already thoroughly tested via the logic layer and API calls, but there needs to be a quick test of the rendering code, e.g. fetch a user's page (thus testing that ckan doesn't crash while rendering it) and then assert that the expected activity streams HTML is in the fetched HTML.	1327322248000000	1328465254000000
1698	enhancement	seanh	seanh	ckan-v1.7	closed	fixed	[super] Tag Taxonomies	"Add drupal-like ""taxonomies"" to CKAN.

Etherpad with user stories, feature list, design and implementation discussion:

http://ckan.okfnpad.org/22

Branch where this is being developed is feature-1698-tag-taxonomies:

https://github.com/okfn/ckan/compare/master...feature-1698-tag-taxonomies

Tickets related to this have [http://trac.ckan.org/query?status=accepted&status=assigned&status=closed&status=new&status=reopened&group=owner&max=1000&order=milestone&col=id&col=summary&col=status&col=owner&col=milestone&col=keywords&keywords=~taxonomies keyword taxonomies]."	1327415756000000	1338204433000000
1705	enhancement	seanh	seanh	ckan-sprint-2012-02-20	closed	fixed	Implement Vocabularies domain model and API	Add Vocabulary domain class, add logic functions for creating, updating, listing, getting, deleting vocabularies, add tests.	1327427254000000	1329131067000000
1706	enhancement	seanh	seanh	ckan-sprint-2012-02-06	closed	duplicate	Tag taxonomies package schema	Enable ckanext's to add new vocabularies to default_package_schema(), use the new vocabularies in templates for dataset view, create, edit pages, helper function for extensions to create a vocabulary schema for a given vocabulary name (must match a vocab already in the db) and add it to default package schema.	1327427938000000	1329131079000000
1722	enhancement	seanh	seanh	ckan-sprint-2012-02-20	closed	fixed	Update the package model for vocabularies	See the etherpad for notes: http://ckan.okfnpad.org/22	1327950373000000	1329303197000000
1723	enhancement	seanh	seanh	ckan-sprint-2012-02-20	closed	wontfix	Update tests for taxonomies	"In the tag-taxonomies branch tags now (optionally) belong to vocabularies and where tags previously had unique names, a CKAN instance can now have two tags with the same name as long as they belong to different vocabularies. The tests for the tags model should probably be updated to reflect this.

Other tests might also need to be updated."	1327950494000000	1329131103000000
1724	enhancement	seanh	seanh	ckan-sprint-2012-02-20	closed	fixed	Update logic action functions for vocabularies	"{{{ckan/logic/action/get.py:tag_list()}}}
Currently (I think) just returns all tags in the CKAN instance that the user has access to.
Needs an optional argument to get only tags from a particular vocabulary?
When called with no argument, should only return free tags?
Also add an optional dataset argument, so you can get tags for a given dataset and/or vocabulary?

{{{ckan/logic/action/get.py:tag_autocomplete()}}}
Add optional argument to get only tags for a given vocab? When called with no argument, should get only free tags?

{{{ckan/logic/action/get.py:tag_search()}}}
Again, optional arguments for restricting by vocabulary and/or dataset."	1327950761000000	1329302302000000
1743	defect	seanh	seanh	ckan-v1.6	closed	wontfix	Activity streams still link to deleted things	For example, after deleting a dataset old activity stream events will still hyperlink to that dataset.	1328212016000000	1328527086000000
1748	enhancement	seanh	seanh	ckan-future	new		Make activity streams conform to http://activitystrea.ms/ standard	"At the very least we should make our JSON output conform to their spec:

http://activitystrea.ms/"	1328450784000000	1338204337000000
1754	enhancement	seanh	seanh	ckan-sprint-2012-02-06	closed	fixed	Add activities for extras	When an extra is added to or removed from a dataset or is updated, an activity should go in the dataset's and user's activity streams.	1328528318000000	1328528330000000
1763	enhancement	seanh	seanh	ckan-v1.7	closed	wontfix	Add some simple model-level tests for activity streams		1328536285000000	1338204708000000
1764	enhancement	seanh	seanh	ckan-v1.7	closed	wontfix	Move activity streams API tests into tests/logic	Which is where the other API tests have now been moved.	1328536499000000	1335878051000000
1765	enhancement	seanh	seanh	ckan-sprint-2012-02-20	closed	fixed	Enhance Tag and Package models with vocabularies	Add a vocabulary column to the tags database table, change tags to have unique (tag_name, vocabulary_id) instrad of unique tag name, update methods in the Tag and Package classes to deal with the fact that tags may belong to vocabularies and that tag names are no longer unique.	1328537061000000	1329131124000000
1775	enhancement	seanh	seanh	ckan-sprint-2012-03-05	closed	fixed	Enable adding many tags at once to controlled vocabularies via API	"We already have API calls for adding one tag or removing one tag at a time to/from a vocab. Want more convenient ways to add or remove many in one call, or just pass in a list of tags to completely replace a vocab's current list.

- Add tags_create() function that calls the existing tag_create() many times?
- Also tags_delete()?
- Or just let tag_create() and tag_delete() accept a list?
- Add tags argument to vocabulary_update() to pass in a list of tags to replace the vocab's current tags? (But it should still be possible to simply rename a vocab without changing its tags and without having to pass in the tag list.)"	1328541056000000	1329754459000000
1776	enhancement	seanh	seanh	ckan-sprint-2012-02-20	closed	fixed	Granular editing of vocabulary tags	Add API calls for adding one or more tags to and removing one or more tags from a vocabulary, without affecting the other tags in that vocabulary and without having to pass the full list of the vocabulary's tags.	1328541213000000	1329302364000000
1778	enhancement	seanh	seanh	ckan-future	assigned		Replace classmethod's with module-level functions	"In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the {{{get()}}} classmethods in the model/ classes.

Doing it with module functions lets us type module.function() instead of module.Class.method().

In many cases we're importing classes directly with {{{from module import Class}}} and then doing {{{Class.method()}}}, but {{{from module import foo}}} is bad style: http://docs.python.org/dev/howto/doanddont.html"	1328543614000000	1340623743000000
1784	enhancement	seanh	seanh	ckan-v1.7	closed	wontfix	Links to renamed objects break	"The HTML templates for some activity stream activities link to objects (packages, resources, etc.) by their names, e.g. /dataset/foo, but if the object has since been renamed then the link will be broken.

Possible solutions are to use the object ID to find its current name at rendering time and link to that, or link to the object by ID rather than by name.

Linking by ID is both simpler to implement and better. Links by ID are ugly, but they are permanent and don't break when an object is renamed. We should be using links by ID everywhere on CKAN, so that when third-party sites copy-paste our URLs and link to us, their links don't break later.

Fixing this means fixing the dataset_link(), resource_link(), group_link() etc. functions in ckan/lib/helpers.py which will change the links everywhere in CKAN where those functions are used to link to objects (which should be everywhere where we link to an object).

If we want to have nice looking by-name URLs and what them to be permanent, then we need to change CKAN so that objects cannot be renamed."	1328613555000000	1338204797000000
1785	enhancement	seanh	seanh	ckan-v1.9	accepted		Replace 'Revisions' page with site-wide activity stream	Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead?	1328632458000000	1340724312000000
1807	enhancement	seanh	seanh	ckan-sprint-2012-03-05	closed	fixed	Recently added/updated datasets for EC portal	"For EC Portal need a page (or HTML snippet embedded in the front page?) listing recently added or updated datasets.

* Could be generated from the activity streams

* Should it list datasets only, or other activities as well?

* May be related to #1785 (Replace 'Revisions' page with site-wide activity stream)"	1329742326000000	1330809641000000
1808	enhancement	seanh	seanh	ckan-sprint-2012-03-19	closed	fixed	Translation of content on dataset view pages	"All data is that is translatable should be translated when viewed.

This includes tag names will be  ID codes. When viewing a dataset page, look up the current language of the interface (ckan_lang or something in the environs) and display the string for the tag in that language.

Related to the dataset view extension points for translation, recently added by kindly/=."	1329742530000000	1331146804000000
1825	enhancement	seanh	seanh	ckan-sprint-2012-03-05	closed	fixed	Write docs for vocabularies API	Done on branch feature-1698-tag-taxonomies	1329845122000000	1330807330000000
2199	enhancement	seanh	seanh	ckan-sprint-2012-03-19	closed	fixed	Translation of content on dataset search pages		1330440107000000	1331146832000000
2200	enhancement	seanh	seanh	ckan-future	new		Add vocabulary_id option to tag_show() logic action function	Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.	1330533678000000	1338204900000000
2207	defect	seanh	seanh	ckan-sprint-2012-03-19	closed	invalid	Fix recently changed datasets activity stream on exportal site	Apparently it's working when developers run it locally but not on the demo site.	1330961739000000	1332154835000000
2208	enhancement	seanh	seanh	ckan-sprint-2012-06-25	closed	fixed	Export strings from ckan and ckanext-ecportal to combined pot file	All strings from the ckanext-ecportal extension (and from CKAN core) need to be exported to a pot file to send to tenforce.	1330961806000000	1339596371000000
2214	enhancement	seanh	seanh	ckan-sprint-2012-03-19	closed	fixed	Translation of content on group view pages		1331146864000000	1331147119000000
2215	enhancement	seanh	seanh	ckan-sprint-2012-03-19	closed	fixed	Translation of content on group list page		1331148579000000	1331653117000000
2216	enhancement	seanh	seanh	ckan-sprint-2012-04-02	closed	fixed	Translation of selected facets on search pages		1331148648000000	1332956791000000
2217	enhancement	seanh	seanh	ckan-sprint-2012-03-19	closed	fixed	Translation of content on tag list page		1331148704000000	1331652833000000
2218	enhancement	seanh	seanh	ckan-sprint-2012-03-19	closed	fixed	Translation of content on tag pages		1331148734000000	1331747842000000
2219	enhancement	seanh	seanh	ckan-sprint-2012-03-19	closed	fixed	Translation of datasets and tags on user pages	User names and descriptions are not translated.	1331148895000000	1331750370000000
2238	enhancement	seanh	seanh	ckan-future	closed	duplicate	Deploy some test vocabs and publishers on test.ckan.net, check tutorial-style instructions for these		1332165311000000	1343392681000000
2241	refactor	seanh	seanh	ckan-sprint-2012-04-02	closed	fixed	Look into making test_activity.py faster	...but only if it can be done reasonably quickly.	1332172354000000	1332426423000000
2242	enhancement	seanh	seanh	ckan-sprint-2012-04-02	closed	fixed	Research buildbot possibilities for CKAN		1332172405000000	1333375436000000
2243	enhancement	seanh	seanh	ckan-v1.9	reopened		Fix ckanext-example		1332172710000000	1340635768000000
2264	enhancement	seanh	seanh	ckan-sprint-2012-04-16	closed	fixed	Do an audit of string translations in CKAN core and ckanext-ecportal	Check to see that all the strings are getting translated, i.e. that all user-visible strings are being passed to gettext. Fix any strings that are not getting translated.	1332864512000000	1339411578000000
2272	enhancement	seanh	seanh	ckan-sprint-2012-04-16	closed	fixed	Setup jenkins on the new dev server		1333375480000000	1334075515000000
2273	enhancement	seanh	seanh	ckan-v1.8	closed	duplicate	Review publisher organisations code with Ross		1333375723000000	1340635788000000
2274	enhancement	seanh	seanh	ckan-sprint-2012-04-30	closed	fixed	Review multilingual branch with kindly, merge into master		1333375804000000	1335644408000000
2276	enhancement	seanh	seanh	ckan-sprint-2012-04-16	closed	fixed	Upgrade Slovakia CKAN instance		1333376435000000	1334579078000000
2290	defect	seanh	seanh	ckan-v1.8	closed	fixed	Mark facet titles in search pages for translation	The strings Tags, Res_format and Groups on e.g. the package index page should be marked for translation. These strings have been refactored in the feature-1821-multilingual-extension branch, so wait until that branch has been merged into master.	1334326651000000	1341238415000000
2299	enhancement	seanh	seanh	ckan-sprint-2012-04-30	closed	fixed	Set Slovakas default language on data.gov.sk		1334579112000000	1334670893000000
2300	enhancement	seanh	seanh	ckan-sprint-2012-04-30	closed	fixed	Write initial docs for multilingual branch		1334579163000000	1335644486000000
2301	enhancement	seanh	seanh	ckan-future	new		Replace old-style string formatting with format() method (at least in strings marked for translation)	"Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words.

Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.:

'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible')

http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting

We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated."	1334579264000000	1340635812000000
2304	enhancement	seanh	seanh	ckan-sprint-2012-05-15	closed	fixed	Watching/following of dataset/users	"User story #037 here: http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: http://ckan.okfnpad.org/27"	1334587233000000	1337158641000000
2305	enhancement	seanh	seanh	ckan-v1.8	closed	fixed	User dashboard with private consolidated activity stream	"User story #035 here: http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: http://ckan.okfnpad.org/27"	1334587329000000	1340879812000000
2318	enhancement	seanh	rgrp	ckan-sprint-2012-04-30	closed	fixed	Bug with Portugese translation and Javascript	"If you switch to Portugese and try to add a resource on dataset edit it will fail. This is because there is a string translation in js_strings.html that has quotes in it. When this is inserted into the file this causes a js exception which then prevents any further js processing.

Specifically:

{{{
CKAN.Strings.youHaveUnsavedChanges = ""Você tem alterações não salvas. Certifique-se de ter clicado ""Salvar Alterações"" abaixo antes de sair desta página."";
Uncaught SyntaxError: Unexpected identifier
}}}

To fix is simple i imagine: need to escape "" correctly in js_strings.html (or translations used).

I have temporarily patched this live in order to allow the hackday here to continue."	1335033889000000	1335774178000000
2335	defect	seanh	seanh	ckan-v1.9	assigned		Unicode characters don't work in CKAN ini files	You get a UnicodeDecodeError, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example.	1335789726000000	1340624635000000
2336	enhancement	seanh	seanh	ckanbuild	accepted		Move Jenkins' install script into ckan core so it can be versioned		1335792914000000	1340639637000000
2337	enhancement	seanh	seanh	ckan-v1.9	assigned		Sort out ini file confusion in docs	"For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/"	1335797876000000	1340624600000000
2339	enhancement	seanh	seanh	ckan-sprint-2012-05-15	closed	fixed	Demo install on RHEL/CentOS, and install instructions		1335876250000000	1336492657000000
2340	enhancement	seanh	seanh	ckan-v1.8	closed	fixed	Get Jenkins to automatically run core extensions tests		1335876289000000	1340040366000000
2341	enhancement	seanh	seanh	ckan-v1.9	accepted		Fix Jenkins issue when testing branches that contain model changes		1335876396000000	1340639472000000
2342	enhancement	seanh	seanh	ckan-v1.9	accepted		Fix Jenkins issue when testing branches with different solr schema versions		1335876445000000	1340639494000000
2344	enhancement	seanh	seanh	ckan-v1.8	closed	duplicate	Get jenkins install script into CKAN core	"After checking out a commit to test from CKAN's GitHub, Jenkins runs a script that creates a new virtual environment and installs CKAN and its dependencies into it, and does some other necessary tasks. Jenkins then runs the tests in this virtualenv.

The install script may have to change from one commit to the next as CKAN's install instructions change, so it would be good if the script was shipped in CKAN core. That way Jenkins will run different versions of the script depending on which commit it's testing and if the tests fail because the script is wrong then that's actually a bug that needs to be fixed in CKAN core. Also the CKAN install instructions could be simplified a lot by just having users run this script instead of doing each step by hand."	1335876673000000	1340040449000000
2345	enhancement	seanh	seanh	ckan-sprint-2012-05-29	closed	fixed	Rewrite action API docs using autodoc		1335876769000000	1337962454000000
2352	enhancement	seanh	seanh	ckan-sprint-2012-05-15	closed	fixed	Fix autodoc on CKAN's readthedocs		1335883991000000	1336492670000000
2368	enhancement	seanh	seanh	ckan-sprint-2012-05-15	closed	fixed	Fix OS X Lion install instructions on wiki	Solr install instructions install the wrong version	1335893719000000	1336477258000000
2369	enhancement	seanh	seanh	ckan-sprint-2012-05-15	closed	fixed	Pull latest translations from Transifex into CKAN 1.7		1335975051000000	1335975063000000
2370	enhancement	seanh	seanh	ckan-v1.9	assigned		Add examples of all config settings in default config file	The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.	1336042237000000	1340624662000000
2376	defect	seanh	seanh	ckan-sprint-2012-05-15	closed	fixed	"Fix ""no-sidebar"" front page layout issue with CKAN 1.7 in some languages"	"This needs to be fixed in the CKAN templates (""no-sidebar"" should not be marked for translation) but for 1.7 just fixed it in the translation files."	1336563965000000	1336563976000000
2377	enhancement	seanh	seanh	ckan-sprint-2012-05-15	closed	fixed	Upgrade data.gov.sk to CKAN 1.7 and apply their new theme		1336564008000000	1336564027000000
2386	task	seanh	seanh	ckan-sprint-2012-05-29	closed	fixed	Setup feature-2375-demo-theme test server		1337094996000000	1337967006000000
2387	task	seanh	seanh	ckan-sprint-2012-05-29	closed	fixed	Upgrade cz.ckan.net to CKAN 1.7		1337095029000000	1337095933000000
2391	task	seanh	seanh	ckan-sprint-2012-06-25	closed	wontfix	Fix cz.ckan.org theme and customisations		1337158553000000	1338397982000000
2396	defect	seanh	seanh	ckan-sprint-2012-05-29	closed	fixed	Tag list page loads very slowly		1337160153000000	1337268769000000
2397	task	seanh	seanh	ckan-sprint-2012-05-29	closed	fixed	Write CKAN docstring guidelines		1337160193000000	1337963273000000
2398	defect	seanh	seanh	ckan-sprint-2012-05-29	closed	duplicate	Ubuntu 10.04 source install instructions not working?		1337160253000000	1337963342000000
2428	enhancement	seanh	seanh	ckan-v1.8	closed	fixed	Reduce/sort out CKAN's dependencies	"This ticket depends on #2423, #2424, #2425, #2426 and #2427.

- Try to remove as many of CKAN's dependencies as possible

- Try to update as many of the remaining ones as possible

- Try to get rid of some dependencies that are fetched from random hg or git servers instead of pypi

- Then try to get the order of deps in the requirements files right so that you end up with the right versions of everything

- Each module in the dependencies file should specify the exact version, to protect us from changes due to them upgrading versions under us and breaking things or changing their own deps

- Consider shipping dependencies with ckan instead of pip installing them?

- This is related to moving CKAN to Ubuntu 12.04

- Add a test case that runs pip freeze and tests that all the right versions of deps are installed?
"	1337946896000000	1341314475000000
2429	enhancement	seanh	seanh	ckan-sprint-2012-06-25	closed	fixed	Get track to send notifications when tickets get assigned to peopel		1337947043000000	1338374447000000
2430	enhancement	seanh	seanh	ckan-v1.9	accepted		Look into marking tickets as dependent on other tickets in trac		1337947086000000	1341234346000000
2431	enhancement	seanh	seanh	ckan-v1.9	accepted		Trac git integration		1337947128000000	1341234362000000
2432	enhancement	seanh	seanh	ckan-sprint-2012-06-25	closed	fixed	Draft new trac methodology	"We want a new way of working with trac tickets and milestones etc:

- We will have a single queue of tickets, ordered by priority, that we all work on

- Tickets only get into the queue after triaging (which the devs do at regular intervals), so we and others can use new trac tickets as a dumping ground for any and all CKAN bugs and ideas without messing up the priority queue. Therefore the priority queue needs to be a special CKAN milestone or something. The devs need to regularly go through the inbox of untriaged tickets and triage them all, moving them all out of the inbox.

- Will we also assign tickets from the priority queue to individual devs in sprint planning meetings like we currently do?
"	1337947327000000	1338325120000000
2434	defect	seanh	seanh	ckan-v1.8	closed	fixed	Fix autodoc on readthedocs again		1337962353000000	1350296272000000
2435	defect	seanh	seanh	ckan-v1.8	closed	worksforme	ckanext-qa doesn't work with CKAN 1.7+		1337963228000000	1340810257000000
2436	enhancement	seanh	seanh	ckan-sprint-2012-05-29	closed	fixed	Write instructions for booting CKAN demo instances on s031		1337968345000000	1337968356000000
2466	task	seanh	seanh	ckan-v1.9	accepted		Fix spam handling on trac		1338213103000000	1341234539000000
2469	enhancement	seanh	seanh	ckanbuild	closed	duplicate	Find a better way to deploy CKAN instances	"We want a much quicker and easier way of deploying multiple CKAN instances to the same or multiple servers and managing them (e.g. configuration, themes, extensions, upgrades, and adding new instances). Currently we just deploy and manage each instance separately, which doesn't scale.. We'd also want to have (as far as possible) one single way of deploying CKAN that works across different situations, e.g. deployment server, demo server, jenkins, local deployment for development, these should share the same tools for installing CKAN and its dependencies.

Components that this might include:

Virtualenv bootstrap script https://gist.github.com/2206132

Ansible http://ansible.github.com/

CKAN deb package"	1338216064000000	1340639702000000
2475	enhancement	seanh	seanh	ckan-v1.9	accepted		Upgrade trac.ckan.org to trac 0.12.3	"This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

http://trac.edgewall.org/wiki/ChangeLog"	1338317176000000	1341234564000000
2476	defect	seanh	johnglover	ckan-sprint-2012-06-25	closed	wontfix	Check that translating lists of strings is being tested in multilingual tests	Check that lists of strings are being correctly translated. See https://github.com/okfn/ckan/commit/f1d68c3d2d4d25a0c0f8a89a68940643fc19b156	1338378078000000	1339151396000000
2477	enhancement	seanh	shevski	ckan-v1.9	accepted		Add extensions section to readthedocs	Need a central up to date place for currently supported extensions (closest we have is http://wiki.ckan.org/List_of_Extensions) & their documentation. I suggest this gets added to  a dedicated section of docs.ckan.org	1338393244000000	1343124123000000
2478	enhancement	seanh	seanh	ckan-v1.9	accepted		Get NeverNotifyUpdaterPlugin installed on trac.ckan.org		1338396734000000	1341234591000000
2481	enhancement	seanh	seanh	ckan-sprint-2012-06-25	closed	fixed	Move Coding_Standards.rst into Sphinx		1338467473000000	1338484696000000
2490	defect	seanh	seanh	ckan-sprint-2012-06-25	closed	fixed	Dataset term translation isn't working on user pages, ckanext/multilingual test failing		1338822577000000	1338824047000000
2506	enhancement	seanh	seanh	ckan-sprint-2012-06-25	closed	fixed	Pull latest translations from Transifex into CKAN 1.7.1		1339175957000000	1339596352000000
2507	enhancement	seanh	seanh	ckan-sprint-2012-06-25	closed	fixed	Search button on dataset search page wraps onto next line	If you change the language to one where the word for 'Search' is longer than the English word (e.g. fr) then the search button wraps onto the next line, below the text field, which looks bad.	1339409432000000	1339596399000000
2516	enhancement	seanh	seanh	ckan-v1.9	accepted		Make 'Assign to:' field on trac.ckan.org into a dropdown list	there's a setting for this	1339578442000000	1341234822000000
2520	defect	seanh	seanh	ckan-v1.9	assigned		Document undocumented config options	"There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either:

ckan.admin.name
ckan.admin.email
ckan.default.group_type
ckan.page_cache_enabled
ckan.cache_enabled
ckan.cache_expires
ckan.extra_resource_fields
ckan.extra_resource_group_fields
ckan.storage.key_prefix
ckan.storage.max_content_length
ckan.feeds.authority_name *
ckan.feeds.date *
ckan.feeds.author_name *
ckan.feeds.author_link *
ckan.mail_from
ckan.gravatar_default *
ckan.plugins
ckan.api_url
ckan.auth.profile
ckan.datastore.enabled
ckan.tracking_enabled

There are also some options that are in the default deployment.ini even though they're deprecated:

ckan.async_notifier
carrot_messaging_library
ckan.build_search_index_synchronously

See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html

It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358"	1339588368000000	1340624908000000
2521	defect	seanh	seanh	ckan-sprint-2012-06-25	closed	fixed	synchronous_search plugin missing from deployment.ini_tmpl	It still contains the old build_search_index_synchronously which no longer works	1339595361000000	1339595469000000
2533	defect	seanh	seanh	ckan-v1.8	closed	fixed	ckanext-harvest is broken with ckan 1.8b	"Looks like recent CKAN cleanups broke some imports:

https://gist.github.com/2935639"	1339754707000000	1339755433000000
2537	enhancement	seanh	seanh	ckanbuild	accepted		Test and document ckanbuild	"https://github.com/okfn/ckanbuild

Verify that what's there so far still works, write a README explaining how it works"	1339775328000000	1340639830000000
2538	enhancement	seanh	seanh	ckanbuild	accepted		Add multiple-instance support to ckanbuild	"Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command.

May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command."	1339775499000000	1340639836000000
2539	enhancement	seanh	seanh	ckanbuild	accepted		Investigate the existing ckan debian package for ckanbuild	Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh?	1339775661000000	1340639845000000
2540	enhancement	seanh	seanh	ckanbuild	accepted		Implement a way of upgrading ckan sites using ckanbuild	When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once.	1339775740000000	1340639850000000
2541	enhancement	seanh	seanh	ckanbuild	accepted		Add non-core extensions to ckanbuild	We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include.	1339775826000000	1340639856000000
2542	enhancement	seanh	seanh	ckanbuild	accepted		Create jenkins job to run ckanbuild, and run tests	It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests.	1339775888000000	1340639863000000
2592	enhancement	seanh	seanh	ckan-v1.8	closed	fixed	Update and simplify source install instructions	"Make the same instructions work for Ubuntu Lucid and Precise and make them simpler

Also see #1471"	1340633697000000	1341234947000000
2600	defect	seanh	seanh	ckan-v1.8	closed	fixed	"Replace broken ""Login to Add an Organization"" button"	If you're not logged in and you click this button, it takes you to the add group page not the login page. Anyway, the default behaviour everywhere else in CKAN is to hide buttons that you don't have permission to use (e.g. add dataset, edit dataset, etc.)	1340721718000000	1340791910000000
2615	enhancement	seanh	seanh	ckan-v1.8	closed	fixed	Add activity stream activity when a user starts to follow something		1340874249000000	1341235002000000
2619	enhancement	seanh	seanh	ckan-v1.9	assigned		Omit private datasets from public activity streams	"Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered."	1340884140000000	1351531137000000
2627	defect	seanh	seanh	ckan-v1.8	closed	fixed	IDatasetForm.form_to_db_schema() is not called	IDatasetForm's form_to_db_schema() method is not getting called by ckan	1341576508000000	1343125213000000
2628	enhancement	seanh	seanh		closed	duplicate	Add docs for upgrading a source install		1341585472000000	1343392717000000
2629	enhancement	seanh	seanh	ckan-v1.9	closed	duplicate	Move ckanext-examplevocabs into master and document		1341586555000000	1343392596000000
2630	enhancement	seanh	seanh	ckan-v1.8	closed	duplicate	UPdate docs after IDatasetForm schema change		1341586625000000	1343392411000000
2656	defect	seanh	dread		new		Feed with few results has bad paging link, causing exception	"This page http://thedatahub.org/feeds/custom.atom?q=wombat has 0 results and contains a link to http://thedatahub.org/feeds/custom.atom?q=wombat&amp;page=0 which the page=0 causes this exception:
{{{
ckan.lib.search.common.SearchError'>: SOLR returned an error running query
Error: ""'start' parameter cannot be negative""
}}}"	1342001112000000	1342001112000000
2682	defect	seanh	seanh	ckan-v1.8	closed	fixed	CKAN's internal tracking counts each view twice, needs unit tests	"CKAN's internal tracking seems to count each page view twice, the problem appears to be with the SQL in the update_tracking() method in ckan/lib/cli.py.

The internal tracking feature needs some tests, and some of the code could maybe do with some more explanatory comments, e.g. what is the intended difference between count and running_total?"	1342446402000000	1343225636000000
2693	enhancement	seanh	toby	ckan-v1.8	closed	fixed	License is out of date	"https://github.com/okfn/ckan/blob/master/LICENSE.txt mentions requires/lucid_conflict.txt which is no longer the case for 1.8 and needs updating

not sure where requirements-testing/docs fit in

@sean I've assigned this to you as you changed the requirements and my plate is full feel free to pass on to someone else"	1342605447000000	1343148356000000
2756	enhancement	seanh	seanh	ckan-v1.8	closed	fixed	Fix handling of extras in group read	It doesn't pass extras_as_string	1343231514000000	1343232429000000
2766	enhancement	seanh	shevski	demo phase 4	assigned		prevent draft datasets making it to activity stream	"The new ckan creates datasets as part of a 3 phase process.  To allow for this partially created datasets can have a state that is 'draft' or 'draft-complete'.  These datasets should not be seen as active by the activity stream.


If we click 'add dataset' and then complete the first phase of adding a dataset then we end up having a activity stream created.  When we add a resource in the next phase (add data) again an activity stream item xxx added resource to dataset is created


We do not want these adding.  Essentially if a dataset has a state.startswith('draft') then we want the activity stream to ignore all actions involving it.


finally when the state is changed form state.startswith('draft') to state=='active' we want a xxx has created dataset ... to be added to the stream.


The best way to do this would be to branch from 2375-demo-theme-stable and get it to work there.


Let me know if you need any help with this ticket or a better explanation of the problem."	1343318795000000	1344543193000000
2770	defect	seanh	seanh	ckan-v1.8	closed	fixed	Some strings potentially not marked for translation	"Luca De Santis reported these strings not marked in 1.7.1:

> - /user/login.html (""Remember me"" & ""Sign in"")
> - /package/search.html and template/package/group/read.html (Tags, Resource Formats, Groups)
> - /package/resource_read.html (the ""No description"" message)
> - the ""Add related item"" dialog (some strings are i18n savvy, others not)
> - the stats page (labels on the graphs)
> - the history page (as far as the log messages are concerned).

lets check that they are marked in 1.8"	1343386840000000	1343389747000000
2771	enhancement	seanh	seanh	ckan-v1.8	new		Documentation and examples for IDatasetForm and IGroupForm	"Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests.

The IDatasetForm example should use tag vocabularies (two birds with one stone)

The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples.

Tab Vocabularies docs should reference IDatasetForm example.

When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe)

There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date."	1343392238000000	1350303564000000
2772	enhancement	seanh	seanh	ckan-v1.8	closed	fixed	Update transifex for 1.8 translations	Upload the new pot file/resource on transifex, put out the call for translations. Some recent commits on master need to be merged into 1.8 first	1343393299000000	1345027445000000
2805	enhancement	seanh	toby		closed	fixed	zh_TW has no Plural-Forms	"the .po file for zh_TW does not contain Plural-Forms which breaks the i18n JavaScript Extractions for ckan-demo.

I've added the following which may or may not be correct - from web search
""Plural-Forms: nplurals=1; plural=0\n""

I suspect it will get overwritten when translations are pulled from transiflex so passing to you as you are the expert in that realm"	1344253863000000	1344254760000000
2815	defect	seanh	seanh	ckan-v1.8.1	new		db_to_form_package_schema() strips tracking summary, isopen	If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates.	1344444427000000	1350303821000000
2818	defect	seanh	danieljohnlewis	demo phase 4	assigned		Improve related item schema	"Problem: When creating a related item (e.g. a Visualisation), if you don't put in a URL it succeeds, but on the related items and apps pages it renders it as a link to the same page.
Expected: Always require a URL and it should only submit
if one is added"	1344504176000000	1346231718000000
2843	defect	seanh	seanh	ckan-v1.8	closed	fixed	Adding existing datasets to organizations is broken	"On the organization read page there is only an 'Add Dataset' button, not a drop-down with add new dataset or add existing dataset as on the default group page.

On the organization edit page you do have the drop-down, but if you click add existing dataset nothing happens except it appends #datasets to the URL"	1344859137000000	1347270569000000
2845	defect	seanh	seanh	ckan-v1.8	closed	fixed	Editing an organization removes all datasets from the organization	Run CKAN with the publisher profile and organizations plugins enabled. Make an organization. Make a dataset and add it to the organization. Edit the organization and change e.g. its description. After saving the organization, all its datasets have been removed from the organization and now belong to no organization.	1344859286000000	1347271110000000
2870	enhancement	seanh	ross		new		1.8 tag_list not defined	"From: <thedatahub.org@s077.okserver.org>
Date: Thu, Aug 16, 2012 at 3:20 AM
Subject: WebApp Error: <class 'genshi.template.eval.UndefinedError'>: ""tag_list"" not defined
To: ckan-sysadmin@okfn.org, kindly@gmail.com


URL: http://ckan.net/dataset/uk-postboxes
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:155 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
       
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
<<                                               r'\1', oldpath)
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:239 in __call__
<<          try:
                   res = WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  res = WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:322 in read
<<          template = template[:template.index('.') + 1] + format
       
               return render(template, loader_class=loader)
       
           def comments(self, id):
>>  return render(template, loader_class=loader)
Module ckan.lib.base:153 in render
<<      try:
               return cached_template(template_name, render_template,
                                      loader_class=loader_class)
           except ckan.exceptions.CkanUrlException, e:
               raise
>>  loader_class=loader_class)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:102 in render_template
<<          if loader_class == NewTextTemplate:
                   return literal(stream.render(method=""text"", encoding=None))
       
               return literal(stream.render(method=method, encoding=None,
>>  return literal(stream.render(method=""text"", encoding =None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
       
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out =out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:569 in __call__
<<      def __call__(self, stream):
               strip_markup = self.strip_markup
               for event in stream:
                   if event[0] is TEXT:
                       data = event[1]
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """"""
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """"""
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """"""
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module ckanext.googleanalytics.plugin:93 in download_adder
<<                  [downloaded %s times]</span>'''
                       count = None
                       for mark, (kind, data, pos) in stream:
                           if mark and kind == START:
                               href = data[1].get('href')
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1175 in __call__
<<          """"""
               callable_value = hasattr(self.value, '__call__')
               for mark, (kind, data, pos) in stream:
                   if mark is ENTER:
                       if callable_value:
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """"""
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
       
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.base:565 in _flatten
<<                  elif kind is EXPR:
                           result = _eval_expr(data, ctxt, vars)
                           if result is not None:
                               # First check for a string, otherwise the iterable test
>>  result = _eval_expr(data, ctxt, vars)
Module genshi.template.base:277 in _eval_expr
<<      if vars:
               ctxt.push(vars)
           retval = expr.evaluate(ctxt)
           if vars:
               ctxt.pop()
>>  retval = expr.evaluate(ctxt)
Module genshi.template.eval:178 in evaluate
<<          __traceback_hide__ = 'before_and_this'
               _globals = self._globals(data)
               return eval(self.code, _globals, {'__data__': data})
>>  return eval(self.code, _globals, {'__data__': data})
Module ?:51 in <Expression u""tag_list(c.pkg_dict.get('tags', ''))"">
<<      <li py:if=""c.pkg_dict.get('tags')"" class=""sidebar-section"">
             <h3>Tags</h3>
             ${tag_list(c.pkg_dict.get('tags', ''))}
           </li>
>>  ${tag_list(c.pkg_dict.get('tags', ''))}
Module genshi.template.eval:309 in lookup_name
<<              val = BUILTINS.get(name, val)
                   if val is UNDEFINED:
                       val = cls.undefined(name)
               return val
>>  val = cls.undefined(name)
Module genshi.template.eval:410 in undefined
<<          """"""Raise an ``UndefinedError`` immediately.""""""
               __traceback_hide__ = True
               raise UndefinedError(key, owner=owner)
>>  raise UndefinedError(key, owner=owner)
UndefinedError: ""tag_list"" not defined
CGI Variables
CKAN_CURRENT_URL	'/dataset/uk-postboxes'
CKAN_LANG	'en'
CKAN_LANG_IS_DEFAULT	True
CKAN_PAGE_CACHABLE	True
CONTENT_TYPE	'; charset=utf-8'
DOCUMENT_ROOT	'/etc/apache2/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'text/html,text/plain,text/xml'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'vi,en-us;q=0.7,en;q=0.3'
HTTP_CONNECTION	'close'
HTTP_HOST	'ckan.net'
HTTP_USER_AGENT	'coccoc/1.0 ()'
PATH_INFO	'/dataset/uk-postboxes'
PATH_TRANSLATED	'/etc/ckan/datahub/apache.wsgi/dataset/uk-postboxes'
REMOTE_ADDR	'127.0.0.1'
REMOTE_PORT	'47065'
REQUEST_METHOD	'GET'
REQUEST_URI	'/dataset/uk-postboxes'
SCRIPT_FILENAME	'/etc/ckan/datahub/apache.wsgi'
SERVER_ADDR	'127.0.0.1'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.0'
SERVER_SIGNATURE	'<address>Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.22 (Ubuntu)'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x7f5312a51650>
beaker.cache	<beaker.cache.CacheManager object at 0x7f5312a51790>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f5312a516d0>>
beaker.session	{'_accessed_time': 1345083636.079552, '_creation_time': 1345083636.079552}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.handler_script	''
mod_wsgi.input_chunked	'0'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'8080'
mod_wsgi.process_group	'datahub'
mod_wsgi.request_handler	'wsgi-script'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(3, 3)
paste.cookies	(<SimpleCookie: >, '')
paste.registry	<paste.registry.Registry object at 0x7f52e47477d0>
paste.throw_errors	True
pylons.action_method	<bound method PackageController.read of <ckan.controllers.package.PackageController object at 0x7f52e477b810>>
pylons.controller	<ckan.controllers.package.PackageController object at 0x7f52e477b810>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x7f52e477be90>
pylons.routes_dict	{'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'}
repoze.who.logger	<logging.Logger object at 0x7f53127b2b50>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 139994769488336>, 'friendlyform': <FriendlyFormPlugin 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>}
routes.route	<routes.route.Route object at 0x7f5312955a50>
routes.url	<routes.util.URLGenerator object at 0x7f52eb8c9090>
webob._parsed_query_vars	(GET([]), '')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f5308983a80>
wsgi.version	(1, 1)
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0x7f52eb8c9090>, {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'})
"	1345114732000000	1345114732000000
2952	enhancement	seanh	icmurray	ckan 2.0	new		incorporate javascriopt translations into translations workflow	There are now javascript translations.  There's a paster command, `trans js`.  This is a ticket to ensure that using `trans js` is documented in the normal ckan translation workflow (which is followed after a feature freeze, prior to release).  And to see if there's any need for further incorporation into the current translation process.	1349118945000000	1349118945000000
3005	enhancement	seanh	seanh	ckan 2.0	new		Add following/unfollowing of groups: model, API, frontend, tests		1351013744000000	1351013744000000
3009	enhancement	seanh	seanh	ckan 2.0	closed	fixed	On-site notification of new dashboard activity	"We want a ""bubble"" with a number in it to appear in the top-right of the site next to your user name when you're logged in, telling you how many new activity stream items you have in your dashboard activity stream.

There's a github issue for this here:

https://github.com/okfn/ckanext-pdeu/issues/11"	1351615762000000	1353949938000000
3028	enhancement	seanh	seanh	ckan 2.0	new		Feature: dashboard activity stream filtering	https://github.com/okfn/ckanext-pdeu/issues/13	1355141244000000	1355141244000000
3029	defect	seanh	dread		assigned		JSONP parameter scuppers Search in API	"{{{http://datahub.io/api/2/search/package?jsonp=jsonpcallback&q=canada}}} returns
{{{
{""count"": 0, ""results"": []}
}}}
I believe this worked in CKAN 1.4 or 1.5, but it is broken on 1.7.1, 1.8 and whatever demo.ckan.org is running. I suspect the jsonpcallback parameter is getting sent to SOLR.

This bug prevents using javascript on another site to search CKAN (although hopefully the action API would work)."	1355238035000000	1355243824000000
1096	defect	rufuspollock	pudo	ckan-future	new		[super] CKAN Hosted	"Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

 * introduce model.Site and c.site
  * site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI) 
 * Subdomain detector to activate sites.
 * use site in Authorizer instead of System, have a NullSite for global things
 * allow cross-site search
 * packages are in a list of sites, m:n rather than 1:n
  * list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally. "	1303235062000000	1339774484000000
1565	enhancement	Rufus Pollock	jilly mathews	ckan-future	new		Admin dashboard finished?	Is testing complete and ready for release?	1324293092000000	1324293092000000
1279	enhancement	ross	amercader	ckan-v1.8	closed	duplicate	Refactor or deprecate Authorization Groups	"Authorization Groups are going to be deprecated in 1.8

It is still unclear how Authorization Groups fit in the new authz model, so they have been left out of the logic and auth refactoring (Tickets #1229 and #1253)"	1313415367000000	1338212028000000
1508	enhancement	ross	rgrp	ckan-v1.6	closed	fixed	[super] Finalize and deploy some of our major (new) features as of Dec 2011	"This ticket is about finalizing, then deploying (to http://thedatahub.org/), and finally announcing some of recent major new features such as:

  * Deploy wordpresser extension for thedatahub (so we can start blogging) - #1533 - WONTFIX
  * ~~(Deploy new Resource and Dataset pages (invite comments))~~
  * Get thedatahub geo-ified i.e. ckanext-spatial - #1601 (?)
    * Any additional dev that needs to be done? (e.g. geo in search ...?)
  * ~~(Deploy / re-enable ckanext-googleanalytics on thedatahub - #1451 (more stuff #1566))~~
    * Rename the extension to ckanext-analytics?
  * Custom forms (and use of 'dataset types') - Ticket (??)
  * ~~(Get QA deployed and running - #1582)~~

It would be best if this was done by someone *unfamiliar* with the extensions so feedback could be provided on quality of install docs etc etc."	1323088817000000	1328000871000000
1512	defect	ross	rgrp	ckan-v1.8	closed	fixed	Modifying user name makes you semi-log-out	"If you modify your user name, it looks like you are logged out, but you are actually logged in and therefore can't login. (Reported by Pablo)

As a work around, you need to logout and log back in. 

= implementation = 
* We should force this by auto-logging you out (DR: maybe!)"	1323100214000000	1342781122000000
1551	enhancement	ross	ross	ckan-backlog	closed	fixed	CKAN auth for webstore changes	Webstore should use auth api ( #1550 ) for authenticating users accessing webstore rather than talking directly to the CKAN database. We also need it to suppose /user/  urls and /userid/ for accessing databases.	1324049966000000	1346662048000000
1574	enhancement	ross	ross	ckan-v1.7	closed	fixed	[super] Storage changes including automated save to Webstore	"Allow uploading of files to push the data into webstore.  Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore.

 1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs [5d]

 2. The user is provided the link to the file as currently. 

 3. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) [5d] - took 6

  * It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file.

 4. The archiver will continue to move the file into permanent storage as per current practice.

== Tickets

 * #1608 Merge storage into Core [5d]
 * #1609 Celery task for ckanext-archiver to write to webstore. [5d] (took 6)
 * #1687: Deploy to datahub [1d]
 * #1681: Core storage documentation [1d?]
  * Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc
  * External: here's how to configure it and here's what you get ... (goes in main sphinx docs)
"	1324315633000000	1338202955000000
1579	enhancement	ross	ross	ckan-backlog	closed	fixed	Full text search in Webstore	We should allow full text search within databases for individual databases, although we may want to wait for any pending ports to postgres.	1324385698000000	1338212496000000
1608	enhancement	ross	ross	ckan-sprint-2012-01-23	closed	fixed	Merge storage into Core	"As part of [#1574 Storage Changes] and http://ckan.okfnpad.org/newstorage it was decided to merge ckanext-storage back into core.

Other changes:

 * Remove /storage/upload page (we have frontend integration elsewhere now and we should also ensure that upload items relate to a resource or other domain object).
 * Provide a local path as well as remote path for form upload requests in API so that frontend can see url to CKAN path rather than cloud storage path ...
"	1325582044000000	1330036982000000
1609	enhancement	ross	ross	ckan-sprint-2012-01-23	closed	fixed	Celery task for ckanext-archiver to write to webstore.	"From super Storage changes - #1574 - and http://ckan.okfnpad.org/newstorage we determined that ckanext-archiver should have a celery task for grabbing local file uploads and writing to webstore

== Analysis ==

When I upload a file to CKAN:

    * End up with file in permanent storage
    * IF file is ot type ... csv,xls,xlsx,sqlite,.sql
        * End up with new db in webstore
            * Where? {username}/{resource-id}/...
                * If a single table: name it after the file name (appropriately slugified)
            * A resource *always* corresponds to a 'database' in webstore ...
            * In Data Explorer have ""Sheets"" tab ...
    * Resource url = /dataset/{x}/resource/{y}/link -> cached_url ..."	1325582253000000	1327057030000000
1610	enhancement	ross	ross		closed	duplicate	Move webstore to Postgres instead of Sqlite	"The default backing store for webstore should be Postgres and not sqlite.  This was agreed as part of the #1574 storage changes and on http://ckan.okfnpad.org/newstorage

"	1325587341000000	1325852472000000
1611	enhancement	ross	ross		closed	duplicate	Implement auth API calls for webstore/external use	"As part of #1574 we decided that it would be better for webstore (and future external services) to be able to authenticate simply with CKAN-Core.  

Currently webstore access the CKAN database to obtain the key for the user but it would be better if this connection was not so tightly bound and that webstore used an API as any other external service might.

Need to discuss further with dread"	1325590191000000	1325846987000000
1649	enhancement	ross	ross	ckan-sprint-2012-03-19	closed	fixed	Verify RDF Extension (ckanext-rdf) works	"* Make sure that the extension works as it should and isn't broken on current releases [3d]

* Make it work with the EC schema, including extras [2d]
"	1326709096000000	1338205274000000
1650	enhancement	ross	ross	ckan-future	closed	wontfix	Add new language/translation of fields for datasets.	"Datasets should have a field that specifies the language used in the dataset, this should be a two character ISO 639-1 code (see http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes for list) . Also make sure all tests and migrations are resolved. [2d]

Determine the best approach to having 'translation_of' references to show which dataset the current one is a translation of.  This may be a translation of field, or a 'copied from' field or another form of package relationship. [2d]
"	1326709392000000	1346670010000000
1657	enhancement	ross	ross		closed	wontfix	Configuration for reverse proxying	"Provide configuration for reverse proxying that will correctly handle the mapping of a URL to a sub-folder (using X-SCRIPT-NAME)

* Analysis of the best solution [1d]

* Implement reverse proxying in tandem with #1653  [2d]

* Document and store the configuration files. [1d]
"	1326711580000000	1338206640000000
1658	enhancement	ross	ross		closed	wontfix	Implement proxy caching	"Implement proxy caching for performance gains

* Analysis of best solution and what we can deliver in a reasonable time [2d]

* Checks for presence of cookie 'auth_tkt' and appropriate setup for media  [2d]

* Remove session requirements for user who isn't logged in [3d]
   
* Document the configuration and setup so that it can be easily modified and replicated [1d]

"	1326712354000000	1338206622000000
1669	enhancement	ross	ross	ckan-v1.7	closed	fixed	[super] Publisher profile	"Implement the necessary backend changes to build profile for publishers using the group refactor.

See http://ckan.okfnpad.org/publisherprofiles

Subtickets are at:

 * Publisher profile analysis #1670 [2d]
 * Publisher profile sysadmin authz #1671 [1d]
 * Publisher profile user authz #1672 [1d]
 * Test auth logic hooks for publisher profile #1674 [3d]
 * Publisher admin authz #1675 [3d]
 * Publisher editor authz #1676 [2d]
 * Add approval status field to groups #1673 [1d]"	1326802473000000	1338204071000000
1670	enhancement	ross	ross	ckan-sprint-2012-03-05	closed	fixed	Publisher profile documentation	"Super #1669

Analysis of all permissions for users given their publishers to ensure that we have covered all of the bases and haven't missed any use-cases."	1326802549000000	1338205180000000
1671	enhancement	ross	ross	ckan-sprint-2012-02-06	closed	fixed	Publisher profile sysadmin authz	"Super #1669

Implement authz for sysadmins in publisher profile"	1326802607000000	1328526614000000
1672	enhancement	ross	ross	ckan-sprint-2012-02-06	closed	fixed	Publisher profile user authz	"Super #1669

Authz for normal users in publisher profile as per sysadmin"	1326802680000000	1328526625000000
1673	enhancement	ross	ross	ckan-sprint-2012-02-06	closed	fixed	Add approval status field to groups	"Super #1669

As groups need to be authorised in some cases, particularly publisher profiles, we need a field to denote how this is managed.  

We should add an 'approval_status' field that will be used to denote its status.  By default this will be ""approved"" but this will allow us to override it with a ""pending"" or ""denied"" status.  The latter should also change the main status of the Group."	1326802753000000	1328526644000000
1674	enhancement	ross	ross	ckan-sprint-2012-02-06	closed	fixed	Test auth logic hooks for publisher profile	"Super #1669

Test all of the new auth logic hooks for publisher profile to make sure they work as expected. Also make sure that we have a test to compare the naming of the core auth funcs with the profile hooks (so we can see what may have changed in one but not the other)."	1326802841000000	1328526677000000
1675	enhancement	ross	ross	ckan-sprint-2012-02-20	closed	fixed	Publisher admin authz	"Super #1669

Authz for the publisher admin (who has permission to administer the publisher group themselves)."	1326802923000000	1328709915000000
1676	enhancement	ross	ross	ckan-sprint-2012-02-06	closed	fixed	Publisher editor authz	"Super #1669

Editor authz for publisher groups need to be implemented and tested."	1326802971000000	1328526670000000
1680	enhancement	ross	ross		closed	wontfix	Group refactor top level element	"The new group refactor allows for a hierarchy of groups where each group has a type - to be able to implement a tree of groups.

It will need a flag within the group to denote that it is a top level group, to circumvent the need to determine whether the group is a child of a parent node (for a specific use case - a publisher representing the department that only contains publishers).

Suggest 'is_top_level'"	1326896362000000	1326900832000000
1681	enhancement	ross	ross		closed	fixed	Core storage document	"Document all of the changes and usage for #1574 so that we can mark it 'done done'.

Make sure we document any setup issues/processes and the settings.py setup required for CKAN so that Mark can announce next week and it should be easily deployable/testable.

Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc

External: here's how to configure it and here's what you get ... (goes in main sphinx docs)"	1326904939000000	1338202930000000
1685	enhancement	ross	ross	ckan-backlog	closed	wontfix	Move archiver download function to core	Archiver is included in some places just for access to download(), we should move this to lib in core.	1326984170000000	1346670530000000
1687	enhancement	ross	ross		closed	invalid	Deploy new storage changes from master to datahub	We need to deploy the new storage changes (and webstore) to thedatahub for further testing	1327056976000000	1329338671000000
1690	enhancement	ross	ross		closed	fixed	Rename storage settings with the ckan prefix	Missed the ckan prefix on the storage settings names so this needs to be fixed.	1327064844000000	1327066713000000
1693	defect	ross	ross		closed	fixed	register_pluggable_behaviour should not be in routing	"We should try and move register_pluggable_behaviour into environment and out of routing as it is causing import issues when using simple search.

These functions already take a map and so they should be fine any time after the call to make_map"	1327321474000000	1338206601000000
1742	enhancement	ross	ross	ckan-sprint-2012-03-19	closed	fixed	Naming of a new package/group can clash with a route	"It is possible to create a group or a dataset called new, or edit, which will then be inaccessible as it is blocked by the existing routes with those names.

We should maintain a list of invalid names in the verification that happens behind the scenes in the UI."	1328116317000000	1331555193000000
1755	enhancement	ross	ross	ckan-sprint-2012-02-20	closed	fixed	DGU Alpha paged view of publishers	The publisher index page should show an alpha paged list of publishers.	1328530634000000	1328631064000000
1756	enhancement	ross	ross	ckan-sprint-2012-02-20	closed	fixed	DGU Treeview of hierarchical publishers	The publisher index page should show a treeview of the publishers within the system. Probably behind a tab.	1328530689000000	1328709846000000
1757	enhancement	ross	ross	ckan-sprint-2012-03-05	closed	fixed	DGU Hide administrators from public view	On publisher view page - hide administrators from public view - only logged in admins of group should see them.	1328530775000000	1338205187000000
1758	enhancement	ross	ross	ckan-sprint-2012-02-20	closed	fixed	DGU Check user management auth	Ensure publisher edit page only allows admins to manage users. #1675 is related.	1328530915000000	1328709897000000
1759	enhancement	ross	ross	ckan-sprint-2012-02-20	closed	fixed	DGU Create publisher form access	Only sysadmin can access create publisher form.	1328530956000000	1329732744000000
1760	enhancement	ross	ross	ckan-sprint-2012-02-20	closed	fixed	DGU Extract current publisher hierarchy	" Get access to live Drupal MySQL database to get latest publisher list for migration.

Write a one-off script for import of the data."	1328531001000000	1328709560000000
1761	enhancement	ross	ross	ckan-sprint-2012-02-20	closed	fixed	DGU History visibility	History on user profile should only be visible to a user who is a member of at least one group.	1328532889000000	1328716686000000
1762	enhancement	ross	ross	ckan-sprint-2012-02-20	closed	fixed	DGU Join publisher form	"A new form for create publisher -> Add publisher access to your account'
New wizard 
Autocomplete publisher name if linked from publisher view.
Email address used to publish should be activated/validated on first use. Original email still used to log in.
"	1328532992000000	1329729305000000
1793	enhancement	ross	ross		closed	duplicate	Publisher changes	Changes to publishers for customer	1329214299000000	1329214522000000
1794	enhancement	ross	ross		closed	fixed	Publisher changes	Changes to publishers for customer that would otherwise clutter up the master branch.	1329214301000000	1332148972000000
1795	enhancement	ross	ross		closed	wontfix	Add approval_status to Package (as for Group)	"The Package model should have a approval_status as the Group model does.
See migration 049."	1329234060000000	1338206587000000
1796	enhancement	ross	dread	ckan-future	closed	fixed	Get rid of lxml dependency	"lxml is used in CKAN core in two places:
 * ckan/lib/helpers.py:239 Parsing Markdown and ensuring it is valid XML (i.e. tags close nicely so the HTML of the rest of the CKAN page is not disruptable).
 * reading the SOLR config XML to determine what version it is.

Both of these could be done with xml.dom built into Python and would save us a sizeable and slightly problematic dependency.

lxml is needed by these extensions: ckanext-inspire, ckanext-harvest, ckanext-csw & ckanext-wordpresser, so we'd have to add this to their requirements."	1329244440000000	1340635557000000
2204	enhancement	ross	rgrp	ckan-sprint-2012-04-30	closed	fixed	[super] Related (Stuff) Extension	"This is a reworking of the existing apps extension.

Initial proposal at http://wiki.ckan.org/Proposals#Apps_in_CKAN and http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN

== Naming ==

What do we call this extension?

 * related
  * RP best IMO but perhaps too close to separate relationships concept?
 * relatedstuff
 * links
 * references

== Proposal ==

""Related Stuff"": Apps as in an application (website/service/tool) that uses this dataset (as in Apps and Ideas extension)

 * This case seems a general example of wanting to link some external thing to a dataset (e.g. this policy paper refs this dataset, this application uses this dataset, this visualization is built on this dataset cf http://lists.okfn.org/pipermail/ckan-discuss/2011-August/001607.html)

== Implementation ==

New table named Related with following structure

{{{
|| id (int) || type || title || description (markdown) || image_url || owner_id || url || created (timestamp)
}}}


 * type = Idea | App | API | Visualization | Post | Paper | News Article
  * Suggest we make this a ckan.ini config option (comma separated ...?)
        * Do we want the possibility of different templates for different types of Related objects?

  * image: ?? Depends where we store images. Simplest option would be to change to image_url and leave it to users to have already uploaded an image somewhere. If not we need to support image uploading and storage. See #1692 (add image attribute to datasets and groups) for more discussion, once implemented the URL here can be an internal url.

 * owner_id = user_id or creating user (see authorization below) 

Related2Dataset (note that related_id, dataset_id tuple should be unique). This allows for m2m connections. If a given related item is only with one dataset this could be simplified. May contain status so dataset owner can turn this on/off.

{{{
|| id || dataset_id || related_id || status
}}}

status should be used to allow for a dataset owner (for dataset_id) to de-activate the relationship between the dataset and the related.

=== Url ===

 * /dataset/{dataset-name}/related/{related-item-id}/{related-item-title-stringified}
  * If a reference item could exist in its own right (and perhaps refer to multiple datasets then it should get its own url at e.g. /related/{id}
 * /dataset/{dataset-name}/related/add => Modal dialog on related tab so we can use API to create them.

/dataset/{id}/related <- list
    * use image_url for small icon etc, title description (shortened?)
    * Click through to full related item (optional)
    * dataset owner is shown show / hide button ... (or on /dataset/{id}/related/{id} )
    * related owner sees an edit button / icon (pops up modal)

/dataset/{id}/related/{id} (optional)

/dataset/{id} will have a Related tab (with bubble with count).
  * Drop down with Add Related -> Pop-up modal and save via API
      (Not used: /related/add with dataset prefilled ... )
/related/{id}/edit

=== Authorization ===

Addition of related item be considered orthogonal to datasets (and hence with separate authorization i.e. i can add the info that site X uses dataset Y without needing permission to edit dataset Y).

Thus any logged in user could add a Related item. We set the owner of the related item to creating user and going forward only that user or a sysadmin can update or delete.

NB: we could have a system where datasets owners have to approve related items before they show up next to their dataset. This would add substantial complexity so I propose we leave out of phase 1.

=== Tasks and estimates {7.5d} ===

[x] Model + Migration for Related table. {0.75d}

[x] Controller for Related (or relevant sections in Package controller). {0.75d}

[x] Routing setup. {0.25d}

[x] Schema for related. {0.5d}

[x] Logic layer actions. {1.0d}

[x] Auth (default + publisher). {0.5d}

[x] Templates + Dataset changes (new tab etc). {1.0d}

[x] JS Application for adding Related objects in a modal. {1.0d}

[x] Testing. {0.75d}

[x] Dataset owner disabling of Related (via M2M table). {0.5d}
     - Updated to allow author of related to delete as well

[x] Documentation. {0.25d}
"	1330862992000000	1337594723000000
2209	enhancement	ross	ross	ckan-sprint-2012-03-19	closed	fixed	Object->RDF mapping	"Super is [1649]

We need to provide customisable mappings between CKAN models and RDF. This needs to be easily configurable for different clients and should take into account objects related to the model being mapped.

Current contenders for impl are:

1. ordf.org's Object Description Mapper (which uses OWL)

2. RDFAlchemy, although this looks like it may result in fixed mappings

3. A custom scheme with a simple json file

4. A template based approach using Genshi templates?

Approach is to get a mapping complete of the attributes directly in the package class, and then provide mappings for related objects (tags/resources etc).

We'll then need to be able to publish the RDF to a triplestore.


See http://ckan.okfnpad.org/rdfmapping for further analysis.

"	1331027277000000	1336566051000000
2210	enhancement	ross	ross	ckan-sprint-2012-03-19	closed	fixed	Add method to group for adding members	"It is necessary to add a helper method to Group for adding members to that group so that 

- nobody needs to work with the Member model
- It handles existing membership and existing membership with a different capacity"	1331028470000000	1331549369000000
2211	enhancement	ross	ross	ckan-sprint-2012-03-19	closed	fixed	Logic layer access to publisher functionality	"Most of the get_group/add_member style calls should be handled, or at least provided via logic layer.

Don't need to break existing code, but do need to provide access via logic layer"	1331028751000000	1331549406000000
2228	enhancement	ross	ross	ckan-sprint-2012-03-19	closed	fixed	IGroupForm should allow overriding of non-edit templates	IGroupForm should allow overriding of non-edit templates so that it can specify the index +/or search +/or view.	1331637803000000	1332149030000000
2239	enhancement	ross	ross	ckan-sprint-2012-04-30	closed	fixed	EC ODP RDF/N3 templates	"Implement the EC/ODP template for RDF to match the requested schemas.

Using the previous work on allowing for templatised RDF representations we should implement the proposed schema for EC/ODP.

We also agreed to allow upload of RDF as an extra (raised to top level in package schema) which will instead be used for the RDF template.  On insertion extra CKAN triples will be added... initially this is just owl:sameAs although we're awaiting EC/ODP agreement on which triples we should add to show the CKAN link.

"	1332169703000000	1335516499000000
2240	enhancement	ross	ross	ckan-sprint-2012-04-02	closed	fixed	Document CityData ideas	Document the ideas raised on 14/03 for CityData and distribute to team for discussion	1332169761000000	1332408769000000
2255	enhancement	ross	ross	ckan-sprint-2012-04-30	closed	fixed	Organizations (Privacy + Ownership)	"[x] The publisher_form extension and the 'publisher' component of the system will be renamed to organizations to both make its functionality clear, and to differentiate it from the publish auth profile.

Ownership of datasets is important and this ticket will enforce membership for a dataset to a single individual organization.  The workflow for adding a dataset to a publisher/organization will stay the same but there will be no mechanism for adding existing datasets to an organization from within the organization forms.

 [x] New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below)

 [x] Datasets created from direct access to /dataset/new will have a drop down showing all of the organizations available to the current user.

 [x] When editing a dataset the organization can only be changed if the current user is the admin of the current organization, and only then to one of the organizations available to the current user.
  
    Only Organization admin can see the edit Organization form.

[x] Privacy will be enforced by changing the dataset form to have a public/private switch next to the organization drop-down which will be set as the capacity with which the dataset is added to an Organization.  Private means visible only within the Organization, and should be the default.

[x] Search will be affected as the public/private status of a dataset will need to be added to the schema 

[x] Only public datasets showing in site search, and private showing in organization dataset search (if user is within the organization). 

[x] Make sure resources are restricted in visibility if the parent dataset is private

[x] Documentation should be updated to reflect these changes.

[x] Verify, then add, then re-add to documentation.
 
 * A User joins or creates an Organization

 * If creator of the Organization User becomes administrator of the Organization otherwise they become a Member.

 * ~~Creation of new Organization must be approved by a System Administrator~~

 * New Members must be approved by the Organization Administrator


"	1332417218000000	1334663670000000
2280	enhancement	ross	johnglover	ckan-sprint-2012-04-30	closed	fixed	Move ecportal to organizations, add history template		1333552029000000	1334834302000000
2293	defect	ross	rgrp	ckan-sprint-2012-06-25	closed	fixed	Rename of Group results in disappearance of associated datasets from group page	"Rename of group makes all the datasets associated with the group disappear on the group listing page. (But they are still there if you look at edit).

Suspect this is because we are using group name (rather than id!!) in the search index ...

As group rename should be rare I'm marking this as minor though actual effect is major.

Assigning to rossjones for review and cost assessment."	1334432078000000	1340266982000000
2303	enhancement	ross	ross	ckan-sprint-2012-04-30	closed	fixed	Move ECPortal rdf dump paster	Move ECPortal rdf dump paster command into core so that it can be run on thedatahub for  virtuoso install.	1334582692000000	1334664098000000
2312	enhancement	ross	ross	ckan-future	closed	duplicate	Analysis of how datasets could belong to users instead of Groups	"** DUPLICATE OF #2548

Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset.  To resolve this it would be great if a dataset could be attached to a user directly - find out how."	1334663770000000	1340188765000000
2322	enhancement	ross	rgrp	ckan-sprint-2012-05-29	closed	fixed	Convenience offset on end of resource urls: /raw, /api, /viewer	"== Raw ==

/raw or /download (latter may be better)

{{{
/dataset/{id}/resource/{resource-id}/raw

=> redirects to resource url (download)
}}}

== API ==

{{{
/dataset/{id}/resource/{resource-id}/api[/....]

=> either redirects or directly serves /api/data/{resource-id}
}}}

== Viewer (Explorer?) ==

Do data viewer but without anything else showing (full-screen?). this would not be just the data explorer but other cases too (including iframe etc ...)

{{{
/dataset/{id}/resource/{resource-id}/viewer

Resource (pre)-viewer without anything else (perhaps include resource title)
}}}

-----

Assigning for kindly for review.

These would take about 20m to implement and would be *very* convenient from my experience of using the data API.

----
Implementation at enhancement-2322-convenience-urls"	1335236581000000	1344086640000000
2323	enhancement	ross	ross	ckan-backlog	closed	fixed	Auto-complete in organizations	"Apparently auto-complete in Organizations may not work.  Check and fix.

See https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5"	1335274354000000	1340188400000000
2349	enhancement	ross	ross	ckan-sprint-2012-06-25	closed	fixed	Make sure semantic.datahub.io gets created	"
[x] Redeploy Sparql endpoint 
     Done at linkeddata.openlinksw.com/sparql

[x] Make sure it is available at datahub.io/sparql or semantic.datahub.io/sparql

[x] Generate daily dumps

[x] Make dumps available via web

[x] Notify Hugh once running

[x] Announce to LD guys

[x]  Think about how we can apply this to publicdata.eu (can we do the same?) ... (being done elsewhere)"	1335880412000000	1340356629000000
2353	enhancement	ross	ross	ckan-sprint-2012-05-15	closed	fixed	Provisional user stories for CMS + Blogging	Provide some provisional user stories for the CMS/Blogging feature.  Should include some of the functionality from wordpresser but be more generic.	1335884700000000	1336566004000000
2364	enhancement	ross	ross	ckan-sprint-2012-05-15	closed	fixed	Investigation of payment gateways	Need to investigate alternative payment gateways	1335889757000000	1337077531000000
2365	enhancement	ross	ross	ckan-sprint-2012-06-25	closed	fixed	Investigation of multisite	"What would as a multisite CKAN look like?

This is really part of the work around turnkey/hosted CKAN"	1335889911000000	1340266964000000
2366	enhancement	ross	amercader	ckan-sprint-2012-06-25	closed	fixed	Remove Authorization Groups	1.7 will hide the link on the footer and put a deprecation notice on the related pages, but they should be completely removed on 1.8.	1335892563000000	1346316322000000
2374	enhancement	ross	dread	ckan-v1.8	closed	worksforme	tag search paging	Currently in the logic function tag_search you can specify limit and offset, but no count is returned. Therefore pagination is not possible for tag results. This is desired though.	1336154025000000	1340287976000000
2379	enhancement	ross	ross	ckan-sprint-2012-06-25	closed	fixed	RDF output, api/sparql	"From: 
Pierre-Yves Vandenbussche

In my use case I need to fetch every sparql endpoint associated to a dataset. With the previous version of your endpoint, I was doing this query:
SELECT DISTINCT ?dataset ?endpoint ?title ?identifier WHERE { 
	?dataset <http://www.w3.org/ns/dcat#distribution> ?distribution. 
	?distribution <http://purl.org/dc/terms/format> [ <http://moat-project.org/ns#taggedWithTag> [ <http://moat-project.org/ns#name> ""api/sparql"" ] ]. 
	?distribution <http://www.w3.org/ns/dcat#accessURL> ?endpoint. 
	?dataset <http://purl.org/dc/terms/title> ?title.  
	?dataset <http://purl.org/dc/terms/identifier> ?identifier. 
} ORDER BY ?title

Using your new version,dcterms:title of a dataset is now a rdfs:label ... OK

Unfortunately, I don't have the information of ""api/sparql""  anymore. So I can not differentiate between a dump file and a SPARQL endpoint... 
____

Add the required information to the RDF template."	1336735947000000	1340033026000000
2380	enhancement	ross	ross	opendatasuite 1	closed	duplicate	DataGM Upgrade	Provide a new test install of DataGM based on the ODS demo site being completed through June.	1337002144000000	1340627558000000
2381	enhancement	ross	ross	ckan-v1.8	closed	fixed	Related item count on ALL dataset pages	Make sure the related item count is available on all dataset pages (inc. follow) when it is enabled.	1337009133000000	1337011327000000
2385	enhancement	ross	ross	ckan-sprint-2012-05-29	closed	fixed	Specifying capacity in search api call	Currently you can specify the capacity in the API search, this should be stripped and default to public for now.	1337092899000000	1337102816000000
2409	enhancement	ross	markw	ckan-sprint-2012-05-29	closed	fixed	Accept headers not handled correctly	"CKAN returns HTML when an Accept header should make it return RDF/XML:

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002237.html
"	1337683898000000	1337856979000000
2410	enhancement	ross	ross	ckan-sprint-2012-05-29	closed	wontfix	New RDF serialization required VOID vocab for related datasets	"Taken from http://lists.okfn.org/pipermail/ckan-dev/2012-May/002218.html

----

A few months ago, the RDF provided when dereferencing a dataset's URI was very useful.
It described the datasets using many of the ""right"" RDF vocabularies (dcterms, void, etc).

The RDF that is returned now no longer has these nice descriptions. 
Instead, it seems to be a mindless transcription of the CKAN record.

For example, the older descriptions gave:

<http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>
   void:subset [
       void:target <http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>, <http://thedatahub.org/dataset/geonames-semantic-web> ;
       void:triples 50 ;
       a void:Linkset
   ], [
       void:target <http://thedatahub.org/dataset/dbpedia>, <http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states> ;
       void:triples 52 ;
       a void:Linkset
   ], [
       void:target <http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>, <http://thedatahub.org/dataset/govtrack> ;
       void:triples 52 ;
       a void:Linkset
   ] ;
   void:triples 130005 ;



but now it's simply:

<http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>
   dct:relation  [
       rdf:value ""130005"" ;
       rdfs:label ""triples""
   ], [
       rdf:value ""52"" ;
       rdfs:label ""links:dbpedia""
   ], [
       rdf:value ""50"" ;
       rdfs:label ""links:geonames-semantic-web""
   ], [
       rdf:value ""52"" ;
       rdfs:label ""links:govtrack""
   ] ;


My old copies of the dataset descriptions mentioned a ""GoLD CKAN 1.0"" that was credited for the nicer modeling.

Where did that go? Can we get it back?

Thanks,
Tim Lebo




<http://semantic.ckan.net/record/91d2c0de-75a4-4bb6-b260-bc2946e1be8b>
   dc:source <http://ckan.net/> ;
   opmv:wasGeneratedBy [
       opmv:used <http://ckan.net/api/rest/package/farmers-markets-geographic-data-united-states>, <http://ckan.net/api/rest/revision/9c2545b8-ebf1-42bf-ac92-633ff5ceba3f> ;
       opmv:wasControlledBy [
           a foaf:Agent ;
           foaf:name ""GoLD CKAN 1.0""
       ] ;
       opmv:wasPerformedAt [
           a time:Instant ;
           time:inXSDDateTime ""2012-01-31T05:34:32Z""^^xsd:dateTime
       ] ;
       a opmv:Process
   ] ;
   a dcat:CatalogRecord ;
   dcat:dataset <http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states> ;

…"	1337684531000000	1338193633000000
2413	enhancement	ross	ross	ckan-sprint-2012-05-29	closed	fixed	Replace/Fork autoneg	Take revelvant parts of autoneg ( https://github.com/wwaites/autoneg.git ) so that we can remove the dependency (just for two functions) and thereby remove flup as well.	1337761125000000	1337766546000000
2414	enhancement	ross	ross	ckan-sprint-2012-06-25	closed	fixed	Remove LXML as a dependency on core CKAN	Core CKAN currently has a dependency on LXML and it would be nice if we could remove it (although this *may* be indirectly through genshi).	1337761192000000	1338471374000000
2423	enhancement	ross	seanh	ckan-sprint-2012-05-29	closed	duplicate	Get rid of CKAN's lxml dependency		1337946234000000	1338193199000000
2424	enhancement	ross	seanh	ckan-sprint-2012-05-29	closed	duplicate	Get rid of CKAN's autoneg dependency		1337946279000000	1338193269000000
2500	defect	ross	icmurray	ckan-v1.8	closed	fixed	get_action should raise an exception if the action requested doesn't exist	"== Original bug report: ==

> Hi CKAN devs,
>
> Can I request that ckan.logic.get_action raises an exception if
> the action requested doesn't exist?

> Two of us had a frustrating time tracking down an exception this
> morning. It's difficult when the exception is raised in python in a
> template because you don't get a line number. The problem was caused
> by a new logic function being added to another extension, and we
> hadn't updated the code there. So if get_action raised an proper
> exception rather than silently returning None then I think we'd have
> found the problem quicker.

> I'd write the code for this, but as I understand that the new system
> would require a ticket, branch, pull request, code review, merge. It
> seems a lot to do for such a small thing.

> Dave"	1339072580000000	1340293636000000
2514	enhancement	ross	seanh		closed	wontfix	Follow/unfollow button looks weird	It's in the tab bar (maybe not the right place for a button? but I wanted it to be next to the follower tab and count) and doesn't like very much like a button	1339578189000000	1340959787000000
2515	enhancement	ross	seanh		closed	fixed	Related tab doesn't look selected when selected	The related tab on the dataset page doesn't change its appearance when selected like the other tabs do	1339578247000000	1340967971000000
2519	enhancement	ross	ross	ckan-v1.9	closed	fixed	Dataproxy (converters) enhancement	"
Whilst the error messages returned from the dataproxy are informative, they aren't particularly human readable.  It isn't clear to the user whether they should retry, or whether the format of their data is causing the problem etc...

Would be lovely if the error messages returned contained a human readable version of the error."	1339584955000000	1345565215000000
2545	requirement	ross	ross	opendatasuite 1	closed	fixed	Need to build DataGM site based on ODS	"
== Requirements ==

We need to clone the demo ODS site for DataGM as soon as we have a mostly working demo so that the DataGM team can try it out and comment/feedback on functionality.

== Interface ==

None

== User Stories ==

None
 
== Tasks ==

[ ] Deploy demo DataGM site.


== Estimates ==


"	1340016640000000	1344337973000000
2549	enhancement	ross	ross	ckan 2.0	closed	wontfix	Merge organization logic into groups	"
== Requirements ==

Merge the organizations plugin into the core groups functionality. Most of this is templating and a small number of changes to the group controller.

Investigate the possibility of removing the Membership model and using FKs.

== Interface ==

This is likely to consist of:

 * Updating the templates in templates/group/

 * Configuring whether we want to constrain datasets to a single group (as per organizations)

 * Implementing the separate user management (add users to group).

 * Checking the publisher auth to not necessarily rely on group type = 'organization'

 * Making sure that the changes still work with publisher auth.


== User Stories ==

None

== Tasks ==


[ ] Code
 
    [ ] Configuration
  
    [ ] Model/Migration
  
    [ ] Fix templates

[ ] Documentation


== Estimates ==
"	1340017421000000	1350562037000000
2577	enhancement	ross	ross	ckan-v1.9	closed	fixed	Dataproxy raises 500 when it can't get CSV	"When given a url parameter for a file it cannot fetch, the app raises a 500 instead of something useful.

Better error reporting in the JSON would be useful"	1340207124000000	1346669735000000
2601	defect	ross	seanh		closed	fixed	500 Server Error when editing datasets with organizations plugins enabled	"Add the organizations and organizations_dataset plugins to ini file, run ckan, login, create an organization, create a dataset and put in organization, edit the dataset and set organization to none, try to edit the dataset again (you must be sysadmin to do this) and get a 500:

{{{
Error - <class 'genshi.template.eval.UndefinedError'>: [] has no member named ""get""
URL: http://127.0.0.1:5000/dataset/edit/fooset
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 162 in __call__
  app_iter = self.application(environ, sr_checker)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__
  return self.wrap_app(environ, session_start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 221 in __call__
  res = WSGIController.__call__(self, environ, start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File '/home/seanh/Projects/ckan/ckan/ckan/controllers/package.py', line 499 in edit
  extra_vars=vars)
File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 148 in render
  return cached_template(template_name, render_template, loader_class=loader_class)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
  return render_func()
File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 101 in render_template
  return literal(stream.render(method=method, encoding=None, strip_whitespace=True))
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/core.py', line 183 in render
  return encode(generator, method=method, encoding=encoding, out=out)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 57 in encode
  return _encode(''.join(list(iterator)))
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 339 in __call__
  for kind, data, pos in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 670 in __call__
  for kind, data, pos in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 771 in __call__
  for kind, data, pos in chain(stream, [(None, None, None)]):
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 586 in __call__
  for ev in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/core.py', line 288 in _ensure
  for event in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 605 in _include
  for event in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/markup.py', line 327 in _match
  for event in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 545 in _flatten
  for kind, data, pos in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/directives.py', line 169 in _generate
  attrs = _eval_expr(self.expr, ctxt, vars)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 277 in _eval_expr
  retval = expr.evaluate(ctxt)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/eval.py', line 178 in evaluate
  return eval(self.code, _globals, {'__data__': data})
File '/home/seanh/Projects/ckan/ckan/ckanext/organizations/templates/organization_package_form.html', line 110 in <Expression u""{'selected':'selected'} if organization.get('id','') == group['id'] else {}"">
  <option value=""${group['id']}"" py:attrs=""{'selected':'selected'} if organization.get('id','') == group['id'] else {}"">${group['title']}</option>
File '/home/seanh/Projects/ckan/ckan/ckan/config/environment.py', line 248 in genshi_lookup_attr
  val = cls.undefined(key, owner=obj)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/eval.py', line 410 in undefined
  raise UndefinedError(key, owner=owner)
UndefinedError: [] has no member named ""get""
}}}"	1340722754000000	1341291379000000
2605	enhancement	ross	ross	ckan-v1.8	closed	fixed	Problem with user.get_groups	"From DR at DGU

Basically get_groups() appears to cache its results, but this falls
down when subsequent calls are in a different session. We get this
when saving a dataset. get_groups() first gets called in the
controller when it is trying to work out what permissions the user
has. It later gets called, and by this time c.userobj is detached, so
get_groups() fails.

I'm working around this by refreshing c.userobj (from c.user) before
calling get_groups(), but I wonder if the get_groups caching could
detect the detached session and bypass the cache in this case?

diff at https://gist.github.com/3003117"	1340793006000000	1343127369000000
2623	enhancement	ross	seanh		closed	invalid	Organization join/authorization buttons weirdness	"Enable the organizations and organizations_dataset plugins. Start CKAN. Login and create a new organization. Logout. Login with a different account that is a sysadmin. Go to the organization you just created with the other account. There are Edit and Join buttons. Click on the Edit button, and the Join button changes to Authorization.

It should probably be an Authorization button right from the start if you are a sysadmin."	1341220902000000	1341232433000000
2624	enhancement	ross	seanh	ckan-v1.8	closed	fixed	Organizations breaks dataset tag editing	"Enable the organizations and organizations_dataset plugins, login to ckan, create a group, create a dataset and add it to the group. Edit the dataset, add some tags to it. The new tags are not saved when you submit.

Disable the organizations and organizations_dataset plugins and tag editing works again."	1341226411000000	1341831716000000
2649	enhancement	ross	ross	ckan-v1.8	closed	fixed	Description on resources not rendered as markdown	"When a resource doesn't have a description, it is rendered as markdown from the description of the dataset.

If the resource does have a description it isn't rendered as markdown ..

See http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96"	1341923135000000	1342084280000000
2650	enhancement	ross	ross	ckan-v1.8	closed	fixed	Description on resources not rendered as markdown	"When a resource doesn't have a description, it is rendered as markdown from the description of the dataset.

If the resource does have a description it isn't rendered as markdown ..

See http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96"	1341923137000000	1341943716000000
2668	defect	ross	dread	ckan-v1.9	closed	fixed	Permission for sysadmin to read a deleted publisher	"As a sysadmin I should be able to view deleted groups/publishers (publisher profile) but I can't.

In addition, there don't seem to be tests for permissions of deleted publishers."	1342171453000000	1345565100000000
2670	defect	ross	dread	ckan-v1.8	closed	fixed	model.Group.get_children_groups returns deleted groups	I believe it should only return active ones. On DGU we see deleted publishers popping up.	1342175117000000	1342440953000000
2736	enhancement	ross	ross	ckan-v1.9	closed	wontfix	Archiver fixes	"We need to check the archiver to make sure it works as we would expect after the changes in #2732:

 * Receive notification of a file that needs processing
 * Check the file's DB entry and upload the file to the configured remote service
 * Update the URL of the file (both in the file's DB entry and the resource)
 * Once we're sure the file is safely archived, we should mark the file as being archived
    so that we know it can safely be deleted at some point.

Maybe this could/should be implemented as a service rather than a celery task?"	1343121237000000	1346670097000000
2865	enhancement	ross	ross		closed	fixed	1.8 unicode error	"
URL: http://thedatahub.org/el/tag/jutatt%C3%A1sok
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:155 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
       
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
<<                                               r'\1', oldpath)
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:239 in __call__
<<          try:
                   res = WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  res = WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.template:30 in view
<<          """"""
               try:
                   return render(url)
               except TemplateNotFound:
                   if url.endswith('.html'):
>>  return render(url)
Module ckan.lib.base:153 in render
<<      try:
               return cached_template(template_name, render_template,
                                      loader_class=loader_class)
           except ckan.exceptions.CkanUrlException, e:
               raise
>>  loader_class=loader_class)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:95 in render_template
<<          template = globs['app_globals'].genshi_loader.load(template_name,
                                                                  cls=loader_class)
               stream = template.generate(**globs)
>>  cls=loader_class)
Module genshi.template.loader:223 in load
<<                      loadfunc = directory(loadfunc)
                       try:
                           filepath, filename, fileobj, uptodate = loadfunc(filename)
                       except IOError:
                           continue
>>  filepath, filename, fileobj, uptodate = loadfunc(filename)
Module genshi.template.loader:286 in _load_from_directory
<<          def _load_from_directory(filename):
                   filepath = os.path.join(path, filename)
                   fileobj = open(filepath, 'U')
                   mtime = os.path.getmtime(filepath)
                   def _uptodate():
>>  fileobj = open(filepath, 'U')
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 73: ordinal not in range(128)
CGI Variables
CKAN_CURRENT_URL	'/el/tag/jutatt%C3%A1sok'
CKAN_LANG	'en'
CKAN_LANG_IS_DEFAULT	True
CKAN_PAGE_CACHABLE	True
CONTENT_TYPE	'; charset=utf-8'
DOCUMENT_ROOT	'/etc/apache2/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_ENCODING	'gzip,deflate'
HTTP_CONNECTION	'close'
HTTP_FROM	'googlebot(at)googlebot.com'
HTTP_HOST	'thedatahub.org'
HTTP_USER_AGENT	'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
PATH_INFO	'/el/tag/jutatt\xc3\xa1sok'
PATH_TRANSLATED	'/etc/ckan/datahub/apache.wsgi/el/tag/jutatt\xc3\xa1sok'
REMOTE_ADDR	'127.0.0.1'
REMOTE_PORT	'49520'
REQUEST_METHOD	'GET'
REQUEST_URI	'/el/tag/jutatt%c3%a1sok'
SCRIPT_FILENAME	'/etc/ckan/datahub/apache.wsgi'
SERVER_ADDR	'127.0.0.1'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'thedatahub.org'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.0'
SERVER_SIGNATURE	'<address>Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.22 (Ubuntu)'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x7f5312a51650>
beaker.cache	<beaker.cache.CacheManager object at 0x7f5312a51790>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f5312a516d0>>
beaker.session	{'_accessed_time': 1345113298.778472, '_creation_time': 1345113298.778472}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.handler_script	''
mod_wsgi.input_chunked	'0'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'8080'
mod_wsgi.process_group	'datahub'
mod_wsgi.request_handler	'wsgi-script'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(3, 3)
paste.cookies	(<SimpleCookie: >, '')
paste.registry	<paste.registry.Registry object at 0x7f5309add610>
paste.throw_errors	True
pylons.action_method	<bound method TemplateController.view of <ckan.controllers.template.TemplateController object at 0x7f52b0ea2d10>>
pylons.controller	<ckan.controllers.template.TemplateController object at 0x7f52b0ea2d10>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x7f52b0ea2b90>
pylons.routes_dict	{'url': u'el/tag/jutatt\xe1sok', 'action': u'view', 'controller': u'template'}
repoze.who.logger	<logging.Logger object at 0x7f53127b2b50>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 139994769488336>, 'friendlyform': <FriendlyFormPlugin 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>}
routes.route	<routes.route.Route object at 0x7f531296ea10>
routes.url	<routes.util.URLGenerator object at 0x7f5309add810>
webob._parsed_query_vars	(GET([]), '')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52d4004990>
wsgi.version	(1, 1)
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0x7f5309add810>, {'url': u'el/tag/jutatt\xe1sok', 'action': u'view', 'controller': u'template'})
"	1345114613000000	1346670331000000
2866	enhancement	ross	ross		closed	fixed	1.8 template error	"

URL: http://thedatahub.org/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:155 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
       
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
<<                                               r'\1', oldpath)
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:239 in __call__
<<          try:
                   res = WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  res = WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.user:409 in perform_reset
<<          c.reset_key = request.params.get('key')
               if not mailer.verify_reset_link(user_obj, c.reset_key):
                   h.flash_error(_('Invalid reset key. Please try again.'))
                   abort(403)
>>  if not mailer.verify_reset_link(user_obj, c.reset_key):
Module ckan.lib.mailer:100 in verify_reset_link
<<      if not user.reset_key or len(user.reset_key) < 5:
               return False
           return key.strip() == user.reset_key
>>  return key.strip() == user.reset_key
AttributeError: 'NoneType' object has no attribute 'strip'
CGI Variables
CKAN_CURRENT_URL	'/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%253Db4c2d03fa8'
CKAN_LANG	'it'
CONTENT_TYPE	'; charset=utf-8'
DOCUMENT_ROOT	'/etc/apache2/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_ENCODING	'gzip,deflate'
HTTP_CONNECTION	'close'
HTTP_FROM	'googlebot(at)googlebot.com'
HTTP_HOST	'thedatahub.org'
HTTP_USER_AGENT	'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
PATH_INFO	'/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d'
PATH_TRANSLATED	'/etc/ckan/datahub/apache.wsgi/it/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d'
QUERY_STRING	'key%3Db4c2d03fa8'
REMOTE_ADDR	'127.0.0.1'
REMOTE_PORT	'54634'
REQUEST_METHOD	'GET'
REQUEST_URI	'/it/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8'
SCRIPT_FILENAME	'/etc/ckan/datahub/apache.wsgi'
SERVER_ADDR	'127.0.0.1'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'thedatahub.org'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.0'
SERVER_SIGNATURE	'<address>Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.22 (Ubuntu)'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x7f5312a51650>
beaker.cache	<beaker.cache.CacheManager object at 0x7f5312a51790>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f5312a516d0>>
beaker.session	{'_accessed_time': 1345066159.520708, '_creation_time': 1345066159.520708}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.handler_script	''
mod_wsgi.input_chunked	'0'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'8080'
mod_wsgi.process_group	'datahub'
mod_wsgi.request_handler	'wsgi-script'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(3, 3)
paste.cookies	(<SimpleCookie: >, '')
paste.parsed_dict_querystring	(MultiDict([('key=b4c2d03fa8', '')]), 'key%3Db4c2d03fa8')
paste.parsed_querystring	([('key=b4c2d03fa8', '')], 'key%3Db4c2d03fa8')
paste.registry	<paste.registry.Registry object at 0x7f52eba637d0>
paste.throw_errors	True
pylons.action_method	<bound method UserController.perform_reset of <ckan.controllers.user.UserController object at 0x7f52e49f7a50>>
pylons.controller	<ckan.controllers.user.UserController object at 0x7f52e49f7a50>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x7f52f5649ad0>
pylons.routes_dict	{'action': u'perform_reset', 'controller': u'user', 'id': u'3086e91c-fe09-4a98-92e1-19de67a9ac9d'}
repoze.who.logger	<logging.Logger object at 0x7f53127b2b50>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 139994769488336>, 'friendlyform': <FriendlyFormPlugin 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>}
routes.route	<routes.route.Route object at 0x7f5312961ad0>
routes.url	<routes.util.URLGenerator object at 0x7f52de7f6390>
webob._parsed_query_vars	(GET([('key=b4c2d03fa8', '')]), 'key%3Db4c2d03fa8')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52fc4f1a08>
wsgi.version	(1, 1)
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0x7f52de7f6390>, {'action': u'perform_reset', 'controller': u'user', 'id': u'3086e91c-fe09-4a98-92e1-19de67a9ac9d'})
"	1345114641000000	1346670324000000
2868	enhancement	ross	ross		closed	fixed	1.8 SOLR error	"From: <thedatahub.org@s077.okserver.org>
Date: Wed, Aug 15, 2012 at 8:34 AM
Subject: WebApp Error: <class 'ckan.lib.search.common.SearchError'>: SOLR returned an error running query: {'sort': 'metadata_modified desc', 'fq': u'groups:""welcome"" capacity:""public"" +site_id:""www.ckan.net"" +state:active', 'facet.mincount': 1, 'rows': 21, 'facet.limit': '50', 'facet': 'true', 'q': '*:*', 'start': -20, 'wt': 'json', 'fl': 'id data_dict'} Error: ""'start' parameter cannot be negative""
To: ckan-sysadmin@okfn.org, kindly@gmail.com


URL: http://thedatahub.org/feeds/group/welcome.atom?page=0
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:155 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
       
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
<<                                               r'\1', oldpath)
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:239 in __call__
<<          try:
                   res = WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  res = WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.feed:180 in group
<<          data_dict['fq'] = 'groups:""%s""' % id
       
               item_count, results = _package_search(data_dict)
       
               navigation_urls = self._navigation_urls(params,
>>  item_count, results = _package_search(data_dict)
Module ckan.controllers.feed:57 in _package_search
<<      # package_search action modifies the data_dict, so keep our copy intact.
           query = get_action('package_search')(context, data_dict.copy())
       
           return query['count'], query['results']
>>  query = get_action('package_search')(context, data_dict.copy())
Module ckan.logic.action.get:1130 in package_search
<<          query = search.query_for(model.Package)
               query.run(data_dict)
       
               for package in query.results:
>>  query.run(data_dict)
Module ckan.lib.search.query:350 in run
<<          except SolrException, e:
                   raise SearchError('SOLR returned an error running query: %r Error: %r' %
                                     (query, e.reason))
               try:
                   data = json.loads(solr_response)
>>  (query, e.reason))
SearchError: SOLR returned an error running query: {'sort': 'metadata_modified desc', 'fq': u'groups:""welcome"" capacity:""public"" +site_id:""www.ckan.net"" +state:active', 'facet.mincount': 1, 'rows': 21, 'facet.limit': '50', 'facet': 'true', 'q': '*:*', 'start': -20, 'wt': 'json', 'fl': 'id data_dict'} Error: ""'start' parameter cannot be negative""
CGI Variables
CKAN_CURRENT_URL	'/feeds/group/welcome.atom?page%3D0'
CKAN_LANG	'ja'
CONTENT_TYPE	'; charset=utf-8'
DOCUMENT_ROOT	'/etc/apache2/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_ENCODING	'gzip,deflate'
HTTP_CONNECTION	'close'
HTTP_FROM	'googlebot(at)googlebot.com'
HTTP_HOST	'thedatahub.org'
HTTP_USER_AGENT	'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
PATH_INFO	'/feeds/group/welcome.atom'
PATH_TRANSLATED	'/etc/ckan/datahub/apache.wsgi/ja/feeds/group/welcome.atom'
QUERY_STRING	'page=0'
REMOTE_ADDR	'127.0.0.1'
REMOTE_PORT	'33139'
REQUEST_METHOD	'GET'
REQUEST_URI	'/ja/feeds/group/welcome.atom?page=0'
SCRIPT_FILENAME	'/etc/ckan/datahub/apache.wsgi'
SERVER_ADDR	'127.0.0.1'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'thedatahub.org'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.0'
SERVER_SIGNATURE	'<address>Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.22 (Ubuntu)'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x7f5312a51650>
beaker.cache	<beaker.cache.CacheManager object at 0x7f5312a51790>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f5312a516d0>>
beaker.session	{'_accessed_time': 1345016040.884951, '_creation_time': 1345016040.884951}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.handler_script	''
mod_wsgi.input_chunked	'0'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'8080'
mod_wsgi.process_group	'datahub'
mod_wsgi.request_handler	'wsgi-script'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(3, 3)
paste.cookies	(<SimpleCookie: >, '')
paste.parsed_dict_querystring	(MultiDict([('page', '0')]), 'page=0')
paste.parsed_querystring	([('page', '0')], 'page=0')
paste.registry	<paste.registry.Registry object at 0x7f5302eef7d0>
paste.throw_errors	True
pylons.action_method	<bound method FeedController.group of <ckan.controllers.feed.FeedController object at 0x7f52f0341750>>
pylons.controller	<ckan.controllers.feed.FeedController object at 0x7f52f0341750>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x7f52f0341450>
pylons.routes_dict	{'action': u'group', 'controller': u'feed', 'id': u'welcome'}
repoze.who.logger	<logging.Logger object at 0x7f53127b2b50>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 139994769488336>, 'friendlyform': <FriendlyFormPlugin 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>}
routes.route	<routes.route.Route object at 0x7f5312961e90>
routes.url	<routes.util.URLGenerator object at 0x7f52f0341210>
webob._parsed_query_vars	(GET([('page', '0')]), 'page=0')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52ed4873f0>
wsgi.version	(1, 1)
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0x7f52f0341210>, {'action': u'group', 'controller': u'feed', 'id': u'welcome'})
"	1345114696000000	1345124503000000
2879	enhancement	ross	ross	ckan 2.0	closed	wontfix	Datasets (Package) should have a 'public' field	"The Package model should have a boolean field added to it describing its visibility beyond that supplied via auth.  

The related permission for viewing packages where public is set to False would be package.view (show, whatever), and *never* visible for unauthenticated users."	1345466389000000	1350562096000000
29	enhancement	rgrp	rgrp	milestone16	closed	fixed	Improve installation process	"  1. install documentation (currently source:trunk/INSTALL is empty)
  2. improve ckan-admin as necessary
  3. create default data (such as licenses)"	1152553638000000	1183648166000000
32	enhancement	rgrp	rgrp	v0.3	closed	fixed	View a tag	"== As a ==

Visitor

== I want to ==

View a tag's page which should list all packages associated with that tag (or link to a page containing that list).

== Notes ==

  * The tags should have a nice primary url such as /tag/<tag-name> (could also use the plural i.e. /tag/<tag-name>)"	1157375968000000	1185472746000000
33	enhancement	rgrp	rgrp	v0.5	closed	fixed	List and search tags	"== As a ==

Visitor

== I want to ==

View a list of the tags on the system and search for a tag by name

== Notes ==

  * As for a search of packages should search for tags that contains a given search string (or could use a glob approach allowing wildcards)."	1157375983000000	1199788094000000
35	defect	rgrp	rgrp	v0.4	closed	fixed	IP Address for non-logged in users is always 127.0.0.1 when deployed behind a proxy (e.g. on ckan.net)	"When IP address is used to identify non-logged in users (for example on create and editing of packages) it is always 127.0.0.1 when using a reverse proxy (standard deployment configuration on a production machine and used e.g. on http://www.ckan.net).

This could be corrected by using HTTP_X_FORWARDED_FOR when available."	1177006633000000	1185471537000000
36	enhancement	rgrp	rgrp	v0.4	closed	fixed	Edit package page should support previews	When editing a package one would like to be able to preview one's changes before saving.	1177007112000000	1185470035000000
37	enhancement	rgrp	rgrp	v0.5	closed	fixed	Purge a Revision (i.e. purge all changes associated with that revision)	"Spam entries have started to occur on ckan.net in the last few months. It would be useful to be able to purge these revisions -- that is permanently delete all changes to domain objects associated with those revision (one might also permanently delete that revision -- or alternatively simply mark it as purged).

In addition to providing this facility from the shell it would also be useful to be able to do this from the web interface (with associated restrictions on usage via an authorization controller of some sort)."	1192650660000000	1199786536000000
38	enhancement	rgrp	rgrp	v0.5	closed	fixed	Make data available in machine-usable form	"Currently all data is only available via the web interface which makes it difficult to obtain in bulk. Data should be provided in other more convenient, machine usable forms:

  * DB Dump
  * JSON for each package page (maybe Atom as well)"	1194521614000000	1200903004000000
40	defect	rgrp	rgrp	v0.5	closed	fixed	Reserved html characters (such as &) in urls mean package does not render for read view	"A url such as: http://someurl.com/xyz?x=1&VERSION=1.1&Service=WFS when set as url or download_url breaks the rendering of the package with an error like:

There was an error rendering the package: not well-formed (invalid token): line 1, column 181

Have checked that removing the & stuff makes the error go away so this looks like an issue with escaping urls when displaying them ..."	1195565228000000	1200993319000000
41	task	rgrp	rgrp	v0.5	closed	fixed	Upgrade to Latest Version of Pylons (0.9.6.*) and Deploy on Production Machine	Should do this sooner rather than later to avoid 'clear water' opening up and need to deploy then asap onto production machine to avoid 'clear water' between trunk and production (which then prevents pushing from trunk to production).	1199786854000000	1199787967000000
42	enhancement	rgrp	rgrp	v0.6	closed	fixed	In WUI where List of Items to Display is Large Results should be Paged	"= As A =

Visitor

= I Want To =

View lists of items but not have too many on a page (as this leads to slow page loads and difficulty navigating the list). Where there are more items in the results of my action that can be fit on a single page the results should be 'paged' in some manner.

= Details =

Main place this issue arises:

  * List of tags
  * List of packages
  * Repository history (/revision)

Should probably have no more than 50 (or so) items on a page. Obvious solution is for actions that return lists to have some kind of pageno parameter and then use this in controller to select what to display."	1199787162000000	1204133257000000
43	enhancement	rgrp	rgrp	v0.10	closed	fixed	Generic Attributes for Packages	"= As A =

User

= I Want To =

Add arbitrary named attributes to packages (an attribute being a name, type, value triple).

= Details =

  * We will do this using a dedicated (versioned) table associated to Package
  * Do we allow multiple attributes of the same name?
    * For the present: No (since we will key by attribute name)
    * Could allow for single attribute but with multiple values using json list ...
  * What types do we allow or do we just rely on JSON to take care of this?

= Questions (Original) =

  * How complex is this to implement?
  * What would an arbitrary user be able to edit? Possibilities:
    1. 'create new attribute' and setting the value (so name and type would be chosen from predefined list).
    2. 'create', setting of name and value (but not type -- type already set in predefined list)
  * Could just use (machine) tags -- though this could be seen as a bit of a hack.
  * Would solve having to create special file/url attributes (though I think that perhaps file stuff is important enough to merit first class support in the domain model -- though, that said, since one won't want to have a file limit adding unlimited file support is very similar to unlimited attributes of arbitrary type)."	1199788109000000	1253709802000000
45	enhancement	rgrp	johnbywater	v0.6	closed	fixed	Immunity to SQL injection attacks		1201110894000000	1204133342000000
46	enhancement	rgrp	johnbywater	v0.9	closed	fixed	Return error documents that look and feel like normal CKAN pages		1201111018000000	1265891789000000
47	enhancement	rgrp	johnbywater	v0.6	closed	fixed	Return OpenID signin pages that look and feel like normal pages		1201183920000000	1215543616000000
48	enhancement	rgrp	johnbywater	ckan-v1.4-sprint-1	closed	fixed	Create favicon	Because CKAN's worth it!	1201202789000000	1297077492000000
49	enhancement	rgrp	rgrp		closed	invalid	Filter Spam in Changes to CKAN Data	"= As A =

sysadmin

= I Want To =

Have revisions to the CKAN data filtered in order to reduce the spam in the system.

= Details =

In the long run this is a quite a generic problem common across several OKF systems and probably can become a general component in the okfmisc repo. For time being focus on a well-factored CKAN-specific solution.

Suggest we follow path of trac: http://trac.edgewall.org/wiki/SpamFilter

Could have a general engine that aggregates spam scores from many different 'plugins' and then marks spam appropriately (actions should be configurable depending on spam level from 'purge' to 'delete' (mark revision as inactive) to 'flag' to 'do nothing').

Main initial plugins would be:

  * regex filter (this would seem very useful here, e.g. do not allow urls in commit messages ...)
    * could augment using the badcontent list approach (can find list on e.g. moinmoin)
  * spambayes and/or akismet"	1204134691000000	1257244973000000
50	task	rgrp	rgrp		closed	fixed	Investigate apt-get in more detail	"Research apt-get especially wajig and python wrappers for apt-get. Post results on the wiki.
"	1215456052000000	1267648356000000
51	enhancement	rgrp	rgrp	v0.7	closed	fixed	Upgrade CKAN to use sqlalchemy and vdm v0.2	"Move CKAN codebase to use sqlalchemy (rather than sqlobject) along with vdm v0.2.

NB: the upgrade itself should not be too hard, the work will be in providing a satisfactory migration script to convert existing data on ckan.net service."	1220900438000000	1223908230000000
52	enhancement	rgrp	rgrp	v0.7	closed	invalid	Paginated results sets should be customizable across domain objects	"  1. Tag results could/should show number of associated packages
  2. Package results should show title (and be in list)
  3. Pagination should be able to be applied to other result sets than simply register listing (e.g. search results ...)
  4. Revisions should be listed in a table with full details"	1223549648000000	1223908425000000
53	enhancement	rgrp	rgrp	v0.8	closed	fixed	View Information about Package history (i.e. package revisions)	"= As A =

Visitor/User

= I want to =

View information about a package history, i.e. what revisions it has.

= Details =

  * History info should be available at package/history/pkg-name
  * Most basic info is just info from revision (when it occurred, author etc)
  * More sophisticated would be to have diff support to see differences between revisions"	1223552139000000	1239133021000000
55	enhancement	rgrp	rgrp	v0.7	closed	fixed	Code to migrate data from v0.6 to v0.7 using dump and load	"Associated to ticket:51 (upgrade CKAN to new vdm) and ticket:54 (dump/load) need to convert v0.6 data for v0.7.

Obvious way to do this is via alteration to data load method."	1223908240000000	1223909891000000
56	enhancement	rgrp	rgrp	v0.8	closed	fixed	Show icons indicating package openness	"== As A ==

User or Visitor

== I Want To ==

See simple icons (e.g. tick/cross) next to packages (in lists or on main page) indicating openness status of package.

== So That ==

It is easy to see the openness status of a package.

== Details ==
  
  * Suggested by Liz Turner
  * Openness = open license at present (no easy way to determine whether accessible)
  * Have separate tick for downloadability (i.e. existence of download url)
  * Could do more stuff in future (e.g. money where tag price- exists, etc etc)"	1230211123000000	1239018857000000
57	enhancement	rgrp	rgrp	v0.8	closed	fixed	Basic datapkg integration with CKAN	"Using datapkg a user can:

  * list packages on CKAN
  * get info about package on CKAN
  * register and update a package on CKAN
"	1238750826000000	1239018717000000
58	enhancement	rgrp	rgrp	v0.9	closed	fixed	Make bookmarklet to enable easy package registration on CKAN	"Make bookmarklet to enable easy package registration on CKAN

== Details ==

  1. This requires redoing web interface to allow 1-step package creation/editing (as opposed to name then everything else approach at the moment)"	1239018517000000	1239974365000000
60	enhancement	rgrp	rgrp	v0.9	closed	fixed	Front page usability improvements	"1. List in sidebar 3 latest created/updated packages.

2. Put package search box on front page (might also want to move some of front page content into an about page).

3. Similarly might want package creation box on front page."	1239094266000000	1246434921000000
61	enhancement	rgrp	rgrp	v0.10	closed	fixed	When dumping data to json do not dump private information like API keys	Cost: 1h	1239123529000000	1265890790000000
63	defect	rgrp	rgrp	v0.9	closed	fixed	Purge is broken after upgrade of vdm	"Since we upgraded to new vdm (v0.4) which has built in purge support existing purge support looks to be broken. Should be reasonably easy to fix this.

Cost: 1h"	1245148623000000	1245656000000000
64	enhancement	rgrp	rgrp	v0.10	closed	fixed	Switch to repoze.who for authentication	"Switch to repoze.who + openid plugin for authentication from authkit.

Already did this in microfacts so should be fairly easy."	1245168853000000	1246127063000000
65	enhancement	rgrp	rgrp	v0.9	closed	fixed	Add version attribute to package	"It is essential packages have a version attribute to assist in tracking releases etc (and to allow compatibility with other packaging systems e.g. distutils which is being used by datapkg).

cost: 2h"	1245262864000000	1246434569000000
66	enhancement	rgrp	rgrp	v0.10	closed	fixed	Improve user account UI	"1. Link at top should say ""Your Account"" when you are logged in
  * when not logged in should say: ""Log in via openid""
  * Open Id info on /account/ should move to /account/login/ or be deleted
  * Going to /account/ when not logged in should redirect to /account/login/

2. When logged in the basic account index page should not give generic information but should reflect fact you are logged in by:

  * Saying something like: ""You are logged in as: ...""
  * Showing your apikey or a link to apikey
  * Giving you a logout link

We can think of a bunch of other stuff that could go on their (recent edits, packages you own etc) but these will be separate tickets."	1245263685000000	1250785405000000
67	enhancement	rgrp	rgrp	v0.10	closed	fixed	List all of a user's recent edits on their home page	"Additional feature related to ticket:66.

cost: 2h"	1245263731000000	1250785122000000
68	enhancement	rgrp	rgrp	v0.9	closed	fixed	Show affected packages in revision list view	"When listing revisions (at /revision/ or in atom feed) show affected packages. Minor but v. useful UI improvement.

Cost: 1h"	1245656153000000	1245697554000000
69	enhancement	rgrp	rgrp	v0.9	closed	fixed	Change to text-only license field and use external license repo	"Switch from license domain object to a simple license field and use license list from new centralised license repo:

<http://knowledgeforge.net/okfn/licenses/>

  * This will '''require''' a migration

Cost: 4h (plus migration ...)"	1245687449000000	1246437494000000
70	enhancement	rgrp	rgrp	v0.9	closed	fixed	Convert from py.test to nosetests	Cost: 1h	1246434753000000	1247827053000000
71	enhancement	rgrp	rgrp	v0.10	closed	fixed	Upgrade to Pylons 0.9.7	Cost: 2h	1246435041000000	1250181211000000
72	enhancement	rgrp	rgrp	v0.10	closed	fixed	Integrate new logo	"Integrate new logo from http://wiki.okfn.org/ckan/logo into site:

  * In title

Also can do favicon (separate ticket:48)

Cost: 1h"	1246441021000000	1251451954000000
73	enhancement	rgrp	rgrp		closed	invalid	Put enquiry template in db so it can be edited by admins	"Currently stored in the python controller file. This is not the right place and putting it in db will allow editing by admins.

May also want to do this for the footer (also in the controller file).

Details:

  * Will need to create a new domain object/db table. Suggest called miscellaneous and consist of key value pairs utilizing JSON type for values.

Cost: 2h."	1247705786000000	1266510385000000
78	enhancement	rgrp	rgrp	ckan-sprint-2011-10-28	closed	fixed	Extend login cookie lifetime	"When you login you get given a cookie but it is very short (life of your browser session I think). 

Work out how to extend to something reasonable (30 days or perhaps forever with logout unsetting cookie).

 1. A quick investigation to see whether this is possible
 2. If possible do it

Cost: ?"	1247829310000000	1314877169000000
81	enhancement	rgrp	rgrp	vdm-0.7	closed	fixed	Support for sqlalchemy 0.5		1248285433000000	1268487327000000
82	enhancement	rgrp	rgrp		closed	fixed	Support diffing of versioned objects	"Should have function/facility to get a diff of a version object between 2 revisions. Function should return a dictionary of fields with diffs.

Details:
  * Do we deal with m2m relationships (and m2one on the many side)?
  * What is diff?
    * For text fields output of python diff command
    * For non-text field guess just simple +/- for what was there before and now

Cost: 4h (don't think this is a huge request)"	1248289499000000	1256565441000000
83	enhancement	rgrp	rgrp	vdm-0.6	closed	fixed	"Allow ""ignored"" fields on versioned objects"	"Allow 'ignored' fields on versioned objects (i.e. attributes which are not 'versioned'). This is not hard to do as we already have most of the necessary mechanisms set up in the Revisioner object.

Cost: 2h
"	1248339384000000	1267648301000000
85	enhancement	rgrp	rgrp		closed	fixed	Convert state from an object to an enumeration	"No real benefit of having State object as opposed to a simple text field with enumerated values and this second option is both simpler and more flexible.

Main hassle here is that it requires migration in all projects that use vdm.

If we're going to do this we should do it sooner rather than later.

Cost: 2h (+ 1h for a conversion method)"	1248339662000000	1263206391000000
86	enhancement	rgrp	rgrp		closed	fixed	Support for stateful dict-like collections	"Already support stateful list-like collections and should extend this to dict-like collections.

(This is prerequisite for implementing versioned ""extra"" (key/value) attributes on packages in CKAN and elsewhere)."	1248430798000000	1249050202000000
87	enhancement	rgrp	rgrp		closed	duplicate	Multiple download links	Multiple download links, including links to mirrors and multiple formats/versions 	1248693302000000	1258470719000000
88	enhancement	rgrp	rgrp	v0.11	closed	fixed	Download (link) metadata	"Support for download metadata (file size, file formats, version information).

See also: ticket:87 (multiple download links)"	1248693385000000	1265891491000000
92	enhancement	rgrp	rgrp		closed	wontfix	Add RDFa to package pages		1249049927000000	1297344859000000
93	enhancement	rgrp	rgrp	v0.10	closed	fixed	Access control for packages	"Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: AccessControl.

Cost: 8d

"	1249055049000000	1253613274000000
95	enhancement	rgrp	rgrp		closed	fixed	Add manifest support	"Support for listing files (manifest) contained within a package.

Traditional manifests just list the files. Suggest in addition we have support for optional metadata in form of key, value pairs."	1249981842000000	1251454716000000
96	defect	rgrp	rgrp		closed	invalid	info command must give up to date information for python-type distributions	"(2008-09-10) at present when setup.py is edited info command may not give up to date information if pkg-info (in egg.info) is not rebuilt.

Suggest: info command needs to rebuilds pkg-info in python-type distributions.
"	1249982410000000	1311176063000000
97	enhancement	rgrp	rgrp		closed	duplicate	Do not create a distribution on a path is something already exists there	"(2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option).

Cost: 1h"	1249983557000000	1318181317000000
98	enhancement	rgrp	rgrp	datapkg-0.7	closed	fixed	Replace use of pastescript templates in PythonDistribution with of something simpler	"Replace use of pastescript templates in PythonDistribution (write) with of something simpler (allowing us to remove dependency on pastescript).
"	1249983810000000	1297210774000000
99	enhancement	rgrp	rgrp		closed	wontfix	Test DbIndex with sqla 0.5	"Test DbIndex with sqla 0.5 (and get it working if it does not work).

Cost: ??

Priority: low because DbIndex is not currently that important to us (also wonder whether we should try using CKAN code here to stop us reinventing the wheel).
"	1249985537000000	1297081088000000
100	enhancement	rgrp	rgrp		closed	fixed	Convert existing data on disk to a datapkg distribution	"(2008-10-29) convert existing directory on disk to a datapkg distribution (PythonDistribution by default). This would be presented in the CLI as a convert command.

Details:
  * Add a metadata file (setup.py), perhaps prompting for input
  * List all existing data in that directory

Cost: 4h
"	1249986628000000	1318181227000000
101	enhancement	rgrp	rgrp		closed	fixed	(Improved) Download support	"  1. Given a url download from that url to disk
  2. Given a package use download url to download to disk

Once on disk should uncompress (if necessary).

== Details ==

  * Should download either to tmp directory (defaulted in config) or a specified path on disk
  * In download would like to support as many types of target urls as possible:
    1. Single files (usually compressed filesets but could be individual files)
    2. Index pages listing files to download
    3. Revision control repositories
    4. ...

  This will obviously be a significant amount of work. Suggest:
    1. Start off just supporting the first option.
    2. Have something like a plugin system to handle new target types
    3. Reuse existing work (e.g. easy_install and pip handle standard vcs such as svn, hg, git etc)

Cost: 1d
"	1249988866000000	1267648607000000
102	enhancement	rgrp	rgrp		closed	fixed	Switch to using UUIDs for revision ids	"This is better for long term sustainability, especially if we want to ""push and pull"" from unrelated repositories in the future.

Details:
  * May wish to retain a number field updated in the usual incremental way (what's the cost/benefit here?)"	1250068175000000	1260285104000000
104	enhancement	rgrp	rgrp		closed	fixed	View a package at a given version	"As a user i want to see a package at ""version"" X (NB: not revision X).

  * When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version)
  * Implementation:
    * Find revision for this version (search revision history for when version field was last 0.7)
    * Show pacakge at that revision (as in ticket:103)
  * On history page also shows versions in list of revisions associated with the package

"	1251292672000000	1311181303000000
107	enhancement	rgrp	rgrp		closed	invalid	Provide information about uploading material	"Often people who are registering resources want to make the associated material available. We should:

  * Provide a dedicated upload page giving instructions as to upload process
    1. Explain we don't store the data itself on CKAN. Suggest uploading to a store somewhere and then linking using download_url (that is its purpose)
    2. Upload to
      * Talis CC for RDF
      * archive.org
      * grid.okfn.org
  * link to this page from next to download_url item on new package page

In long run (now ticket:186) we may automate this by providing a ckan upload facility which caches the data and then reuploads it to relevant service (disadvantage is ""we"" are the owner of the data on that service ...)"	1251466274000000	1296341644000000
109	enhancement	rgrp	dread		closed	fixed	Tag search in the REST API	"Similar to ticket:108 but for tags:

/api/search/tag?q

For tags search is extremely simple since you can only search by name.

== Possible Extras ==

  * Allow ""exotic"" ordering of results e.g. ordering by number of packages with that tag.
  * Return this number with tag list."	1251976297000000	1291829457000000
111	enhancement	rgrp	dread	v0.10	closed	fixed	Create user object	"Object properties:
  * id (uuid)
  * apikey (uuid) -- migrated from the apikey table
  * name (username = openid)

/account is being renamed to /user in the wui."	1252315994000000	1252331626000000
112	enhancement	rgrp	dread	v0.11	closed	wontfix	Test db schema created via migration scripts	"'paster db upgrade' should result in the same database as 'paster db create' on a clean system. In particular, all tests should pass. 

Note: equivalent of paster db upgrade isn't being run from tests at the moment (we haven't been able to get this to work)."	1252316495000000	1265294069000000
124	enhancement	rgrp	rgrp	v1.0	closed	fixed	Display Generic Package Attributes in WUI	Split out from ticket:43	1253709702000000	1254735558000000
129	enhancement	rgrp	dread	ckan-backlog	closed	invalid	Secure db access by channelling query generation through authz module	"Controllers and templates should not access db objects directly - they should do all access via authz module giving username. They are handed by a query that has already been filtered by the packages they are authorized to read. 

(Additional idea to be discussed: When they request a package object, they are handed an copy of the db object - disconnected from the database - so it the db object can't be changed.)

A couple of tests can be reenabled when this is done:
ckan.tests.functional.test_authz.TestUsage.test_admin_list_deleted ckan.tests.functional.test_authz.TestUsage.test_search_deleted"	1253886136000000	1267719162000000
132	defect	rgrp	dread		closed	fixed	Security hole - read package/group list (REST)	"Using REST interface you can list packages and groups without authorization being checked.

Can be fixed using more advanced query to check authz."	1254389493000000	1273254514000000
133	defect	rgrp	dread		closed	fixed	Security hole - search package/group (WUI & REST)	"Using WUI or REST interface you can search packages and groups without authorization being checked. 

On the REST interface you can also read all the attributes of the packages using the 'all-fields' option.

Can be fixed using more advanced query to check authz.
"	1254390168000000	1273253977000000
134	defect	rgrp	dread	v0.11	closed	fixed	admin interface is only available to sysadmins		1254735314000000	1255430998000000
135	enhancement	rgrp	dread	v0.11	closed	fixed	sysadmins defined in the db		1254735368000000	1265293940000000
136	enhancement	rgrp	dread	v0.11	closed	fixed	User has publicly viewable page	"A user's 'home page' is at:
user/<user.id>
e.g. user/28394723982-03849472

Step 1:
* Current user home page with recent edits

Step 2:
Readonly
* Number of edits
* Number of Packages they are admin of

How do we do a nicer URL for the home page - can we use their openid login?

Follow on tickets: ticket:138 and ticket:142"	1254741650000000	1255169466000000
137	enhancement	rgrp	dread		closed	duplicate	User has editable home page	" * Generic text box for markdown about the user 'About'

Model's user table reflects these:
 * 'about' attribute
"	1254741703000000	1254741830000000
138	enhancement	rgrp	dread	v1.0	closed	fixed	User has editable home page	" * Generic text box for markdown about the user 'About'

Model's user table reflects these:
 * 'about' attribute
 * 'created' attribute is a default=datetime.now() - so when user was created

Need edit page for this.

Only user should be able to edit it.

Follows on from ticket:136 and ticket:142

Cost: 2h"	1254741789000000	1258466074000000
139	defect	rgrp	rgrp	v0.11	closed	worksforme	Ensure you cannot add same tag twice to a package	"Seems some package have ended up with same tag twice: http://www.ckan.net/package/fred2dot0

Check whether this is still possible and if it is fix it ...

Cost: 2h"	1254768552000000	1255188974000000
144	enhancement	rgrp	dread	v0.11	closed	duplicate	Most popular packages listed on homepage	"Based on number of views.

Related to ticket:143."	1255010391000000	1265284457000000
145	enhancement	rgrp	dread	v0.11	closed	fixed	Group shows its administrators	When viewing a group you see a list of the users who have admin rights to it. (Akin to owners / moderators)	1255010403000000	1265294041000000
154	enhancement	rgrp	dread	v0.11	closed	wontfix	Package form guidance	"Give guidance on what to put in 'author' and 'maintainer' fields (e.g.
name? URL?)

Suggestion from Jonathan Gray"	1255621572000000	1257535066000000
163	enhancement	rgrp	dread	v0.11	closed	fixed	Regularly put CKAN RDF on Talis CC	"Related to:
ticket:90 Link to RDF version of CKAN data on Talis Connected Commons
ticket:164 Creating RDF data from CKAN

Regularly re-upload the CKAN RDF data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)
"	1256052587000000	1265891374000000
165	enhancement	rgrp	dread		closed	invalid	Use 'tag:' instead of 'tags:' for wui package search	"Because people might try and do ""tags:gov sweden"" when ""tag:gov sweden"" makes more sense (sweden is searched for in all fields in these cases).

But in REST i/f search allow ""tag""=""gov"" OR ""tags""=""gov sweden"" in parameters."	1256200526000000	1311181391000000
168	enhancement	rgrp	dread		closed	duplicate	Show admins for a group in group view		1256291481000000	1257414795000000
170	enhancement	rgrp	dread	v0.11	closed	fixed	Paginate search results	"e.g.

http://www.ckan.net/package/search?q=science&search=Search+Packages+%C2%BB

Results are capped at 20, but can't see past first page."	1256313584000000	1265891157000000
171	enhancement	rgrp	rgrp	datapkg-0.7	closed	fixed	Create consolidated config for CLI	Create consolidated set of config by merging config from .datapkgrc and command line options and use that instead of mixture of command line options and other stuff ... (likely to be less buggy!)	1256488011000000	1297210925000000
172	enhancement	rgrp	rgrp	v0.11	closed	fixed	Build ckan documentation using sphinx and upload	"Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible.

NB: improving the documentation is another matter (as is integrating e.g. existing api docs).

Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net):

  * http://knowledgeforge.net/ckan/ckan/doc/
  * (Other options: http://www.ckan.net/doc/ http://doc.ckan.net)
"	1256489019000000	1257532331000000
173	enhancement	rgrp	dread	v0.11	closed	fixed	Diffing WUI	"Use vdm diff functionality (see ticket:82) to implement diffing of packages between versions in WUI.

In package history you should be able to select to revisions and show the diff.

Estimated cost: 12h"	1256565412000000	1265293968000000
174	defect	rgrp	dread	v0.11	closed	fixed	Packages search broken for 'only openly licensed'	"the ""only openly licensed packages"" tick box seems
to return blank page every time - as do queries for only openly
licensed and only downloadable packages. The ""only downloadable
packages"" tick box by itself seems to work fine. E.g. try searching
for 'science'.

Raised by Jonathan Gray"	1256650867000000	1265891093000000
175	enhancement	rgrp	dread	v0.11	closed	fixed	Creation date	"'created' attribute on User, Group and Rating

datetime object: default=datetime.datetime.now()

See example on revision

Requires migrate script"	1256754070000000	1265291037000000
177	enhancement	rgrp	dread		closed	invalid	Service documentation	"This is docs for users of a CKAN service (e.g. ckan.net), as opposed to someone installing / administering the software. (The latter has docs on knowledgeforge.

Editability - wiki?"	1257243285000000	1273050236000000
178	enhancement	rgrp	dread	v0.11	closed	fixed	Excel package importer	"/tool
New 'tools' section of the WUI with 'Excel package import' page.

/tool/import
Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview
When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:
 * license - name of license
 * tags - space separated
 * groups - space separated
 * anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups."	1257244034000000	1265305558000000
180	enhancement	rgrp	jwyg	v0.11	closed	duplicate	Tag cloud as way to view CKAN tags	Create big tag cloud with all CKAN tags - perhaps weighting with size and colour...	1257534254000000	1265284374000000
181	enhancement	rgrp	jwyg		closed	invalid	New 'stats' box on right hand sidebar	"With up to date statistics such as:

  * total packages
  * total tags
  * total users
  * most active users
  * last update
  * etc.

Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See ticket:184"	1257534385000000	1296339510000000
183	enhancement	rgrp	rgrp		closed	worksforme	Browse packages by rating	At moment order packages by title.	1257534606000000	1290604779000000
185	enhancement	rgrp	dread		closed	invalid	Package form field hiding	"Make parts of the package form hidden by default, to make it look simpler. Users are invited to click a button to reveal the less important fields.

Fields hidden by default:
version, author, author_email, maintainer, maintainer_email"	1257763437000000	1291830039000000
186	enhancement	rgrp	rgrp		closed	duplicate	Automated upload to archive.org s3	"(Follows on from ticket:107). We want to provide facility for users to automatically upload material.

  * Create an upload page
  * Store the material on archive.org using s3 - http://www.archive.org/help/abouts3.txt
  * Main issue is how we cache large datasets ..."	1257803430000000	1296341182000000
187	enhancement	rgrp	rgrp	v0.11	closed	fixed	Full-text search	"Standard search should search notes field in addition to name, title and tags (discussed in ticket:108 but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results.

If we do this we need:
  1. To weight across fields in a sensible way
  2. We can also use proper text search on title or ...

Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: http://www.postgresql.org/docs/8.3/static/textsearch.html

Using this with sqlalchemy: http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/

Issues with fulltext search:

 * tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking.
 * if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better.
 * can we split the name on dash or underscore before being indexed?
 * natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'.
 * do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres?
 * we want to weight name and title higher than other fields - achievable with custom trigger."	1257869950000000	1265892403000000
188	enhancement	rgrp	rgrp	v0.11	closed	duplicate	Improve package listing views	"Propose change to tabular-like format showing these attributes (perhaps should be configurable?)

  * Openness status
  * Title (not sure name is needed)
  * Tags

Cost: 4h"	1257870031000000	1265294090000000
192	enhancement	rgrp	dread		closed	fixed	Time-related package field	"Cost - 2 days

The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09).

Example part of package:

'''Date:'''	5/10/09
[[br]] or [[br]]
'''Date:'''	1/1/09 – 1/4/09"	1258388163000000	1291733895000000
193	enhancement	rgrp	dread		closed	wontfix	Searching by time-related field	"Cost - 2 days

Search interface has new options to filter and sort the results by the time-related field of the package. Search options are included in both Web UI and Search API.

The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package. Need to decide for a time-related field value that is date range, what date is used for the search.

Example search parameters:

|| reldate-range=5/4/09- || Exclude packages related to earlier than 5/4/09 ||
|| reldate-range=5/4/09-5/12/09 || Exclude packages related to date outside of 5/4/09-5/12/09 ||
|| order_by=reldate || Sort by date package is related to. Defaults to newest first. ||
|| order_by=reldate-newest || Sort by date package is related to, newest first. ||
|| order_by=reldate-oldest || Sort by date package is related to, oldest first. ||

Related to ticket:192"	1258388169000000	1340626463000000
194	defect	rgrp	dread	v0.11	closed	fixed	Check star ratings aren't influenced by search engine crawlers	rel=nofollow or robots.txt ?	1258471512000000	1265284389000000
195	enhancement	rgrp	rgrp	v0.11	closed	fixed	Association listings should be alphabetically sorted (e.g. a group's packages)	"Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name.

Cost: 2h"	1258475448000000	1258585500000000
198	enhancement	rgrp	dread		closed	fixed	Change package and tag ids to uuids	"See how we did it already for other things.

Note: on ckan.net older PackageRevision.id might not be identical to Package.id but this may need sorting at this point."	1258980613000000	1266837606000000
199	defect	rgrp	casbon		closed	fixed	Fix failing tests and provide documentation on how to run tests in datapkg	"As of 80:099bdc5b07a6 datapkg has 4 test fails and 1 test error.

There is no documentation on how to run them, so I am using 'python setup.py test'.

{{{
datapkg.tests.test_cli.TestCLI.test_ckan ... FAIL
datapkg.tests.test_cli.TestCLI.test_walkthrough ... FAIL
datapkg.tests.test_index.TestDbIndex.test_get ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/index.py:97: SADeprecationWarning: Use session.add()
  self.session.update(pkg)
FAIL
datapkg.tests.test_index.TestDbIndex.test_get_when_loaded_as_new_and_init_not_called ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/tests/test_index.py:57: SADeprecationWarning: Use session.expunge_all()
  self.index.session.clear()
FAIL
}}}
"	1259087361000000	1267649255000000
200	defect	rgrp	rgrp		closed	invalid	PythonDistribution write method should write all available metadata	Current write method just creates a bare bones package and does not write any of the available metadata other than name.	1259229583000000	1311176118000000
201	enhancement	rgrp	dread	v1.0	closed	fixed	Stress test ckan	"Check:

* high numbers of simultaneous users
* incomplete transactions
* etc."	1259844836000000	1265284419000000
205	enhancement	rgrp	dread		closed	wontfix	Custom package importer	"Importer function is exposed to logged in users.

Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'. 
 * Basic is as it is currently
 * Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords.

All fields should be in step with the government custom input form.

For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text.

Estimate: "	1260197819000000	1297068450000000
209	enhancement	rgrp	rgrp		closed	fixed	Validate email address when creating an enquiry	p:wui 2009-09-09 validate email address when creating an enquiry	1260288671000000	1260288728000000
212	enhancement	rgrp	rgrp	v0.11	closed	fixed	"Switch to using m.okfn.org ""kforge"" media (css and behaviour)"	"We are already using standard kforge css though copied locally and with a few modifications.

We should switch to using common copy on m.okfn.org. Some work will be required to factor out any changes we have made.

Cost: 2h"	1260703728000000	1260703969000000
215	enhancement	rgrp	dread	v1.0	closed	fixed	UI Review - Home page	" * 'Login via OpenID' -> 'Login with OpenID'?
 * Elements:
    * Search
    * about CKAN
    * 3 column Packages view (see ticket:219)
    * Small tag cloud (RHS)

Also, what about ticket:143 ?"	1260879732000000	1266513523000000
225	enhancement	rgrp	dread		closed	invalid	UI Review - swap URL order	"Not so keen on URLS such as /package/edit/mypackagename. These seem to be task-oriented rather than resource oriented (which would be /package/mypackagename/edit), and unstable: there is a common root for /package/edit/mypackage and /package/history/mypackage which is not reflected in the URL structure.

"	1260880487000000	1311178276000000
228	enhancement	rgrp	rgrp		closed	duplicate	Deal with duplicate packages	This needs to be thought out ...	1262085763000000	1290596875000000
232	enhancement	rgrp	rgrp		closed	invalid	Add strapline to CKAN logo	"Strapline (?): Sharing, discovery and reuse of data and content

Need to consult on this.

Cost: 1h"	1262975291000000	1297246297000000
233	enhancement	rgrp	rgrp	v1.1	closed	fixed	Allow simple site-specific customization/overriding of templates	"Options:

1. Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works)

1. Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist).

Can do this using:

<xi:include href=""base.html""><xi:fallback /></xi:include>"	1263000396000000	1273080352000000
241	defect	rgrp	dread	v1.0	closed	fixed	License doesn't preview correctly	4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package.	1265625546000000	1270569769000000
243	defect	rgrp	dread		closed	worksforme	Repetition of number of packages in a group	"http://ckan.net/group/publicdomain has:

""There is 16 package in this group.
There are 16 packages in this group."""	1265750250000000	1266837796000000
244	defect	rgrp	dread		closed	fixed	Several links preceded by link URL	"e.g. Text on ckan.net home page:

""All material available under an /licenseopen license""

Also seen on the package edit page:
""/user/loginClick here to sign in""

I believe this is due to the i18n additions from Benoit. Maybe needs new Genshi. Maybe best to work round in the meantime."	1265750419000000	1271248968000000
245	enhancement	rgrp	rgrp		closed	duplicate	Support for composite primary keys	"Problem here is that foreign key then becomes ""complicated"" (composite).

  * Could also deprecate continuity_id field in favour of the basic foreign key on ie"	1265882630000000	1297066620000000
246	enhancement	rgrp	rgrp		closed	duplicate	Support for primary key not named id	At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys)	1265882862000000	1297066757000000
249	enhancement	rgrp	dread		closed	invalid	Regex search	"Search of package name and title (and other fields) using regular expressions.

Current example use-case: Wanting to specify packages with title beginning with 'B'.

Issues:

 * Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields?

Implementation:

* Postgres reg ex searching detailed here:
http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE"	1265994509000000	1311182450000000
258	enhancement	rgrp	dread	v1.0	closed	fixed	Add uuids into package in REST	Add uuids into package in REST interface.	1266954722000000	1273596163000000
259	defect	rgrp	dread	v1.0	closed	fixed	isitopen enquiry broken	"Pressing the 'make an enquiry' link on the package read page gives 404. (Also links should also be changed from isitopen.ckan.net to isitopendata.org)

Example link is: 

http://isitopen.ckan.net/enquiry/create/?ckan_package=zeno

which gets forwarded to:

http://www.isitopendata.org/enquiry/create/?ckan_package=zeno 

which gives 404"	1267012666000000	1273596153000000
262	enhancement	rgrp	rgrp		closed	fixed	Mark package as duplicate of another package	"Need a way to mark packages as duplicate.

Ideas/Questions:
  * Could work off a tag (meta-duplicate or somesuch)
  * What happens if we want to delete the package (to we leave a place-holder, do we merge content?) "	1267139682000000	1296468392000000
268	defect	rgrp	dread		closed	duplicate	Select groups in Package edit form		1268068896000000	1285070682000000
269	enhancement	rgrp	dread		closed	fixed	Improve gov package form	"If the notes field could use a WYSIWYG editor with word cleanup this would really help users who may well be pasting in text from Word (and I guess may have been approved or written by someone else)

having auto-complete on tags would both make the system easier to use - and reduce the risk of synonymous tags being created inadvertently.

Does the department drop down options list interact with user permissions - so that users who only have rights to one department only see that option ?

Similarly for licenses - can this reduce to the set of allowed options for this user/

What level of validation is there on the fields?"	1268220853000000	1291897538000000
272	enhancement	rgrp	rgrp		closed	fixed	Atom/RSS feeds for individual tags and groups	"I want to be able to easily check when a package is added (or removed?) from a given tag or groups. To do this I would like access to an Atom/RSS feed for a given tag or group.

  * This would allow people to embed feed widgets in other places"	1268988652000000	1340631775000000
274	defect	rgrp	rgrp		closed	fixed	Allow searching by any extra field	"At the moment only support a very limited set of extra fields (see docs for details). Should support arbitrary extra fields (note no need to put these in special full text index though ...)

Cost: 1h"	1269034562000000	1287402800000000
283	enhancement	rgrp	dread		closed	wontfix	Manage deletions of unwanted packages	"== Use case ==

As a user I want to notify the CKAN admins of a spammed or unsuitable package for deletion.

== Suggested solution ==

In the package view side-bar, there is a note: ""To have this package completely removed, contact the [ca.ckan.net administrators admin@ca.ckan.net].""

== Other solutions ==

A more complicated solution would be to allow packages to be tagged for deletion, which would auto-alert administrators, and allow easier administration of this. But this might be overkill. 

c.f. http://en.wikipedia.org/wiki/Deleting_an_article"	1270660210000000	1311325526000000
289	enhancement	rgrp	dread	v1.0	closed	fixed	Document config options		1271248690000000	1271249368000000
291	defect	rgrp	dread		closed	worksforme	Investigate search options encoding	"On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in.


{{{
diff -r 813ad8b5de0b ckan/lib/search.py
--- a/ckan/lib/search.py	Mon Mar 01 22:23:36 2010 +0100
+++ b/ckan/lib/search.py	Thu Apr 15 19:11:42 2010 +0200
@@ -63,6 +63,9 @@
     def query(self, options):
         '''For the given search options, returns a query object.'''
         self._options = options
+        # temporary hack!
+        if self._options.q:
+            self._options.q = self._options.q.encode('utf8')
         general_terms, field_specific_terms = self._parse_query_string()
 
         if not general_terms and \

}}}
"	1271351670000000	1273254895000000
300	defect	rgrp	dread	v1.0	closed	fixed	Resource ordering issue	"Failing test: ckan.tests.models.test_resource.TestResourceLifecycle.test_03_reorder_resources

Not clear how visible this is to the user.

Related to ticket:292"	1272285994000000	1272384474000000
306	enhancement	rgrp	rgrp		closed	duplicate	datapkg build command	"Need to be able to build a distribution. Need:

  * new 'build' command
  * specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution.

"	1272474212000000	1318181194000000
307	defect	rgrp	rgrp	datapkg-0.7	closed	fixed	datapkg upload command	"Support uploading a distribution somewhere.

  * New cli command: upload
  * Pluggable uploader framework. Suggest initial support 

Depends on ticket:306"	1272474224000000	1288027815000000
308	enhancement	rgrp	rgrp		closed	duplicate	Autocomplete package names & tags in package search	"Extracted from ticket:216.

Dubious of its merit."	1273050549000000	1275302577000000
312	defect	rgrp	dread		closed	invalid	Race condition creating PackageRating	"On ckan.net there are a few packages which cause a 403 exception when you try to rate them:
http://www.ckan.net/package/rate/coins-data?rating=3

The exception is occuring here:
ckan.rating:39 in set_rating
rating_obj = rating_query.one()
InvalidRequestError: Multiple rows returned for one()

It looks like this package got rated twice in quick succession, creating two similar PackageRating objects. This race condition needs to be solved and these particular rating objects repaired."	1273482785000000	1311176173000000
313	enhancement	rgrp	dread	v1.1	closed	fixed	Allow packages to be specified by IDs in REST interface	"If a package name changes, a simple CKAN client may not be aware of this (not monitoring the push notifications, revisions or feed), so it is preferable to refer to the package by its (invariant) ID.

It is still useful to refer to a package by its name though, so both should be valid arguments in the REST interface.

"	1273606248000000	1275404524000000
316	defect	rgrp	dread		closed	fixed	Search URL escaping	"If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error.

e.g.
http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk
fjdkf2B´gfhgfkgf{gpk

returns this exception:

{{{
URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&search=Search+Packages+%C2%BB
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:50 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:52 in search
<<                  collection=query,
                       page=request.params.get('page', 1),
                       items_per_page=50
                   )
                   # filter out ranks from the query result
>>  items_per_page=50
Module webhelpers.paginate:333 in __init__
<<              self.item_count = item_count
               else:
                   self.item_count = len(self.collection)
       
               # Compute the number of the first and last available page
>>  self.item_count = len(self.collection)
Module webhelpers.paginate:204 in __len__
<<      def __len__(self):
               return self.obj.count()
       
       # Since the items on a page are mainly a list we subclass the ""list"" type
>>  return self.obj.count()
Module sqlalchemy.orm.query:1094 in count
<<              q = q.params(params)
               q = q._legacy_select_kwargs(**kwargs)
               return q._count()
       
           def _count(self):
>>  return q._count()
Module sqlalchemy.orm.query:1103 in _count
<<          """"""
               return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
       
           def _col_aggregate(self, col, func, nested_cols=None):
>>  return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
Module sqlalchemy.orm.query:1125 in _col_aggregate
<<          if self._autoflush and not self._populate_existing:
                   self.session._autoflush()
               return self.session.scalar(s, params=self._params, mapper=self.mapper)
       
           def compile(self):
>>  return self.session.scalar(s, params=self._params, mapper=self.mapper)
Module sqlalchemy.orm.session:635 in scalar
<<          engine = self.get_bind(mapper, clause=clause, instance=instance)
       
               return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
       
           def close(self):
>>  return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
Module sqlalchemy.engine.base:834 in scalar
<<          """"""
       
               return self.execute(object, *multiparams, **params).scalar()
       
           def statement_compiler(self, statement, **kwargs):
>>  return self.execute(object, *multiparams, **params).scalar()
Module sqlalchemy.engine.base:844 in execute
<<          for c in type(object).__mro__:
                   if c in Connection.executors:
                       return Connection.executors[c](self, object, multiparams, params)
               else:
                   raise exceptions.InvalidRequestError(""Unexecutable object type: "" + str(type(object)))
>>  return Connection.executors[c](self, object, multiparams, params)
Module sqlalchemy.engine.base:895 in execute_clauseelement
<<          else:
                   keys = None
               return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
       
           def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None):
>>  return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
Module sqlalchemy.engine.base:907 in _execute_compiled
<<          context.pre_execution()
               self.__execute_raw(context)
               context.post_execution()
               self._autocommit(context)
>>  self.__execute_raw(context)
Module sqlalchemy.engine.base:916 in __execute_raw
<<              self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context)
               else:
                   self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
       
           def _execute_ddl(self, ddl, params, multiparams):
>>  self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
Module sqlalchemy.engine.base:958 in _cursor_execute
<<              self.engine.logger.info(repr(parameters))
               try:
                   self.dialect.do_execute(cursor, statement, parameters, context=context)
               except Exception, e:
                   self._handle_dbapi_exception(e, statement, parameters, cursor)
>>  self.dialect.do_execute(cursor, statement, parameters, context=context)
Module sqlalchemy.engine.default:133 in do_execute
<<      def do_execute(self, cursor, statement, parameters, context=None):
               cursor.execute(statement, parameters)
       
           def is_disconnect(self, e):
>>  cursor.execute(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128)
}}}"	1274265928000000	1291831177000000
329	defect	rgrp	dread	v1.1	closed	fixed	Bad dates cause exception on Gov form	"== Reproduction ==

Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message.

Affects all versions of CKAN.

== Why it's happening ==

The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught."	1275060617000000	1275079189000000
355	defect	rgrp	rgrp		closed	fixed	Dashes versus underscores in package names	Sort out how we deal with dashes versus underscores in package names.	1277221996000000	1311177552000000
358	enhancement	rgrp	dread	ckan-v1.5	closed	duplicate	Resources in REST API	"(spun out of ticket:336)

Resource added to model API at:

api/rest/resource


=== Example model request ===

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028

returns:
{{{
 [{""id"": ""a3dd8f64-9078-4f04-845c-e3f047125028"",
   ""package_id"": ""b8a325c8-af2a-43f3-8245-9db7d73dfbfe"",
   ""URL"": ""http://scraperwiki.com/lincolnshire-councillors"", 
   ""format"": ""CSV"", 
   ""Description"": ""Scrape of www.lincs.gov/councillors.pdf by ScraperWiki."",
   ""hash"": """", 
   ""position"": 2
 }]
}}}

== Authorization ==

1. Have it generic (ie. not per resource) and use an action/role on system
2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
3. Introduce Resource in authorization system (requires migration)

Mixed model

{{{
Create / Edit:

if resource associated to package:
    check_permissions(package, update)
else:
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)
}}}"	1277483282000000	1310128782000000
360	defect	rgrp	rgrp	datapkg-0.7	closed	fixed	New sqlite-based DB index	"In 0.5 and 0.6 have been more oriented to getting file indexes working to support the convenience of using packages on disk directly. However we now need to go back to having a better central/default index and the best model for this is a simple db index using sqlite (which is now supported in the standard library).

NB: this is a retrospective ticket (been working on this for a couple of months)"	1277803174000000	1288004891000000
361	enhancement	rgrp	rgrp	datapkg-0.7	closed	fixed	Allow specification of the default index in configuration		1277803305000000	1291135756000000
404	requirement	rgrp	pudo	datapkg-0.8	closed	fixed	Make registries pluggable	Allow using different indexes, including ckan, file, dcat, ... 	1281346690000000	1297072955000000
405	task	rgrp	rgrp	datapkg-0.8	closed	fixed	Retrieval options for package resources	"Download Command (was install command) should support the following modes: 

 * Download only the first listed resource (current behaviour, slightly arbitrary) 
 * Download resources by interactive selection 
 * Download resources by MIME type (cf #235) 
 * Download all resources. 

"	1281346806000000	1297214833000000
409	task	rgrp	rgrp	datapkg-0.8	closed	fixed	Refactor cli to be more 'pluggable'	"  * ~~(Load cli commands from setuptools entry point name datapkg.cli)~~
  * Document how other can write new commands
  * Refactor base Command class to support this pluggability (if necessary)"	1281348659000000	1297074197000000
458	task	rgrp	johnbywater		closed	wontfix	Request to DGU for access to database machine		1282299950000000	1294415537000000
464	task	rgrp	dread		closed	duplicate	Request dgu db server access		1282306104000000	1282325194000000
538	task	rgrp	johnbywater		closed	wontfix	Update CKAN Trac ticket report queries	"Update report queries with more recent versions (below are report queries shipped with trac 0.11.7):

NB: Report 1 has already been done.


 id |                                               query                                               
----+---------------------------------------------------------------------------------------------------
  1 | 
    : SELECT p.value AS __color__,
    :    id AS ticket, summary, component, version, milestone, t.type AS type, 
    :    owner, status,
    :    time AS created,
    :    changetime AS _changetime, description AS _description,
    :    reporter AS _reporter
    :   FROM ticket t
    :   LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
    :   WHERE status <> 'closed'
    :   ORDER BY CAST(p.value AS integer), milestone, t.type, time
    : 
  2 | 
    : SELECT p.value AS __color__,
    :    version AS __group__,
    :    id AS ticket, summary, component, version, t.type AS type, 
    :    owner, status,
    :    time AS created,
    :    changetime AS _changetime, description AS _description,
    :    reporter AS _reporter
    :   FROM ticket t
    :   LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
    :   WHERE status <> 'closed'
    :   ORDER BY (version IS NULL),version, CAST(p.value AS integer), t.type, time
    : 
  3 | 
    : SELECT p.value AS __color__,
    :    'Milestone '||milestone AS __group__,
    :    id AS ticket, summary, component, version, t.type AS type, 
    :    owner, status,
    :    time AS created,
    :    changetime AS _changetime, description AS _description,
    :    reporter AS _reporter
    :   FROM ticket t
    :   LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
    :   WHERE status <> 'closed' 
    :   ORDER BY (milestone IS NULL),milestone, CAST(p.value AS integer), t.type, time
    : 
  4 | 
    : 
    : SELECT p.value AS __color__,
    :    owner AS __group__,
    :    id AS ticket, summary, component, milestone, t.type AS type, time AS created,
    :    changetime AS _changetime, description AS _description,
    :    reporter AS _reporter
    :   FROM ticket t
    :   LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
    :   WHERE status = 'accepted'
    :   ORDER BY owner, CAST(p.value AS integer), t.type, time
    : 
  5 | 
    : SELECT p.value AS __color__,
    :    owner AS __group__,
    :    id AS ticket, summary, component, milestone, t.type AS type, time AS created,
    :    description AS _description_,
    :    changetime AS _changetime, reporter AS _reporter
    :   FROM ticket t
    :   LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
    :   WHERE status = 'accepted'
    :   ORDER BY owner, CAST(p.value AS integer), t.type, time
    : 
  6 | 
    : SELECT p.value AS __color__,
    :    t.milestone AS __group__,
    :    (CASE status 
    :       WHEN 'closed' THEN 'color: #777; background: #ddd; border-color: #ccc;'
    :       ELSE 
    :         (CASE owner WHEN $USER THEN 'font-weight: bold' END)
    :     END) AS __style__,
    :    id AS ticket, summary, component, status, 
    :    resolution,version, t.type AS type, priority, owner,
    :    changetime AS modified,
    :    time AS _time,reporter AS _reporter
    :   FROM ticket t
    :   LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
    :   ORDER BY (milestone IS NULL), milestone DESC, (status = 'closed'), 
    :         (CASE status WHEN 'closed' THEN changetime ELSE (-1) * CAST(p.value AS integer) END) DESC
    : 
  7 | 
    : SELECT p.value AS __color__,
    :    (CASE status WHEN 'accepted' THEN 'Accepted' ELSE 'Owned' END) AS __group__,
    :    id AS ticket, summary, component, version, milestone,
    :    t.type AS type, priority, time AS created,
    :    changetime AS _changetime, description AS _description,
    :    reporter AS _reporter
    :   FROM ticket t
    :   LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
    :   WHERE t.status <> 'closed' AND owner = $USER
    :   ORDER BY (status = 'accepted') DESC, CAST(p.value AS integer), milestone, t.type, time
    : 
  8 | 
    : SELECT p.value AS __color__,
    :    (CASE owner 
    :      WHEN $USER THEN 'My Tickets' 
    :      ELSE 'Active Tickets' 
    :     END) AS __group__,
    :    id AS ticket, summary, component, version, milestone, t.type AS type, 
    :    owner, status,
    :    time AS created,
    :    changetime AS _changetime, description AS _description,
    :    reporter AS _reporter
    :   FROM ticket t
    :   LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
    :   WHERE status <> 'closed' 
    :   ORDER BY (COALESCE(owner, '') = $USER) DESC, CAST(p.value AS integer), milestone, t.type, time
"	1283190081000000	1294414537000000
581	task	rgrp	johnbywater	ckan-v1.2	closed	fixed	Determine package name from metadata document	Need to establish procedure for automatically generating package names from metadata documents.	1284210874000000	1284689274000000
715	defect	rgrp	johnbywater	ckan-v1.4-sprint-2	closed	fixed	The group list page shall show the group title *or* group name	"Symptom #1: Groups without titles don't get a clickable link, because there is no text. Defaulting to the group name would solve this trouble.
Symptom #2: Package groups drop-down list show's 'None' when a group doesn't have a title.

Therefore, we need to centralize presentation of group title, so it defers to group name. And we need tests.

Also please sort groups by title on group list page."	1287583089000000	1297796784000000
776	enhancement	rgrp	dread		closed	duplicate	Avoid generating vdm warnings	"We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided?

{{{
2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object
2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object
2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object
2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object
2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object
2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object
}}}"	1288372797000000	1297066840000000
797	enhancement	rgrp	rgrp	ckan-v1.3	closed	fixed	Rework core html layout to mirror wordpress twentyten	"WP twentyten has an excellent core html structure. Furthermore, using that structure makes us compatible with all the WP twentyten compatible themes.

  * Convert to wp twentyten html
  * Switch css to be based off twentyten css (and perhaps rework somewhat)"	1289402873000000	1289402982000000
806	enhancement	rgrp	rgrp	ckan-v1.3	closed	fixed	Metadata created and last modified timestamps for packages	"Packages should provide create and last modified attributes which are included in package dict exposed via api.

These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc)"	1289854894000000	1324034356000000
827	defect	rgrp	dread		closed	fixed	metadata_created_and_modified error	"Intermittent failure on ckan default branch. Looks like brittle test.

{{{
======================================================================
FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified
----------------------------------------------------------------------
Traceback (most recent call last):
  File ""/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py"", line 186, in runTest
    self.test(*self.arg)
  File ""/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py"", line 250, in test_02_metadata_created_and_modified
    assert out == exp, (out, exp)
AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938))
}}}

http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio"	1290678624000000	1290766239000000
828	enhancement	rgrp	rgrp	ckan-v1.3	closed	fixed	Update cache support in home controller to use cache config in ini file	"Currently cache usage in home controller is inconsistent with elsewhere and, in particular, does not respect cache_disabled and is not configurable using path notation (e.g. ckan.controllers.home. ...).

This is especially frustrating in a development environment where is currently running into fact front page is being cached.

Fix this."	1290693154000000	1290696586000000
829	defect	rgrp	dread	ckan-v1.4-sprint-6	closed	wontfix	Admin CRUD broken	Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception.	1290695043000000	1303838115000000
831	task	rgrp	rgrp	ckan-v1.3	closed	fixed	Migrate ckan trac to ckan.org	Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc.	1290760820000000	1290760864000000
832	enhancement	rgrp	rgrp	ckan-v1.3	closed	fixed	Convert stats to a plugin	"Convert stats section to a ckan plugin.

Stats need not be in core and making it a plugin would make it easier to extend and expand."	1290765915000000	1296334980000000
852	enhancement	rgrp	wwaites	ckan-v1.5	closed	fixed	[super] Dataset upload and archiving	"3 (4) use cases: (With and without access to auth keys.)

 1. upload form on ckan (by default local system has auth key for storage)
 2. datapkg upload (client) (by default without key) (plumbing done)
 3. background job archiving downloads (by queue client) 

Most important are:

1. Upload form on ckan.net (with access to keys)
2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

=== Sub-tickets ===

''' Required infrastructure '''

 * #826 - Resource 'extra' fields
 * #358 - Resources in REST API
 * #235 - Resource format normalization and detection

''' User upload via Web UI '''

 * #877 - File upload in WUI
 * #878 - Integrate file upload with workflow around package resources

''' User upload via CLI/datapkg '''

 * #853 - plumbing for credential-less rest upload
 * #879 - Storage auth API

Also related:

 * #405 - Retrieval options for package resources

=== More Info ===

 * Upload/storage discussion: http://ckan.okfnpad.org/storage
  * For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/
"	1291722993000000	1315821628000000
869	enhancement	rgrp	rgrp	ckan-v1.3-sprint-1	closed	fixed	Add plugin interface to allow editing and extending of config by extensions	"Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)"	1291989395000000	1292235262000000
877	enhancement	rgrp	rgrp	ckan-v1.4-sprint-2	closed	fixed	File upload in WUI (extension)	"Support uploading files to a storage system from WUI with a simple upload form.

=== Implementation ===

  * Implement as an [wiki:Extensions Extension]
  * Locate at /upload
  * Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
  * Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
  * This wants to be tied in to general package workflow. See parent ticket:852."	1292868146000000	1298624165000000
878	enhancement	rgrp	rgrp	ckan-sprint-2011-09-12	closed	fixed	Integrate file upload with workflow around package resources	"Create/edit a package -> add a resource -> Oh, i haven't uploaded yet, 'upload here' -> /upload -> standard upload form

Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata?

If yes suggests (need to ticket):

  * 'ResourceExtras' for extra metadata on resources
  * Makes 'Resource' into first class object (and make available in WUI)?

== Depends ==

 * #877 (Upload in WUI) - should be part of that extension"	1292868269000000	1315820838000000
889	enhancement	rgrp	rgrp	ckan-v1.3-sprint-2	closed	fixed	Support extra footer material in config option (e.g. for google analytics)	"Add a config option 'template_footer_end' which is inserted in layout_template just before </body>.

This allows sysadmins to add extra items, especially, scripts directly into site without having to do any theming and is especially useful for things like google analytics.

Aside: going forward may want to turn this into a extension.

Cost: 1h"	1293713189000000	1293715109000000
898	defect	rgrp	dread		closed	fixed	Changes stored indefinitely	"Every change to every object is being stored in memory, which could add up to quite a lot of memory.  

This fixes it by making sure the objects are in a weakref.  https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f"	1294659490000000	1294662408000000
899	enhancement	rgrp	dread		closed	fixed	Optimise check_real_change	"There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e

From David Raznick"	1294659583000000	1294916615000000
901	enhancement	rgrp	dread		closed	fixed	vdm support for sqla 0.6 and sqlite		1294660144000000	1294662466000000
902	defect	rgrp	dread	ckan-v1.4-sprint-1	closed	fixed	Package buttons not clickable / Improve submenu tabs	"On a package page there are three buttons to the right - View, Edit, History. Each of these buttons contain an icon and the text, but you can only activate the button by clicking on the icon. When you click on the part of the button where the text is (about 50% of the button) then nothing happens.

Using: Chrome, Firefox"	1294756898000000	1297078885000000
904	task	rgrp	Stiivi	ckan-v1.4-sprint-3	closed	fixed	Review CKAN documentation	"What's bad at the moment?

 * lack of documentation e.g. config (very poorly documented)
 * too many sources of documentation
 * no common theming

Sources:

 * http://ckan.org/
 * http://packages.python.org/ckan 
 * http://wiki.okfn.org/ckan/ - user
 * Deprecate: http://knowledgeforge.net/ckan/doc/ckan/

Resulting meta-ticket with things to do: ticket:927"	1294832610000000	1299840539000000
916	enhancement	rgrp	pudo	vdm-0.8	closed	duplicate	Port new vdm to Mongodb	"We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG. 

"	1295277011000000	1297066902000000
923	defect	rgrp	dread		closed	worksforme	Search box doesn't work in leaderboard page	"1. Go to: http://ckan.net/stats/leaderboard#content
2. In the far top-right of the browser, select the search box in 'water'.
3. Press enter to search. Nothing happens.

Tried in: chrome, firefox"	1295867328000000	1340632144000000
927	enhancement	rgrp	rgrp	ckan-v1.4-sprint-3	closed	fixed	[super] Improvements to CKAN documentation	"List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along.

== Major ==

 * ~~(Make blog.ckan.org into ckan.org and move ckan.org to trac.ckan.org/)~~
 * Document caching system - ticket:841
 * Howto guide for writing an extension - ''started on this at http://wiki.ckan.net/Extending_CKAN''

== Miscellaneous ==

 * Contributors / credits page: ticket:928
 * ~~(Data proxy documentation: move it to packages.python.org/dataproxy)~~
 * ~~(List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))~~
 * Document license configuration and license system - ticket:973
 * Refactor trac frontpage moving general feature overview to ckan.org
 * Create proper roadmap information on http://trac.ckan.org/
 * Add link to wiki.ckan.net to ckan.org
"	1295869492000000	1300105638000000
928	enhancement	rgrp	rgrp	ckan-v1.4-sprint-1	closed	fixed	Documentation - CKAN credits / contributors page	" 1. Remove contributors list from about page (out-of-date and does not belong there)
  * A ""Powered by CKAN"" link in footer is enough
 2. Add contributors / team / credits page to ckan.org
  * May want to separate credits (e.g. software use), from contributors (also is team different from contributors)
  * (perhaps with photos!) so that people can see who is part of it and from which organisations. (this is medium-term, not really part of this ticket)
 3. ? CREDITS.txt file to the source repository and add a link to it on bitbucket for more detailed credits info.

Cost: 1h"	1295872661000000	1297115136000000
929	defect	rgrp	rene.kapusta	ckan-v1.4	closed	fixed	Handle the case when the licenses service is down better	"I'm using v 1.3.1a and get a lot of ""WebApp Error: <type 'exceptions.Exception'>: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>"" error messages.

{{{
Module ckan.forms.package:87 in build_package_form
<<      # Options/settings
           builder.set_field_option('name', 'validate', package_name_validator)
           builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
           builder.set_field_option('state', 'dropdown', {'options':model.State.all})
           builder.set_field_option('notes', 'textarea', {'size':'60x15'})
>>  builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
Module ckan.model.package:283 in get_license_options
<<      @classmethod
           def get_license_options(self):
               register = self.get_license_register()
               return [(l.title, l.id) for l in register.values()]
>>  register = self.get_license_register()
Module ckan.model.package:278 in get_license_register
<<      def get_license_register(self):
               if not hasattr(self, '_license_register'):
                   self._license_register = LicenseRegister()
               return self._license_register
>>  self._license_register = LicenseRegister()
Module ckan.model.license:46 in __init__
<<              from licenses.service import LicensesService2
                   self.service = LicensesService2(group_url)
                   entity_list = self.service.get_licenses()
               else:
                   from licenses import Licenses
>>  entity_list = self.service.get_licenses()
Module licenses.service:44 in get_licenses
<<          except Exception, inst:
                   msg = ""Couldn't connect to licenses service: %s"" % inst
                   raise Exception, msg
               try:
                   licenses = loads(response_body)
>>  raise Exception, msg
Exception: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>
}}}"	1295995409000000	1299840884000000
953	enhancement	rgrp	rgrp	ckan-v1.4-sprint-1	closed	fixed	Add tagline/description to template and set in configuration	Have a tagline / description are under main title and set if from ckan.site_description config variable.	1296809808000000	1296809834000000
962	enhancement	rgrp	rgrp	ckan-v1.4-sprint-5	closed	fixed	Improvements to data preview extension	"Basic implementation done (and deployed):

https://bitbucket.org/okfn/ckanext-datapreview

However plenty to improve, e.g.

 * Support more formats (use external systems for preview?)
  * json (!)
  * html (trivial!)
  * sparql
  * ...
 * Do not display preview if no preview

Also suggest reworking to use external services rather than doing preview 'in house' (doing in house places heavy reliance on data proxy service and on converting data to a standard format)."	1297072524000000	1301364987000000
964	enhancement	rgrp	rgrp	datapkg-0.8	closed	fixed	Create Resource Downloader and make it pluggable	In downloading packages we need to download resources and we need a ResourceDownloader object for this. these should be pluggable so that we can add support for different types of resources.	1297211004000000	1297211237000000
973	enhancement	rgrp	rgrp	ckan-v1.4-sprint-1	closed	fixed	Document license configuration and license system.	"Document how license system works and specifically how licenses can be configured.

Cost: 2h"	1297332948000000	1297678851000000
975	defect	rgrp	rgrp	ckan-v1.4-sprint-1	closed	fixed	Placeholder attribute not displayed in some browsers leading to poor UX	"Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE).

This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search.

We should fix this by finding a way to display placeholder attribute in all browsers."	1297343755000000	1297344448000000
992	defect	rgrp	rgrp	ckan-v1.3	closed	fixed	Use X-forwarded-for whenever it is available to set remote IP address	"At the moment we only use X-forwarded to determine remote user address when remote-addr header is 127.0.0.1. However if a site is behind a cache remote_addr will always be the IP of the cache.

We should fix this by using x-forwarded-for header whenever available and only use remote-addr when it is not available.

Cost: 10m"	1298058532000000	1298060474000000
999	enhancement	rgrp	dread	ckan-v1.4-sprint-3	closed	fixed	Factor out ckan.net theme changes to a separate theme repo and apply	Rather than polluting the ckan core code base on ckan.net install with ckan.net specific changes these adaptations should be moved out into a dedicated ckan.net theme.	1298386729000000	1300707328000000
1001	enhancement	rgrp	rgrp	ckan-v1.4-sprint-4	closed	fixed	API should use normal user credentials if available	"When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key.

The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...).

This is critical to incorporating any ajax editing into the frontend.

As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author)."	1298489705000000	1301310351000000
1003	enhancement	rgrp	rgrp	ckan-v1.4-sprint-3	closed	fixed	CKAN Javascript library and demonstration web interface	"A plain javascript library for interfacing with CKAN would be very useful (why? see below!). It would also be nice to have a pure html + javascript web interface to CKAN both for its own sake and to act as a demonstrator for the library.

Why?

 * Development of bespoke interfaces -- much easier to edit html + javascript than to change ckan core
  * E.g. for specific communities e.g. geodata, science
  * Specialized tasks - multi-package editing
 * Very easy deployment and integration (e.g. can drop in to getthedata or other sites)
"	1298490086000000	1300100411000000
1008	defect	rgrp	pudo	ckan-v1.4-sprint-2	closed	fixed	eval() of user display name in template head	"We're currently setting the user preferences links via a javascript snippet that also evals the name. This should be removed as we're not displaying the user name any longer. 

We should also have page fragment caching in Genshi, which is not currently implmented. 

cf. http://api.rubyonrails.org/classes/ActionController/Caching/Fragments.html"	1298632686000000	1298821826000000
1010	enhancement	rgrp	rgrp	ckan-v1.4-sprint-2	closed	fixed	List CKAN users in WUI	"Should have user listing at /user/ rather than user account page.

 * list users, sorted by number of packages contributed/edited
 * Move user home page to /user/{user-id}
 * Paginated"	1298649180000000	1298740889000000
1032	enhancement	rgrp	rgrp	ckan-v1.6	closed	fixed	[super] Resources in WUI	"Add resources into Web User Interface.

 * Locate at: /dataset/{dataset}/resource/{id}
 * CRUD
 * Authorization

Done:

 * ~~#945 - Richer resources - Resource Groups, new fields, improved UI~~
 * ~~#1445 - Resource View page in WUI~~
 * ~~#1450 - Dataset view pages to match~~

Moved to superticket #1506: 

 * #978 - Edit Resource Extras in Web UI
"	1299782021000000	1330348463000000
1070	enhancement	rgrp	rgrp	ckan-v1.5	closed	fixed	Plan a new domain model and layer architecture for CKAN	"See http://wiki.ckan.net/Domain_Model especially section on v2.

 * New domain model is planned but not yet finally agreed.
 * Layer architecture is complete and implemented"	1301910940000000	1310117129000000
1074	enhancement	rgrp	rgrp	ckan-v1.5	closed	fixed	Refactor authz web user interface to have common code and templating	"Currently repeat the same template and code across Package Authz, Group Authz, and Authz Group authz.

Having now implemented a new, cleaner setup in ckanext-admin we should port this back into core.

 * Common template code (checkbox template)
 * Logic code (or just common code) for wiring into authz system
 * Look for all places thoroughout the system where usernames, authzgroups or groups need to be typed into boxes, and make sure that they auto-complete appropriately.

Will also deliver a significant improvement in the form of ajax user lookup."	1302271586000000	1314303581000000
1106	defect	rgrp	rgrp	ckan-v1.4-sprint-6	closed	fixed	Bugs related to routes arising from API refactor + removal of default routes	"Various bugs I've been encountering:

 * Autocomplete of tag names no longer works (no longer works on http://ckan.net/). Appears to be due to no longer have a routes for apiv2 (i'm seeing failing calls to:  http://ckan.net/apiv2/package/autocomplete?callback=callback&incomplete=a)
 * Incorrect url generated for API in page footer (e.g.  http://ckan.net/rest/get_api) due to use of old 'rest' rather than new 'api'

Latter issue was masked by existence of 'default' routes:

{{{
   map.connect('/{controller}', action='index')
   map.connect('/:controller/{action}')
   map.connect('/{controller}/{action}/{id}')
}}}

Having these is, I think, bad practice as it is better to be explicit and we should therefore remove asap.

In addition I think we should be cautious about 'default' routes in core such as:

{{{
    map.connect('/api/rest/:register', controller='api', action='list',
        conditions=dict(method=['GET'])
        )
}}}

As it makes it harder for extensions to introduce their own APIs (here one could perhaps add something at /api/rest/{my-object} but only by using before_map rather than after_map).
"	1303747360000000	1303834069000000
1119	enhancement	rgrp	rgrp	ckan-v1.4-sprint-7	closed	fixed	Fully functional storage extension with file upload	"Previous work in #877 and #879 + #853 (storage API). In this ticket:

 * Improve authorization
 * Establish convention for laying out files on disk
 * Add documentation
 * Fix bugs with #879 (does not currently work -- boto may have changed)

"	1304094382000000	1305037201000000
1151	enhancement	rgrp	timmcnamara	ckan-sprint-2012-03-05	closed	wontfix	Preview for geographic data should be a map	"Data viewer / previewer for a resource that has a KML file, or others, such as GeoRSS and GeoJSON, we should provide a map.

 * Support for KML files: #1458

This is non-trivial for external files as we need a way to jsonify. For files stored locally this is more of a recline issue (and will require a bit of work to either guess columns or allow user to specify them)."	1306043217000000	1330908188000000
1163	enhancement	rgrp	rgrp	ckan-backlog	new		Improvements to Storage Extension	"Storage is now working but there are 

 * Integrate with Resources (e.g. create a resource for each file upload and give option to associate with a package)
  * Should we introduce rule that files *not* associated with a Resource are periodically deleted?
 * Allow setting of a file name/path before upload
 * Allow for file overwriting/deleting etc (how should this work -- do we want to allow this sort of thing)
 * Integrate local file upload stuff in api/auth/*

== Different Backend Issues ==

Local file store is rather different from 'remote' storage in various ways:

 * For remote you don't want to use many buckets as there are bucket limits while for local you want to. Should we there have a single path that users provide which we then partition differently for different backends.
"	1306408778000000	1310133808000000
1190	enhancement	rgrp	rgrp	ckan-v1.5	closed	fixed	[super] CREP 0004 Data API and Data Processing System	"For some time (e.g. 1y+!) we have known that we want to integrate some kind of datastore / data processing system with CKAN. We've had a CREP in progress on this for some months (may copy that here at some point):

http://wiki.ckan.org/CEP0004

We can distinguish 3 modules that are needed:

1. ""Webstore"": A datastore with dataapi - #1208

Suggestion is this would be sqlite based with a simple sql based API. http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query}

2. Automated conversion of suitable resources into datastore upon resource creation so that e.g. they are accessible via the API. #1398

3. A data processing system which utilizes this datastore. One could
get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) - #1432


== More info ==

 * Overview diagram: https://docs.google.com/drawings/d/1XK7dcpFXNlMzVFgLPYPZUXOsFRPzgCEvS-w7FcvydEQ/edit?hl=en_GB"	1308227611000000	1323270522000000
1195	defect	rgrp	dread	ckan-sprint-2011-10-28	closed	fixed	Storage extension to handle incorrect routes better	"We've seen exceptions caused by people getting the file storage link wrong, adding a slash to the end.

e.g.
http://ckan.net/storage/f/file/e696d857-e997-41c8-be08-408697168ca8/

We should aim to handle all requests better than causing exception. 404 (or even 302 in this case) is better than 500."	1308576769000000	1311964374000000
1202	enhancement	rgrp	rolf		closed	invalid	Links to datapkg utility don't lead to info about it	"In the presentation of data packages, the link in ""The information on this page and the downloads / resources are also available using the datapkg command line utility."" (http://knowledgeforge.net/ckan/doc/datapkg/install.html) ends up at the home page of ckan.org

Perhaps link it to https://bitbucket.org/okfn/datapkg/overview for now, to save some searching for people who want to find out more?"	1309263613000000	1315821804000000
1208	enhancement	rgrp	rgrp	ckan-sprint-2011-10-28	closed	fixed	Webstore: a datastore with web API	" * SQLite based
 * Use Cyclone for async http server on top with auth
 * Designed to not be specific to CKAN or anything else

Current code is here: https://github.com/okfn/webstore.

== API Spec ==

==== Read ====

Two basic ways to query::

{{{
    GET: /{owner}/{db-name}/?sql=...
    GET: /{owner}/{db-name}/?table=...&attr=value&attr=value&limit=...
}}}

Returns::

{{{
  {
      u'keys': [u'id', u'name'],
      u'data': [
          [1, u'jones'],
          [u'aaa', u'jones']
          ]
  }
}}}

==== Write ====

POST to::

    /{owner/{database}/{table}

Payload is json data structured as follows::

  {
      unique_keys: [list of key attributes]
      data: {dict of values}
  }


=== Authentication and Authorization ===

Authentication: use basic auth header.


Authorization:

  * Default: all read, owner can write
  * Restricted: owner can read and write, everyone can do nothing

Possible future: config file can specify a python method (TODO: method
signature)


=== Integration with Other Systems ===

TODO: Specify how to delegate authenatication to user database in some other system.

"	1309804460000000	1312191646000000
1239	defect	rgrp	rgrp	ckan-sprint-2011-10-28	closed	fixed	Remove and re-add tag on package not working	"See reports on list at:

 * http://lists.okfn.org/pipermail/ckan-dev/2011-July/001103.html
 * http://lists.okfn.org/pipermail/ckan-dev/2011-July/001107.html

Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix."	1311441020000000	1312453156000000
1247	task	rgrp	dread	ckan-sprint-2011-10-28	closed	fixed	Put docs on ReadTheDocs	Put docs on ReadTheDocs instead of eu13.	1311941948000000	1311955185000000
1251	enhancement	rgrp	rgrp	ckan-backlog	closed	fixed	Rename wiki.ckan.net to wiki.ckan.org	Inline with general naming policy going forward.	1311965147000000	1317315159000000
1252	enhancement	rgrp	rgrp	ckan-v1.5	closed	fixed	Deprecate ckan.net in favour of thedatahub.org	"Deprecate ckan.net in favour of thedatahub.org

 * Announce
 * Create thedatahub.org and have it run same data as ckan.net
 * Announce closing down of ckan.net with 302
 * Implement 302 permanent

Also redirect wiki.ckan.net to wiki.ckan.org"	1311965339000000	1318160334000000
1271	enhancement	rgrp	rgrp	ckan-sprint-2011-10-28	closed	fixed	CORS support	"CORS - http://www.w3.org/TR/cors/ - support.

This is what you do in Apache. Should do this in lib/base.py or similar.

{{{
    Header always set Access-Control-Allow-Origin ""*""
    Header always set Access-Control-Allow-Methods ""POST, PUT, GET, OPTIONS""
    Header always set Access-Control-Allow-Headers ""X-CKAN-API-KEY, Content-Type""

    # Respond to all OPTIONS requests with 200 OK
    # This could be done in the webapp
    # This is need for pre-flighted requests (POSTs/PUTs)
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} OPTIONS
    RewriteRule ^(.*)$ $1 [R=200,L]
}}}"	1313241839000000	1313753663000000
1282	enhancement	rgrp	rgrp	ckan-v1.5	closed	fixed	JS and WUI tidy up	"Consolidate js. Remove cruft.

 * Move stuff to vendor
  * Have all vendor libs local (?)                                             
  * jquery, jquery-ui etc
 * Consolidate autocomplete to use jquery-ui
  * package autocomplete (groups)
  * tag autocomplete (package)
  * user autocomplete (authz group create)
   * turns out that authz group edit had autocomplete disabled so refactoring meant enabling ...
 * Move forms.css into main  
 * Remove tag cloud from front page (can be example going forward)
 * Remove bookmarklet (since in wiki at: http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F)
 * Move name slug generation into standard js and clean up

Possible:
 1. JS unit tests for the given pages ...
 2. Option to use external site to serve js and css (e.g. CDN)

                                                       
"	1314090389000000	1314404540000000
1294	enhancement	rgrp	rgrp	ckan-v1.5	closed	fixed	[super] Package creation and editing UX improvements	"Largely an integration of work from ckanjs

 * #1295 Simplify package create form
 * Improvements to edit form
  * Split edit form into sections
  * Remove preview from edit form (introduce inline preview on description/notes field)
 * #1296 Improved resource adding/editing on dataset/page page
 * #878 File upload integrated into resource creation
 * #1297 In-place editing of notes field for existing packages
"	1314405811000000	1315948257000000
1295	enhancement	rgrp	rgrp	ckan-sprint-2011-09-12	closed	fixed	Simplify package create form	"Sub ticket of #1294

 * Initial screen just title, name, description (notes), license (and tags?).
 * State that you can add data and more information on next screen. This takes you to Package 'view' where you can then do more stuff.
  * Flash some instructions? e.g. that resources are missing?
 * Name slug should by default look uneditable (cf wordpress way to do the slug)"	1314406014000000	1315948009000000
1300	enhancement	rgrp	rgrp	ckan-sprint-2011-11-07	closed	fixed	Core changes to base theme	"Core changes to theme in order to make it easier to re-theme:

 * page_heading
 * side bar menu must be switchable (left to right) (through config or css)
 * Add optional_footer to complement optional_head
  * And adopt rule for template writers that all extra js must go in optional footer
  * That way we could move optional js to the bottom of the page to improve page load times.
 * Re-add support for body-class ... (seems to be removed by #1108)
 * more divs.
 * better labelling for css (and javascript)"	1314862223000000	1319794520000000
1311	enhancement	rgrp	rgrp	ckan-backlog	new		Modal user register and login form	"Subticket of: #1294

Rather than having to visit a dedicated page it would be good if registration and login could be done from a modal form (separate or combined ...).

== Why ==

 * It could be used from dataset creation page in situations where user needs to be registered / logged in to create a dataset (so we could allow someone to start creating a dataset and only get them to login at the end ...)
 * It allows for quicker and easier logging in

== Implementation ==

 * See Friedrich's work on the datahub"	1315297227000000	1315297227000000
1319	enhancement	rgrp	dread	ckan-sprint-2011-10-24	closed	fixed	Require an email address on User registration	"Superticket: #1343

This avoids difficulties when they forget their password.

* (?) Email validation (navl does not have one!)
* Require email confirmation to be activated (?) (Answer: no, not at present -- see discussion below)"	1315415932000000	1324036947000000
1343	enhancement	rgrp	rgrp	ckan-backlog	new		[super] User related improvements (login, user pages etc)	" * Disallow account creation via openid - #1386
 * Require email field - #1319
  * Require email confirmation to be activated (?)
 * Improvements to user page (e.g. show activity and more info about user) - #1396
 * Modal user login - #1311"	1316017098000000	1318528138000000
1357	enhancement	rgrp	rgrp	ckan-sprint-2011-10-10	closed	fixed	Integrate new version of data preview	"We have an all new and much improved data preview system developed as part of QA in July. Need to integrate this.

 * Is pure javascript
 * Will have this in CKAN core but considered to be a plugin (so need datapreview in ckan.plugins list to work -- will enable this by default)
 * Will depend on the http://wiki.ckan.org/DataExplorer"	1317076044000000	1318164458000000
1359	enhancement	rgrp	rgrp	ckan-sprint-2012-02-20	closed	fixed	Radically simplify standard list of licenses	"Major UX grip and incredibly simple to do. May also want to have basic autocomplete support within the list (trivial to do as well I believe).

(Talk to RP about this re how we define lists of licenses)."	1317077855000000	1329132464000000
1361	enhancement	rgrp	rgrp	ckan-v1.5	closed	fixed	Simple search support	"Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install).

Propose:

 * ckan.simple_search config option
 * If set:
   * query via simple query to database backend
   * do no specialized indexing

== Extras ==

Remove

  * TagSearchQuery from lib/search (just do a search directly)?
  * #1360: filter by downloadable and filter by open search options

== Possible Extras ==

(Probably future improvements)

 * Re-introduce full-text search indexing where supported in e.g. postgres and use for querying


"	1317078276000000	1317082893000000
1384	task	rgrp	shevski	ckan-backlog	new		CKAN wiki needs updating to refer to thedatahub.org instead of ckan.net and datasets instead of packages	Most articles still refer and link to ckan.net, wiki.ckan.net and to packages	1318414077000000	1318414077000000
1386	enhancement	rgrp	rgrp	ckan-sprint-2011-10-24	closed	fixed	Disallow account creation via openid	"Superticket: #1343

Creation of accounts from OpenID causes large number of problems:
 * No guarantee of other profile info
 * Poor username (just from openid)

We therefore will:

 1. Require creation of user account via register (no auto-creation of accounts via openid)
  * sidebar of register page should point to login page rather than openid signin
 
 2. Permit association of an openid with an account ...
  * How: paste in your openid url (requires change to edit form to have this)
    * won't work for google with their weird openid urls
  * [future] Nicer way is to have login via openid while logged in (which association then happening in background)

 3. Migration (for 1+2):
  * (?) Generate decent usernames for all existing accounts
  * As this is nontrivial suggest instead we allow editing of usernames (by account owner and sysadmins). This is useful in its own right and is a reasonable 80/20 solution.
    * This does present the problem of re-associating commits with the new username. Best solution to this would be to switch revisions to point to userid rather than username. See #1534

 4. [optional] Change display_name to always be username
  * Change My Account to Username at top right of all pages (once we know
    usernames are short ...)
    * could do this straight away by truncating long usernames (e.g. truncate
      at 20 chars ...)
    * Also should we lose the icon?
"	1318425493000000	1324036930000000
1396	enhancement	rgrp	rgrp	ckan-sprint-2012-02-06	closed	fixed	Improvements to user page UX - 1d	"Super ticket: #1506

  * Show member since
  * Display email (in obfuscated form)
  * List all owned datasets (in standard manner)
    * Will need change to logic layer to show datasets on user dict (this is useful for the API as well ...)
  * (Re)move list of changes
    * Should have activity (once activity available)

WONTFIX:

  * ?? Allow users to provide a home page, organization (requires migration)
"	1318527967000000	1330086237000000
1401	defect	rgrp	flofokus	ckan-sprint-2012-01-23	closed	fixed	Some links don't take account when CKAN is mounted at a non-root URL	"if you want your ckan to reside not at
http://yourhost.org/
but rather e.g. at
http://yourhost.org/ckan/
you can use this hint
http://wiki.pylonshq.com/display/pylonsdocs/Configuration+Files
and set ckan.site_url accordingly in your .ini.
However, there are some instances where internal links are created without the required prefix."	1318621586000000	1328001014000000
1407	defect	rgrp	dread	ckan-sprint-2011-12-19	closed	fixed	Stats extension not working	Graphs don't show any more at http://thedatahub.org/stats	1319014605000000	1323764239000000
1412	defect	rgrp	seanh	ckan-sprint-2012-02-06	closed	worksforme	application.js crashes when viewing a dataset with no resources	"Create a new dataset, don't add any resources to the dataset yet, view the dataset in ckan, if you have firebug enabled you'll see the error ""resource is undefined"" at line 699 of application.js."	1319450625000000	1328226385000000
1429	enhancement	rgrp	rgrp	ckan-backlog	new		Provide DOIs for datasets in a CKAN instance	"DOI = digital object identifier = http://www.doi.org/

As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way.

== Details ==

* Probably implement as extension rather than core"	1319977305000000	1319977305000000
1446	enhancement	rgrp	rgrp	ckan-sprint-2012-01-09	closed	fixed	Data Explorer v2	"Super ticket: #1602 (Data Previewer v2)

We already have first pass of Data Explorer that was released as part of #1357.

 * Merging https://github.com/okfn/dataexplorer with http://github.com/okfn/recline and developing it
   * https://github.com/okfn/recline/issues

Tickets include (* indicates improvement over current explorer)

 * Core Backbone Models representing Dataset and Tabular data: https://github.com/okfn/recline/issues/10
 * New theme: https://github.com/okfn/recline/issues/22
 * Read-only mode: https://github.com/okfn/recline/issues/17
 * Introduce hash navigation / state support (*): https://github.com/okfn/recline/issues/19
 * Re-enable editing in DataTable (*): https://github.com/okfn/recline/issues/13
 * [super] DataTable view (in Backbone) (*): https://github.com/okfn/recline/issues/14
 * DataExplorer parent view (*): https://github.com/okfn/recline/issues/12
 * Simple graph widget using flot: https://github.com/okfn/recline/issues/11

Est: 10-15d (should be broken down -- partly is in recline issues)"	1320665596000000	1326281658000000
1458	enhancement	rgrp	rgrp	ckan-future	assigned		Support previewing kml files in data viewer	"Super ticket: #1151 (viewing geo data)

E.g. preview of http://thedatahub.org/dataset/louisvillecrime should bring up a map"	1320936488000000	1340632932000000
1459	enhancement	rgrp	rgrp	ckan-backlog	new		Featured Dataset feature	"Provide way to mark a dataset as featured. Featured database show up on the front page.

TODO: detail this more."	1321113012000000	1321113012000000
1472	enhancement	rgrp	rgrp	ckan-sprint-2011-11-21	closed	fixed	Convert ckanclient to git and github from mercurial		1321624905000000	1321721874000000
1518	defect	rgrp	markbrough	ckan-sprint-2011-12-19	closed	fixed	CKAN Upload fails if filename has spaces in it	"E.g. uploading a file with spaces in it:
OECD Monthly Exchange Rates.zip

Gives a 404 Not Found response to the following file:
http://test.ckan.org/api/storage/metadata/2011-12-05T193046/OECD%20Monthly%20Exchange%20Rates.zip

The upload wheel keeps spinning and the user is not informed that the upload has failed.

Analysis: turns out that google storage (possibly s3 but not checked) replaces with ' ' in keys with '+' on upload. This breaks things because we try and look up metadata about upload using the filename/key we put in but of course that does not exist because google has changed name.

Fix is trivial: replace ' ' in keys / filenames with '-'.
"	1323114236000000	1330020742000000
1533	defect	rgrp	amercader	ckan-backlog	closed	wontfix	Make sure ckanext-wordpresser works on CKAN 1.5.1	"Superceded by #2234

Super ticket: #1508

Etherpad: http://ckan.okfnpad.org/wordpresser

The wordpresser extension does not work with ckan 1.5.1. The wordpress page is not loaded.

Looks like this changeset may be involved
https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1

I think that even before that changeset, the page was shown without any margins.

In any case, the extension seems to be too sensible to changes in the ckan layout so perhaps it should take into account the CKAN version and have a fixed set of xpaths and templates that work for each version

"	1323277660000000	1338202676000000
1559	enhancement	rgrp	jilly mathews	ckan-sprint-2012-04-02	closed	fixed	Comments Extension / Disqus	Polish off comments extension dev and test. estimate 2 days.	1324291720000000	1332242129000000
1571	enhancement	rgrp	jilly mathews	ckan-backlog	assigned		[super] Issues Extension	"""Proper"" issues extension.

 * Etherpad: http://ckan.okfnpad.org/issues
 * Code: https://github.com/okfn/ckanext-issues

Overview

 * This would be an extension and improvement of existing todo extension - #842 (https://github.com/okfn/ckanext-issues)
  * Current extension does work but lacks polish
  * Should we polish or do significant improvements?
 * Functionality - see etherpad

== Steps ==

 * ~~(Rename todo(s) -> issue(s) in existing extension)~~"	1324294225000000	1330863183000000
1576	enhancement	rgrp	rgrp	ckan-sprint-2012-01-09	closed	fixed	Move stats extension back into core - 0.5d	" * Move into core (keeping as an extension) and default enable
 * Do we need tests for controller? (No: already there)
 * Documentation (?) - e.g. of basic config options
 * Remove lib/stats.py which is still in core
  * dread moved it to extension in https://github.com/okfn/ckanext-stats/commit/293b2544737d8ab7610884bd5e999ee37f1ab813 but did not remove from core
 * Deprecation notice on ckanext-stats repo and update extension registry

Est: 0.5d.

Questions:

 * Why do this?
   * tiny extension with few dependencies - and really nice to have out of the box
   * trial for doing this on larger scale
 * Do we keep as extension (even if in core)?
  * Ans: Yes, keep as extension because:
   * Already set up that way
   * Cleaner
   * Easier to disable / enable 
 * How do we integrate with main them (e.g. have stats link)
  * Ans: not sure (this is part of more general issue of how we update theme for varying changes elsewhere). Best answer is to have some widgetization in theme.
 * Hide ratings section (at least until we reinstate ratings #1598)
  * Ans: no, let's not bother (and having ratings there encourages us to do #1598 and/or find out whether people are interested in ratings)"	1324317313000000	1327056070000000
1577	defect	rgrp	dread	ckan-backlog	new		Can't upload file with foreign chars in filename	"Looks like uploading a file with foreign characters fails due to encoding reasons.
{{{
URL: http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:123 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckanext.storage.controller:2 in auth_form
Module ckan.lib.jsonp:26 in jsonpify
<<      Very much modelled after pylons.decorators.jsonify .
           """"""
           data = func(*args, **kwargs)
           return to_jsonp(data)
>>  data = func(*args, **kwargs)
Module ckanext.storage.controller:301 in auth_form
<<          method = 'POST'
               authorize(method, bucket, label, c.userobj, self.ofs)
               data = self._get_form_data(label)
               return data
>>  authorize(method, bucket, label, c.userobj, self.ofs)
Module ckanext.storage.controller:79 in authorize
<<      if method != 'GET':
               # do not allow overwriting
               if ofs.exists(bucket, key):
                   abort(409)
               # now check user stuff
>>  if ofs.exists(bucket, key):
Module ofs.remote.botostore:53 in exists
<<          if bucket is None: 
                   return False
               return (label is None) or (label in bucket)
           
           def claim_bucket(self, bucket):
>>  return (label is None) or (label in bucket)
Module boto.s3.bucket:87 in __contains__
<<      def __contains__(self, key_name):
              return not (self.get_key(key_name) is None)
       
           def startElement(self, name, attrs, connection):
>>  return not (self.get_key(key_name) is None)
Module boto.s3.bucket:144 in get_key
<<          response = self.connection.make_request('HEAD', self.name, key_name,
                                                       headers=headers,
                                                       query_args=query_args)
               # Allow any success status (2xx) - for example this lets us
               # support Range gets, which return status 206:
>>  query_args=query_args)
Module boto.s3.connection:388 in make_request
<<          if isinstance(key, Key):
                   key = key.name
               path = self.calling_format.build_path_base(bucket, key)
               boto.log.debug('path=%s' % path)
               auth_path = self.calling_format.build_auth_path(bucket, key)
>>  path = self.calling_format.build_path_base(bucket, key)
Module boto.s3.connection:88 in build_path_base
<<      def build_path_base(self, bucket, key=''):
               return '/%s' % urllib.quote(key)
       
       class SubdomainCallingFormat(_CallingFormat):
>>  return '/%s' % urllib.quote(key)
Module urllib:1222 in quote
<<              safe_map[c] = (c in safe) and c or ('%%%02X' % i)
               _safemaps[cachekey] = safe_map
           res = map(safe_map.__getitem__, s)
           return ''.join(res)
>>  res = map(safe_map.__getitem__, s)
KeyError: u'\xed'
CGI Variables
AUTH_TYPE	'cookie'
CONTENT_TYPE	'; charset=utf-8'
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING	'gzip,deflate,sdch'
HTTP_ACCEPT_LANGUAGE	'en-US,en;q=0.8'
HTTP_CACHE_CONTROL	'max-age=259200'
HTTP_CONNECTION	'keep-alive'
HTTP_COOKIE	'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; ckan_user=elenaibp; ckan_display_name=""Elena Mondo""; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; __utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10.1324298316; __utmc=119670349; __utmz=119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)'
HTTP_HOST	'thedatahub.org'
HTTP_REFERER	'http://thedatahub.org/dataset/edit/budget-library-czeck-republic'
HTTP_USER_AGENT	'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7'
HTTP_VIA	'1.1 localhost (squid/3.0.STABLE19)'
HTTP_X_FORWARDED_FOR	'87.114.74.190'
HTTP_X_REQUESTED_WITH	'XMLHttpRequest'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf'
REMOTE_ADDR	'193.34.146.142'
REMOTE_PORT	'55419'
REMOTE_USER	u'elenaibp'
REMOTE_USER_DATA	'userid_type:unicode'
REMOTE_USER_TOKENS	['']
REQUEST_METHOD	'GET'
REQUEST_URI	'/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf'
SCRIPT_URL	'/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf'
SERVER_ADDR	'193.34.146.146'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'thedatahub.org'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.0'
SERVER_SIGNATURE	'<address>Apache/2.2.14 (Ubuntu) Server at thedatahub.org Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.14 (Ubuntu)'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x7f22601c7dd0>
beaker.cache	<beaker.cache.CacheManager object at 0x7f22601c7b50>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f22601c7a90>>
beaker.session	{'_accessed_time': 1324298703.071357, '_creation_time': 1324293077.4139669}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	'ckan.net'
mod_wsgi.reload_mechanism	'1'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 8)
paste.cookies	(<SimpleCookie: __utma='119670349.1809834699.1323782464.1324293066.1324298316.4' __utmb='119670349.3.10.1324298316' __utmc='119670349' __utmz='119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode' ckan_apikey='decd48b1-49ee-4250-bff4-98ccca9c02a5' ckan_display_name='Elena Mondo' ckan_user='elenaibp' hide_welcome_message='1' thedatahub_net='27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e'>, 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; ckan_user=elenaibp; ckan_display_name=""Elena Mondo""; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; _ _utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10...)|utmcmd=(none)')
paste.registry	<paste.registry.Registry object at 0x7f226194df50>
paste.throw_errors	True
pylons.action_method	<bound method StorageAPIController.auth_form of <ckanext.storage.controller.StorageAPIController object at 0x7f2261dad990>>
pylons.controller	<ckanext.storage.controller.StorageAPIController object at 0x7f2261dad990>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x7f2261daddd0>
pylons.routes_dict	{'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'}
repoze.who.identity	<repoze.who identity (hidden, dict-like) at 139785645747120>
repoze.who.logger	<logging.Logger instance at 0x7f225e23c098>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 139785625065680>, 'friendlyform': <FriendlyFormPlugin 139785618095248>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f2260874c10>, 'auth_tkt': <AuthTktCookiePlugin 139785625065808>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f2260874c90>}
routes.route	<routes.route.Route object at 0x7f22601a1090>
routes.url	<routes.util.URLGenerator object at 0x7f2261dadf50>
webob._parsed_query_vars	(GET([]), '')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multiprocess'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f2261da9af8>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0x7f2261dadf50>, {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'})
}}}"	1324317659000000	1325473564000000
1599	enhancement	rgrp	rgrp	ckan-sprint-2012-01-09	closed	fixed	[contrib] Simple embeddable dataset count widget (esp for group count)	"Simple embeddable widget for use on 3rd party sites showing dataset counts for a given search query. Have a specific version just for groups.

 * Simple group count widget in JS for embedding in wordpress and elsewhere
  * Requested by several people (e.g Guo Xu from Econ working group). Already have something like this in CKAN JS for doing an embeddable search box.
 * All you need to do is do a dataset query over the API e.g. http://thedatahub.org/api/search/dataset?groups=economics and then embed in some html! 

Estimate: 30m (for someone who knows their jquery).
"	1325246358000000	1325555201000000
1600	enhancement	rgrp	rgrp	ckan-v1.7	closed	wontfix	[super] User guide with integrated help in the WUI	"Provide help to users in the WUI, with this linking to a detailed user guide.

 * User guide (suggest at help.thedatahub.org (or possibly help.ckan.org)
 * Snippets in the WUI with links to full user guide.
  * Provide helper methods for creating these links with config options for setting path to user guide so that the link location can vary on install by install basis
 * General info about publishing/working with/etc data (where we don't duplicate elsewhere)

Est: (?) 5-10d

== Topics ==

See http://ckan.okfnpad.org/datahub-help

 * Step-by-step for adding a dataset including tips
  * Reuse: http://datapatterns.org/datahub.html
  * Quick overview of relevant parts of domain model (dataset and resource)
 * API (perhaps link to main docs)
 * Command line interface and dpm
 * Using groups

== Why do this ==

We already have wiki.ckan.org and docs.ckan.org. Why do this?

Answer for docs:

 * This is administrative guide and focused on general CKAN software.
 * Hard to provide completely generic help.
 * Integratable with the DataHub site

Answer for wiki:
 * No spam
 * General feeling that wikis are a little bit loose and untidy. We want this to be smart, clean and useful.
 * Integratable with the DataHub



"	1325257824000000	1333362051000000
1602	enhancement	rgrp	rgrp	ckan-v1.6	closed	fixed	[super] Data Previewer / Viewer v2	"We already have first pass of Data Preview that was done as part of #1357. This is about v2 of this. Two main components:

 * A new version of the separate Data Explorer library (now called Recline Data Explorer) - #1446 '''Est: 10-15d'''
  * Why:
   * Cleaner, backbone based solution.
   * Provides editing support if wanted.
   * Collaboration with Max Ogden
 * Integrate this into CKAN (into resource view page) - see #1630 '''Est: 1d'''
 * Improvements to Data Viewer / Previewer system in CKAN itself - '''Est: 2-3d'''
  * ~~(Update Data Preview-er (View-er) to support images ...)~~
  * ~~(WONTFIX here - (?) Update Data Preview-er to support geodata - #1151)~~
   
"	1325267544000000	1328175719000000
1630	enhancement	rgrp	rgrp	ckan-sprint-2012-02-06	closed	fixed	Integrate Data Explorer v2 into Data Viewer	"Super ticket: #1602

Steps:
 * Include (or git submodule) data explorer into vendor
 * Update existing js integration code
 * Fix css and any bugs

Est: 1d

Actual: 1d"	1326279098000000	1330085600000000
1660	defect	rgrp	lucychambers	ckan-sprint-2012-02-06	closed	wontfix	CSV preview broken - OpenSpending	"This CSV resource used to preview but now the format appears to be unsupported: ""We are unable to preview this type of resource: x-osdata-csv""

http://thedatahub.org/dataset/lbhf-spending-2010/resource/9661abbd-2816-4d58-8b20-3cb0eb770c69

This is used as an example by the OpenSpending team all the time."	1326717846000000	1328013627000000
1691	enhancement	rgrp	rgrp	ckan-sprint-2012-01-23	closed	invalid	paster user create command takes password on command line	"Needed to support automated deployment more easily.

Est: 15m"	1327077314000000	1327314081000000
1719	defect	rgrp	dread	ckan-sprint-2012-02-06	closed	fixed	Broken links for non-Gravatar use icons - 0.25d	"Super ticket: #1506

e.g. http://thedatahub.org/user The users with Gravatar have their nice user icons, but the majority of users (without Gravatars) have 'broken link' symbols. Same problem wherever users are shown, such as the dataset history pages.

Broken on 1.5.2b - currently on thedatahub.org."	1327938788000000	1328541630000000
1797	enhancement	rgrp	rgrp	ckan-sprint-2012-03-05	closed	fixed	[super] Webstore v2 - Integrated and based on ElasticSearch	"Main tasks:

 * Integrated seamlessly into CKAN (authorization etc)
 * Based off ElasticSearch

Etherpad: http://ckan.okfnpad.org/webstore

= Tasks =

 * ~~New API point (config + controller) [0.75d]~~
   * /api/data/{resource-id}
 * ~~Authorization [0.5d]~~
 * ~~Documentation [0.25d]~~
 * ~~Upgrade data viewer to use ES [0.5d]~~
 * ~~Convert webstorer to new setup - http://github.com/okfn/ckanext-webstorer [1d] - #1753~~
  * See below
 * ""Create DataStore Resource"" in Resource creation (??)
  * Just creates a resource with blank url and datastore enabled (webstore_url non null)
 * ~~Tick box to enable DataStore in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)~~
  * Also need to use this info in DataStore extension?
 * ~~Create nice instructions on Resource page about using the DataStore~~
 * ~~Rename webstore -> datastore~~
 * ~~Deploy [0.5d]~~

= Details =

== Layout on ES ==

{{{
elasticsearch/{index}/{type-id}

index = ckan-{ckan-id}
type-id = resource-id (1)
}}}

== Request path ==

Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results

See as an example: https://github.com/okfn/elastic-proxy

== Questions ==

1. How do we decide what to use in the viewer e.g. Webstore data versus DataProxy
    * Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't.
2. How do we decide when webstore is activated
    * Ans: see previous.
3. Do we (and how do we) allow for resources which are just for webstores
    * Ans: we have to set webstore_url = True ... on save. Or add a big button saying: ""Create Blank Resource"" ...
4. How does webstorer work ...
    * CSV import - NS: make sure we add e.g. rownum or __id__ = rownum ...
    * Separate mini-service (webstorer.thedatahub.org ...)
5. How do we selectively disable recline where not appropriate (e.g. query support makes no sense for DataProxy)
    * Ans: do not bother at present. Focus on Webstore route.
6. Where does geodata support come in (in Recline or separately)
    * This is a separate issue

== User Story 1 ==

User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save

=> Result = New Dataset + Resource + Data API to resource and nice Preview of the data

Solution:

 1. Allow normal upload process
 2. Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database.
 3. Update the resource in some way (set webstore_url)

= User Stories =

== User Story 2 ==

User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise)

=> Result = New Dataset + Resource + nice Preview of the data (wherever possible)


== User Story 3 ==

User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store.

=> Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user

Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...)"	1329306185000000	1331412644000000
1828	enhancement	rgrp	rgrp	ckan-sprint-2012-03-05	closed	fixed	Upgrade to Recline v0.3	Recline v0.3 has introduced some breaking changes in API and usage. Upgrade to use it.	1329943304000000	1330525412000000
2221	enhancement	rgrp	zephod		closed	fixed	Tests are broken for some of us: NotAPairTreeStoreException	"Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here:
https://gist.github.com/2007985

"	1331318781000000	1331561095000000
2222	enhancement	rgrp	zephod		closed	duplicate	Tests are broken for some of us: NotAPairTreeStoreException	"Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here:
https://gist.github.com/2007985

"	1331318783000000	1331318825000000
2224	enhancement	rgrp	rgrp	ckan-sprint-2012-04-02	closed	wontfix	Simplify javascript and css dependencies and add minified version	" * Get rid of jquery UI
  * Requires switching to bootstrap lookahead plus a small bit of coding for ajax part (see https://gist.github.com/2007241)
 * Remove modernizr (what do we use it for?)
 * Convert jquery.tmpl to jquery mustache (?)
 * Remove ckanjs moving core parts of it directly into our code.

Compress everything for releases and add minified version to build (with switch to enable use of this?)"	1331385623000000	1338205358000000
2226	enhancement	rgrp	rgrp	ckan-sprint-2012-03-19	closed	fixed	[super] Refactor and improve documentation (v1.7)	"Documentation is key to the success of projects!

= Move material into main docs from wiki =

 * http://wiki.ckan.org/Writing_asynchronous_tasks
 * http://wiki.ckan.org/Domain_Model
 * http://wiki.ckan.org/Deployment

= Create a User Guide =

Basic step-by-step on getting started with CKAN (See start on set of slides here e.g. http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/)

 * Publishing data
 * Authorization and workflows
  * Publisher Profile (Workflow)

= Break into sections =

Suggest something like:

{{{
# General
* Intro
* Conceptual Overview
* What CKAN helps you do (http://ckan.org/)
* FAQ ? (or point to the wiki)

# For Administrators

* Installation and Setup
  * Configuration options
* Customization - Theming etc
* Authorization and Workflows
* Storage

# For Users (Publishers, Data Wranglers, etc etc)
* Walkthrough of publishing a dataset
* Storage
* API (see Developers section)

# CKAN Developers

* Domain Model
* API
* Extensions
* i18n

CKAN Developers
* Buildbot
}}}"	1331481965000000	1332475494000000
2236	enhancement	rgrp	rgrp	ckan-sprint-2012-03-19	closed	fixed	Updates to Recline and integrate them	" * (Recline) Better graphing (multiple graph types etc)
 * (Recline) Bootstrapify
 * Integrate"	1331910381000000	1332153688000000
2278	defect	rgrp	rgrp	ckan-sprint-2012-04-16	closed	fixed	DataStore enabled is disabled by default but is needed for DataStorer to run	"Major UX gotcha reported by several people is that DataStore is disabled by default and must be enabled for DataStorer to run and import data.

Suggest switch to enable datastore by default for file and file.upload types (disable for API). People can always disable if they want."	1333441048000000	1333441177000000
2279	enhancement	rgrp	rgrp	ckan-v1.8	new		Write tutorial for our DataStore API / Data Explorer	"Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this.

This is documentation / example only (no code!)"	1333482263000000	1338205426000000
2292	defect	rgrp	rgrp	ckan-sprint-2012-04-30	closed	fixed	Group edit form no longer allows for adding of multiple datasets at once	"Can only add one dataset at a time now (not sure when this broke but was working ~2-3m ago).

Assigning to seanh for review and estimation. I'd rate this as quite annoying but not absolutely critical (hence assigned priority)."	1334431388000000	1335021876000000
2874	enhancement	rgrp	rgrp		assigned		Clean up bin directory	Full of obsolete material	1345190508000000	1345190515000000
142	defect	pudo	dread		closed	fixed	Update User object with additional fields such as 'nickname'	"Suggested fields:

  * id, name (username/profile name), openid, email, password, extras (JsonType)
    * + current items: apikey, about, created)
    * current contents of name is usually openid and should migrate to openid field (could match based on starting with http://)
  * May also want a fullname field (or perhaps that can
just go in extras - may be better to have its own field as useful to search on this )

Related changes (probably separate tickets):
  * Improve user home page to include this additional information (and have it be editable)
  * Use user 'name' (username) for home page links (rather than current user/<id>)
  * Where we show lists of revisions, each revision author is now a link to the author's user page.

Related to ticket:136 and ticket:138."	1255010314000000	1289219098000000
146	defect	pudo	rgrp		closed	worksforme	Problem when an admin user logs in between commencing editing and saving	"  1. Visitor starts editing a package (or creates a new package)
  2. Visitor logs in before previewing or committing
  3. Visitor is an admin (or sysadmin)
  4. Preview or commit fails

Reason (conjecture): the form we use for package editing is now dependent on whether you have admin privileges (we allow editing of state by admins). When you login the old form is now incorrect and this results in the formalchemy fieldset breaking."	1255013773000000	1291829862000000
190	defect	pudo	dread		closed	fixed	Package comments	"Cost 7 days

When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API. 

The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user.

Example at bottom of package page:

'''Leave a comment:'''
||Subject||  _________  ||
||Comment||  _________  || 
[[br]]
||Submit button||

'''Comments:'''
||'''Explanation doc'''||
||''Posted on 25h May 2009 by !http://bertdavies.myopenid.com''||
||It says on the pollution data web page that not all the stations have a CO2 sensor, so you have to extrapolate from the ones that do. See my visualisation of CO2 across London for an idea of what you can do: bertdavies.com/pollution-2008.jpg||

||'''More info'''||
||''Posted on 24th May 2009 by !http://ronsmith.myopenid.com''||
||Excellent data, but why is there no value in the CO2 column for some of the testing stations?||

= Implementation details: = 
Comments table is with columns:
||id || package_id || date (date) || comment (multi-line text)||
"	1258387105000000	1280820852000000
231	defect	pudo	rgrp	ckan-v1.3	closed	fixed	Make group versioned	"Make group object revisioned/versioned.

  * Involves upgrade script both for group object and group_revision object
  * No changes required to WUI for present"	1262853923000000	1288516929000000
242	defect	pudo	rgrp	v1.1	closed	fixed	Miscellaneous tidying up (v0.11)	"Dumping ground for various minor and miscellaneous items (mainly refactorings):rgrp

  * controllers/tag.py: why does this not use lib/search.py (but controllers/packages.py search method does)?

done:
  * lib/search.py: inheritance would be nicer than switching on entity in search (i.e. have SearchPackage, SearchTag etc)"	1265715103000000	1280823876000000
273	task	pudo	rgrp	v1.1	closed	fixed	Investigate search index options and create tickets	"Work already here: http://knowledgeforge.net/ckan/trac/wiki/SearchEngine

Add knowledge there or to:

http://wiki.okfn.org/SoftwareTools/Search
"	1268996933000000	1278578527000000
309	defect	pudo	dread		closed	fixed	Tag list shows unused tags	"Deleted tags are visible on the /tag page and are searchable.

Delete tags are ones that were in use once and then discarded, so this includes ones which are not used on any packages any more, including misspelling, mistakes, and deprecated tags. It would makes sense to remove them.

Example tag that is viewed at /tag and searchable:
http://ckan.net/tag/2rand[0"	1273247648000000	1280743432000000
317	enhancement	pudo	rgrp	v1.1	closed	fixed	Make search pluggable	"Make lib/search.py pluggable so that we can plug in different search systems (e.g. SOLR).

Suggest we define a base Search class from which specific search implementations inherit (e.g. SQLSearch, SOLRSearch etc). The specific one being used would then be set via a config variable."	1274295418000000	1279286041000000
327	defect	pudo	pudo		closed	wontfix	Create a web hook worker for CKAN	" This will be useful as an example consumer of the new queue notifications (#325). 

 * A user can register any URL to be notified upon an event.  
 * Possible extensions: GET with ID only vs. POST with serialized object
 * E-Mail notifications are also hooks, essentially. 

We need to have a UI module that allows for the hook CRUD and some feedback (e.g. non 200 status codes)"	1274807361000000	1296467361000000
332	defect	pudo	rgrp		closed	fixed	Autocomplete tag on click	"During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box.
"	1275302887000000	1280743320000000
334	defect	pudo	dread		closed	fixed	Wrong link for package feed icon	"This is a problem with the package page e.g. http://ckan.net/package/open-election-data-project . The feed icon links to [1]  and the text next to it ""Subscribe"" links to [2]. Surely these should be the same? The second link seems to right one to me.

 * [1] http://ckan.net/package/open-election-data-project?format=atom
 * [2] http://ckan.net/package/history/open-election-data-project?format=atom&days=7"	1275407445000000	1280743667000000
359	requirement	pudo	rgrp	datapkg-0.7	closed	fixed	Create Storage with pluggable extensions	"Storage represents something you can upload to (download is done in ticket:406).

  * storage sub package with defined interface (maybe just use OFS)
  * Only support uploading single files for the moment (though can do multiple files).
    * Relationship to build (ticket:306)
    * No support for syncing e.g. hg/git repos -- that's up to you as a user
  * Upload command

At end of this we can deprecate Repository.

Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication."	1277750654000000	1291135692000000
366	enhancement	pudo	dread	ckan-v1.4	closed	fixed	A flash message says why you are redirected to the 'user login' page	"You get redirected for a number of different reasons - often confuses me let alone the average user!

Cost: 1h"	1279128058000000	1300212171000000
401	task	pudo	pudo		closed	fixed	Provide base worker as queue consumer	We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app)	1281018724000000	1286189243000000
402	task	pudo	pudo	ckan-v1.3	closed	duplicate	Archiving worker to back up package resources from a CKAN instance	"Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

  * Naming scheme?
    * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
      * What happens if names change
    * File: filename? hash?
  * Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages. 

Extra points for: 

 * Properly checking for source file modification (Last-modified, Etag)  
 * Using PIP VCS Backends for retrieval
 * OFS/S3 Storage "	1281018912000000	1296467635000000
406	requirement	pudo	rgrp	datapkg-0.8	closed	fixed	Download sub-system	"Factor out current downloader into dedicated download subpackage, abstract interface and make it pluggable so we can have downloaders loaded as per type of resource to download.

  * Also rename InstallCommand -> DownloadCommand

And we should support interactive retrieving of resources ...

== Implementation Ideas ==

{{{
for resources in pkg.resources:
    # datapkg/hg 
    load_storage_handler(resource.format)
}}}
"	1281346822000000	1296467427000000
408	requirement	pudo	pudo	ckan-v1.2	closed	fixed	Integrate CMS via proxying	"* As sysadmin/user of CKAN I want to integrate a CMS to support easy addition of additional content ...
* I want to integrate with navigation 
* As a minimum I want to be able to add content pages (more stuff would be e.g. comments, sidebars, main menu ...)

Why: CMS are great at creating content and have very good editors and theme support ...

=== Previous work

http://groups.google.com/group/pylons-discuss/browse_thread/thread/f8969e28b0a007c7
* deliverance: retheming on the fly 

http://knowledgeforge.net/shakespeare/hg/file/677eef9402e6/shakespeare/controllers/template.py

1. Rewrite urls for proxying
2. You need a theme to substitute into
3. pylons

Extreme version:
* 2 sources -> pylons/ckan app, cms
* 1 theme source: ckan or cms

1 frontend/proxy app which re-themes everything

Owned by CKAN:

* package/
* tag/
* group/
* api/
* /

Everything else proxied.
"	1281348581000000	1281609572000000
410	requirement	pudo	pudo	ckan-v1.2	closed	fixed	Latest comments widget on home page	This would be an extension to the disqus plugin that would patch itself into home/index. 	1281348733000000	1281619720000000
411	requirement	pudo	pudo	ckan-v1.2	closed	fixed	Deploy and test Varnish on eu3	"* Need to figure out cache invalidation for CKAN
* Evaluate against an internal cache (i.e. memcached) "	1281349275000000	1287393033000000
431	task	pudo	pudo		closed	wontfix	Solr index testing tool	There seem to be a few conditions under which either queue processing or the indexer fail in their current state. To get a more systematic picture of these failures, we should have a small testing tool to compare the index to the database of a live CKAN instance. 	1281963532000000	1295260262000000
460	defect	pudo	johnbywater		closed	fixed	State field changed by non-sysadmin	"This package:

http://ckan.net/package/dbtune-audioscrobbler

was:
 1. created by Richard (logged-in)
 2. edited by Richard (logged-in)
(According to the logs, at this point the state was changed from 'active' to 'deleted') -- RP was it set to 'deleted' or just ''?
 3. pudo changed the state back to active

Similarly an incident with bibbase package where field set to '' (see http://ckan.net/revision/diff/bibbase?diff=702bb0a3-03b7-49ac-87ad-e489c414962f&oldid=5447842d-b6ed-41d9-9cfd-8bb73b85c409)

Need to investigate how this got changed, fix if necessary and report back to Richard. Note that package 'admins' as well as sysadmins can change the state of a package (though note that bibbase did not appear to have an owner).

Suggested solution (for setting to ''):

  * Ensure in ckan/forms.py that there is a validator for state field that ensures only set to valid values.
  * Check that we do not allow state to be changed in the api except by package owner or sysadmin"	1282300002000000	1311176868000000
517	requirement	pudo	pudo	iati-1	closed	fixed	Standard domain model for entries	"cf. Tariqs mail

Updated version: https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&hl=en#gid=0

== Original Version ==

{{{
Publishing Entity:
Publishing Entity Type: (Donor, Recipient, Community Data..)
Donor
Country
Activity period:
Verification status: enumeration of statuses (checked, not checked etc)
Resource links: to the actual IATI record
Number of activities: ...
Date record updated:
Date data updated:
License: Need this field even if it may be a standard license

So naively mashing these together, we get something like:

Identifier
Title
Abstract
Donor
Country
Publisher
Publisher Type
Verification Status
Department
Contact
Contact e-mail
Licence
Resource format
Resource URL
Resource ID
Activity period
Number of activities
Date record updated
Date data updated
}}}"	1282893081000000	1284041561000000
518	requirement	pudo	pudo	iati-1	closed	fixed	IATI WUI Editing	 * Create custom form	1282893167000000	1283896718000000
519	requirement	pudo	pudo	iati-1	closed	fixed	IATI look and feel	 * Use what is on iatiregistry.org at the moment. 	1282893224000000	1283536828000000
520	requirement	pudo	pudo	iati-1	closed	fixed	Browseable web interface onto the data	 * e.g. find/browse by country and by publishing entity and by donor	1282893270000000	1283538080000000
521	requirement	pudo	pudo	iati-1	closed	fixed	IATI Customized search		1282893401000000	1283897124000000
522	requirement	pudo	pudo	iati-2	closed	fixed	Create publishing entity account	" * Publishing entity emails system editors
 * System editor creates publishing entity account and emails publishing entity with details
 * Priority: 4/5"	1282893668000000	1287392999000000
523	requirement	pudo	pudo	iati-1	closed	fixed	Read/write APIs for users and publishing entities	"Read/write APIs for users and publishing entities to access and register data

 * Users have read access
 * Publishing entities have both (though can only write to 'their' records)
 * Priority: 4 for write (read access is 3)"	1282893743000000	1283897688000000
524	requirement	pudo	pudo	iati-2	closed	fixed	Limit registering data to authenticated users either via API or manually.	"Limit registering data to authenticated users either via API or manually.

 * Edits may only be made by authenticated entities
 * Priority: 5"	1282893904000000	1285594971000000
525	requirement	pudo	pudo	iati-2	closed	fixed	Confirm and approve publishing entity account	" 
 * So some kind of confirmation or approval by someone more senior?
 * Alternative: just have an account request.
 * Priority: 3"	1282894040000000	1285595152000000
526	requirement	pudo	pudo		closed	wontfix	Integrate validation tool into the registry	"Assumes 8 is done: 

8. Tool for  validation an IATI XML record (referenced by a URL) for compliance with the IATI specification

 * Outside of scope of registry
 * Multiple validators with different levels of rigour"	1282894300000000	1340626152000000
527	story	pudo	rgrp		closed	fixed	Create domain model and load sample data		1282899025000000	1293097531000000
528	task	pudo	rgrp		closed	fixed	Setup demonstration instance at iati.ckan.net		1282899110000000	1283536475000000
529	task	pudo	rgrp		closed	fixed	Obtain sample metadata		1282899161000000	1283536554000000
530	task	pudo	pudo		closed	fixed	Map IATI attributes to CKAN metadata schema	"From lopad:

{{{
Publishing Entity (often/always? equal to the logged in user)
    * Domain logic: only can add entries for ""yourself""
Donor
Country
Activity period: period to which this record relates
Verification status: enumeration of statuses (checked, not checked etc)
Resource links: to the actual IATI record
Number of activities: ...
Date record updated:
Date data updated: 
License: Need this field even if it may be a standard license
}}}"	1282899187000000	1287392210000000
531	task	pudo	rgrp		closed	fixed	Load sample IATI metadata	"  * Parse data
  * Load via API
  * Develop test for load via API (can be local or remote -- with paste webfixture may be easier)"	1282899244000000	1283536676000000
559	defect	pudo	pudo	ckan-v1.5	closed	invalid	Fix package lookup on group edit form	"Lookup is needed to add a package but won't show up at some times. 

Originally reported by Anja Jentzsch, re questions. "	1283542776000000	1310127694000000
647	requirement	pudo	pudo	ckan-v1.3	closed	fixed	Authorization Groups for group-based access control	allow group-level authorization on all system entities	1284889418000000	1287391981000000
650	story	pudo	pudo	ckan-v1.2	closed	fixed	WUI editing for Authorization Groups	" * Copy & adapt Group controller tests to suit AuthorizationGroups
 * Copy & adapt Group controllers and templates for Authorization Groups"	1284890073000000	1287391932000000
669	task	pudo	pudo		closed	fixed	Refactor Solr to become a generic worker		1286189594000000	1293057161000000
670	task	pudo	pudo		closed	fixed	HowTo: Creating a sysadmin		1286190347000000	1293056844000000
671	task	pudo	pudo		closed	fixed	Implement and show user names in WUI		1286190454000000	1293056852000000
672	enhancement	pudo	pudo	ckan-v1.3-sprint-1	closed	fixed	Facets in CKAN search results	"This has been done in IATI ckan but should be ported over to CKAN main and put into the UI in a nice fashion.

IATI: http://bitbucket.org/okfn/iati/changeset/378431974c76 (related to #520)

Requires SOLR search backend (for postgres should fall back gracefully -- just do not show them)."	1286190736000000	1291715338000000
695	bug	pudo	dread		closed	fixed	Search indexing broken on ckan.net	"e.g. searching for 'buddhist' or 'sanskrit', you don't get this newly created package:
http://ckan.net/package/digitalsanskritbuddhistcanon"	1286991201000000	1287766973000000
697	requirement	pudo	johnbywater	iati-3	closed	fixed	The system shall indicate which package groups can be removed when editing a package	This seems to require doing something else with such a group's 'delete' button, such as showing it greyed, or perhaps not showing it at all.	1287066103000000	1291644457000000
699	defect	pudo	TimDavies		closed	fixed	Paging search results re-sets Open Only and Downloadable Only flags	When getting a search results page (e.g. http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1) and then using the on-page links to page through results, the next link / numbered page links go to http://ckan.net/package/search?q=development&page=2 rather than http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1&page=2 meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings. 	1287143698000000	1296499061000000
700	bug	pudo	dread	iati-3	closed	fixed	Groups in package form	"Editing groups in forms doesn't work for me, with latest code from this morning:

 1. Clean db
 2. paster create-test-data
 3. paster sysadmin create http://davidread.myopenid.com/
 4. paster serve development.ini
 5. In browser, log in to CKAN
 6. Create new package: name=abc group=Roger's books
 7. [Preview] - yes group appears
 8. [Save] - shows package and group hasn't appeared ERROR
 9. Check http://127.0.0.1:5000/api/rest/package/abc and reedit package it also doesn't appear here either.
"	1287394476000000	1290000656000000
701	requirement	pudo	johnbywater	iati-3	closed	fixed	The system shall support changing package groups when editing a package		1287403776000000	1291644605000000
703	story	pudo	johnbywater	iati-3	closed	fixed	Add package to package group using the package form fieldset		1287403920000000	1289909096000000
704	story	pudo	johnbywater	iati-3	closed	fixed	Remove package from package group using the packge form fieldset		1287403961000000	1289910491000000
705	story	pudo	johnbywater	iati-3	closed	fixed	Show package groups on the package read page		1287404004000000	1291644479000000
706	task	pudo	johnbywater	iati-3	closed	fixed	Test for removing package from package group using the package form fieldset		1287404073000000	1289909028000000
707	task	pudo	johnbywater	iati-3	closed	fixed	Test for adding package to package group using the package form fieldset		1287404126000000	1289910513000000
708	task	pudo	johnbywater	iati-3	closed	fixed	Test for showing package groups on package read page		1287404169000000	1289910506000000
716	requirement	pudo	pudo	iati-3	closed	fixed	Visual improvments to IATI CKAN		1287583741000000	1290000673000000
717	bug	pudo	pudo		closed	fixed	"Fix and validate ""setup_default_user_roles"" in IATI"		1287583892000000	1290005099000000
718	story	pudo	pudo	iati-3	closed	fixed	Show recent activities on home page		1287583989000000	1288459823000000
719	story	pudo	pudo	iati-3	closed	fixed	Improve overall layout of IATI	... probably by adapting aidtransparency.net	1287584092000000	1289910497000000
720	story	pudo	pudo	iati-3	closed	wontfix	Include CMS pages in IATI registry	"Perhaps using deliverance?

 * Move About to the right of the nav tabs and make it an about IATI page
 * Make a Help / FAQ section linked of main menu
 * Front page more dynamic - latest data registered and news (microblog thing)
 * make the front page blurb noticable / shorter etc. 
 * link to iatistandard.org from the front page blurb"	1287584189000000	1288459344000000
721	story	pudo	pudo	iati-3	closed	fixed	Make the table preview a link to a new page.		1287584244000000	1288459534000000
722	story	pudo	pudo	iati-3	closed	fixed	Further adapt l10n for IATI	" * Change ""Publishing Entity"" to ""Publisher"""	1287584314000000	1288459703000000
723	story	pudo	pudo	iati-3	closed	fixed	Hide action links pointing to unavailable actions (package create)		1287584448000000	1288459610000000
724	requirement	pudo	pudo	iati-3	closed	fixed	Improve publisher workflow		1287584548000000	1289835202000000
725	story	pudo	pudo	iati-3	closed	fixed	Change to allow anyone (logged in) to create a publisher	"With a pending state set (""unapproved"") "	1287584630000000	1289296038000000
726	story	pudo	pudo	iati-3	closed	fixed	Send message to sysadmins to approve new publishers		1287584670000000	1289835192000000
727	story	pudo	pudo	iati-3	closed	fixed	Sysadmin can approve a publisher		1287584754000000	1289835196000000
775	task	pudo	dread		closed	fixed	Search warning	"We're getting this warning a great deal on live servers. Is these really a sign of the system not operating correctly or can we reduce the level to an INFO?

e.g. on hmg.ckan.net:
{{{
2010-10-29 17:12:08,262 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,333 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,375 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,406 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,480 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,613 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
}}}"	1288372692000000	1295260144000000
782	defect	pudo	rgrp	ckan-v1.3	closed	fixed	Add and remove users from sysadmin role from the command line	"Need to be able to create (and remove) sysadmins easily (used to be able to do this via ini file but this functionality was removed a while back in authz system reorganization it seems).

Suggest creating a new paster command as first step.

{{{
# user identifier = openid (?)
paster sysadmin add {user identifier}
paster sysadmin remove {user identifier}
}}}

Suggest this also be added to main CKAN docs.

This should go into metastable as well as default so that it goes into the v1.2 release (can do this via a transplant from one branch to the other)"	1289205287000000	1289218661000000
808	enhancement	pudo	rgrp	ckan-v1.4-sprint-2	closed	fixed	Allow i18n language choice on site (and possibly set automatically)	"Allow choice of language on ckan instance.

  * Configuration variable listing language choices (could try doing this automatically but recommend against this)
  * Switcher option in UI (perhaps on user profile page)
  * Default it automatically based on browser settings (?)

Think this is high value (and cost is low)

Cost: low (4h)


"	1289944394000000	1297783658000000
810	enhancement	pudo	pudo	ckan-v1.5	closed	wontfix	"Move ""add packages"" field up in group form"	Subticket of forms super ticket #961	1289992287000000	1310128477000000
835	enhancement	pudo	pudo	ckan-v1.3	closed	fixed	Authorization CLI and Documentation	"As a sysadmin I want to take CKAN out of wiki-mode into normal mode. In normal mode visitor and logged in users cannot create or add packages.

Suggest put this as an internal command in authz system and add to cli as paster command."	1291029108000000	1291644820000000
847	enhancement	pudo	rgrp	ckan-v1.3-sprint-2	closed	fixed	CKAN search using SOLR backend	"This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010).

3 key aspects of this:

  * Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN)
    * Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable
    * Asynchroously: via worker with queue
     * #324 Search indexing using notifications
     * #669: Refactor Solr (indexer) to become a generic worker
     * #874: Extract Solr search backend into an extension
  * In core support using SOLR backend as well as postgres for queries
    * #317: Make search pluggable (?)
  * Integrate into WUI (if changes needed)
   * #672 - Facets in CKAN search results

Extras:

  * Solr index testing tool #431"	1291639273000000	1295259902000000
855	defect	pudo	dread		closed	invalid	Authenticator tests	The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests.	1291723967000000	1311176988000000
859	defect	pudo	dread		closed	invalid	Tests for User model	Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement?	1291724348000000	1311177461000000
865	defect	pudo	pudo	ckan-v1.3-sprint-1	closed	fixed	Support external strings in JS	"* Provide and include a ""language.js"" file. "	1291801938000000	1295259773000000
875	enhancement	pudo	rgrp	ckan-v1.4-sprint-1	closed	fixed	Search backend supports solr interface and query API mimics solr	"Consolidate search API interface (and backend) on solr (solrpy) type interface.

 * Support for standard query structure
 * Support for facet options

Do '''not''' need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

 1. Disable (happens automatically)
 2. Implement - suggest using group by etc

== Extras ==

 * Front-page tag cloud: change this to use facets
  * Accept this means that if facets not functional in backend we have no tag cloud
"	1292844957000000	1297085261000000
882	defect	pudo	pudo		closed	wontfix	Update i18n user docs and backport genshi i18n:domain fixes from WDMMG		1293192044000000	1340626094000000
903	defect	pudo	pudo	ckan-v1.3	closed	fixed	Proper handling of deleted packages in Solr serps	Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr. 	1294831054000000	1296588070000000
908	defect	pudo	rgrp	ckan-v1.4-sprint-1	closed	fixed	Issues deploying extensions with modwsgi / uwsgi	"Some extensions import pylons features (such as request) at root.

Have encountered following issues:

1. You need to import ckanext before any ckan stuff in your wsgi script (extension stats and dataapi)

2. Running under wsgi daemon mode (normal mode is fine!) the front page (front page only) will not work (does not occur with disqus or synchronous_search but does with stats and dataapi extensions). You get errors like this in the log:

{{{
[Mon Jan 17 09:01:19 2011] [error] Exception KeyError: KeyError(-1218594160,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Mon Jan 17 09:01:20 2011] [error] Exception AttributeError: ""'NoneType' object has no attribute 'clearing'"" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
}}}
"	1295264624000000	1296730578000000
910	enhancement	pudo	pudo	ckan-v1.5	closed	invalid	Refactor CKANrdf to support extras	Extra fields should be represented in some manner. 	1295266026000000	1310128544000000
911	defect	pudo	pudo	ckan-v1.3-sprint-3	closed	fixed	Add pager to dcat-tools/rdfsolr		1295266211000000	1296467375000000
912	defect	pudo	pudo	ckan-v1.5	closed	wontfix	Rethink result row representation in dcat-tools/rdfsolr	" * Should license go in the bottom line? 
 * Formats should be styled as in CKAN
"	1295266299000000	1306774876000000
913	enhancement	pudo	pudo	ckan-v1.5	closed	wontfix	Handle licenses in dcat-tools	" * Create a list of all valid predicates where licenses are stored.
 * Create a license normalization registry in GSpread
 * Fill spreadsheet from OKD/OSI lists. "	1295266407000000	1306774962000000
914	defect	pudo	pudo	ckan-v1.3-sprint-3	closed	fixed	Handle formats in dcat-tools	" * Create a list of valid predicates for format information (accessURL et al) 
 * Execute #235 via Google Spreadsheets and include in DCat enrichment "	1295266569000000	1297069958000000
915	defect	pudo	pudo	ckan-v1.3-sprint-3	closed	fixed	Deploy dcat-tools/rdfsolr to publicdata.eu as a placeholder	We should present something there, why not start with a meta-search. 	1295267040000000	1296467518000000
935	defect	pudo	pudo		closed	fixed	"""groups"" field bug in IATI"	"An internal server error was reported when trying to create a new package (not when editing):

  WebApp Error: <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) null value in column ""name"" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'} 

The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the ""groups"" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server. 

Possible causes include: 

 * Stale pyc file
 * Threading issue
 * Dependency issue

As the problem is gone for now, it cannot be further investigated atm. 

"	1296302915000000	1306778682000000
938	enhancement	pudo	rgrp	ckan-v1.4-sprint-1	closed	fixed	Message flashing / notification in WUI	"Good user interfaces provide feedback to users. We should provide more feedback about the success / failure of actions using message flashing (see http://flask.pocoo.org/docs/patterns/flashing/).

We already do some of this (slightly poorly), e.g. on adding an authorization the app says: ""X authorization added"".

== Implementation ==

 * Standardize this process and add section to html for this purpose.
 * Already done in other OKFN apps e.g. yourtopia.net, wheredoesmymoneygo and commentonit
 * Testing: suggest keeping this very simple

== Estimate ==

 * Cost: 0.5h (html etc) + 1h to do some integration
"	1296398920000000	1297077023000000
940	defect	pudo	rgrp	ckan-v1.3	closed	wontfix	OpenID login via google for same user generates 2 different openid urls	"This has been noted twice in the last couple of weeks.

Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See:

 * http://groups.google.com/group/google-federated-login-api/web/the-most-important-technical-issue-in-using-the-google-accounts-api?pli=1
 * http://stackoverflow.com/questions/3549953/how-to-get-unique-identifier-from-openid-google

To fix this we should set the openid.realm (not sure how atm).

If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache).

Cost: 1h"	1296401521000000	1297071748000000
942	defect	pudo	pudo	ckan-v1.4-sprint-1	closed	fixed	Fix SERP pager argument passing	"It only passes ""q"", but the whole query must be serialized. "	1296469490000000	1297074807000000
943	enhancement	pudo	pudo	ckan-v1.4-sprint-1	closed	fixed	Create CKAN Wiki	Set up a Mediawiki to contain info on CKAN.net (community) conventions etc. 	1296469547000000	1297076498000000
944	defect	pudo	pudo	iati-4	closed	fixed	Upgrade IATI to CKAN 'default'	" * Move code base, fix any breakage. 
 * Deploy to eu8 
 * Test 
 * Deploy to eu14 

== Why ==

Later CKAN codebase has a bunch of improvements / fixes that it would be good to have.

== Estimate ==

 * 8h

"	1296469688000000	1306774766000000
946	defect	pudo	dread	ckan-v1.3	closed	fixed	No default system rights	"CKAN currently comes out of the box being uneditable. We'd like to encourage people to have open CKANs and therefore by default init the db with this right:
{{{
visitor -> is editor on -> System system
}}}"	1296499922000000	1296833383000000
949	enhancement	pudo	pudo	ckan-v1.4-sprint-1	closed	fixed	Generic username/password login	"based on repoze.who-friendlyForms

"	1296658126000000	1297074827000000
960	defect	pudo	rgrp	ckan-v1.4-sprint-1	closed	fixed	Support non-ascii character in internal error messages	"Just saw an error like the following which looks like it is to do with having a user object with non-ascii characters in it. Either get rid of 'cast' to str type here or do it in a unicode aware way.

Should do this not just here but everywhere we can find in the code base.

{{{
Module ckan.controllers.package:302 in edit
<<          am_authz = self.authorizer.am_authorized(c, model.Action.EDIT, pkg)
               if not am_authz:
                   abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
       
               auth_for_change_state = self.authorizer.am_authorized(c, model.Action.CHANGE_STATE, pkg)
>>  abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)
}}}

== Estimate ==

Cost: 1h"	1297069721000000	1297169056000000
980	defect	pudo	pudo	ckan-v1.4-sprint-1	closed	fixed	Fixes to HR Infoshare	"We've fixed a few things in CKAN that were discovered and tracked by HRI in their own issue tacker. This includes: 

 * add authz checks to package edit links (cset:0752316cd2fe) 
 * replace gettext with _ in controllers, to support unicode error msgs (cset:822340e6077e)
 * handle broken html in notes field without crashing on package read (cset:4b6be037dda0)
 * update i18n (cset:37d57dc3c492, cset:ea03173f5e77, cset:f16f4ee40fe7)"	1297430918000000	1297503177000000
985	enhancement	pudo	pudo	pdeu-1	closed	duplicate	Harvest other data catalogues	" * data.london.gov.uk (1d, ping DERI)
 * catalogues from Paris and Vienna (1d)

Others?
 * digitaliser.dk (1d)
 * data-publica.com (2d)
 * data.suomi.fi (2d)
 * dati.piedmonte.it (1d)
 * portalu.de (2d)"	1297679799000000	1306408134000000
987	defect	pudo	pudo		closed	duplicate	Common harvesting framework	"We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include: 

 * CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
 * Catalogue scraping for LOD2 experiments (yields RDF graphs)
 * Atom/DCat for LOD2 production (yields RDF graphs) 
 * OAI-PMH for http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios. 

In general, such a system would have the following stages: 

 * Source selection: find what to download/scrape/harvest/parse
 * Index retrieval (i.e. package index) 
 * Item retrieval (i.e. package entity)
 * (Optional: Serialization) 
 * Normalisation 
 * Loading/Merging into CKAN

Exisiting harvesters are at: 

 * CSW: https://bitbucket.org/okfn/ckanext-csw/src/
 * Scraper+CKAN: https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/"	1297684756000000	1311177705000000
1007	defect	pudo	pudo		closed	wontfix	Backport WDMMG translations for extensions i18n	"Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class. 

cf. https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16 "	1298630621000000	1340626203000000
1009	enhancement	pudo	rgrp	ckan-backlog	new		Improvements to user accounts sytem	" * Forgot password (email a new password)
 * Confirm email
 * Do not show register page if you are logged in (redirect to home page)
 * ticket:1010 - listing of users.
  * Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)
"	1298635991000000	1310128574000000
1017	defect	pudo	sebbacon		closed	fixed	Problem assigning users to authz groups through web interface	"Against ckan-1.3.1, when I create an authz group called ""administrators"" and visit /authorizationgroup/edit/administrators, I am unable to add more than one user to it.

Each time I add additional users, the existing user on the list is replaced with the new one."	1299071127000000	1299668555000000
1019	enhancement	pudo	pudo	ckan-v1.4-sprint-3	closed	fixed	Webhooks notification service	"We propose the following push approach using individual packages:

1. New, updated and deleted packages are pushed to a url endpoint on Wordpress (WPURL). This endpoint will be a configuration option on the CKAN side. A POST or PUT http method will get used. The payload will be a json document of the following form:

  {  
    payload: The entity data as available from the REST API 
    entity-type: 'Package', 
    operation-type: 'create'|'update'|'delete' # one of these options 
  }

It will be sent as the body of the request with content-type set to application/json.

2. The wordpress side will provide a 200 on success. Any other response will be taken as a failure. On failure, submission will be archived and failure logged and notified to system administrator. Submission can be resent later automatically by sysadmin after review.

3. Pushes will happen continuously and approximately simultaneously with updates (a webhooks type model)

4. [optional] CKAN side will support configuring authorization for basic authentication if applied on WP side.

5. [optional] List queue status (including failures) on ckan adminstrative dashboard.
"	1299166784000000	1299166930000000
1021	enhancement	pudo	pudo	ckan-v1.4-sprint-3	closed	fixed	Config option to disable OpenID	HRI don't like federation, want to login normal way only. Make this a config option and perhaps even mess with runtime repoze config	1299492920000000	1299518828000000
1022	enhancement	pudo	pudo	ckan-v1.4-sprint-3	closed	fixed	Error reporting in CKAN worker API	Report on errors by any worker daemons, send them out via E-Mail	1299493047000000	1299512991000000
1023	defect	pudo	pudo		closed	wontfix	Re-queueing of worker requests with errors	Schedule worker calls to be ran several times, with certain intervals. 	1299493179000000	1340626231000000
1024	enhancement	pudo	pudo	ckan-v1.4-sprint-3	closed	duplicate	Lock down all controller actions in CKAN	When a user visits the site, he/she may only see a login box. 	1299493428000000	1299668648000000
1027	enhancement	pudo	pudo	ckan-v1.4-sprint-3	closed	fixed	Authorization checks on all controller actions	We want to have authz checks on all controller actions so that we can lock down CKAN to a login-only mode. 	1299666256000000	1299682082000000
1044	defect	pudo	dread		closed	fixed	Sysadmins locked-out of API without Right: (visitor, SITE_READ, System)	"The problem is that in ckan/controllers/rest.py the BaseApiController has this method:
{{{
    def __before__(self, action, **env):
        BaseController.__before__(self, action, **env)
        if not self.authorizer.am_authorized(c, model.Action.SITE_READ, model.System):
            abort(401, _('Not authorized to see this page'))
}}}
which works on the basis of your c.user, rather than your apikey. All API users are treated as visitors (since API users don't get a login cookie) and even a sysadmin's apikey is blocked unless there is a right for a Visitor to SITE_READ.

Also needs tests.

(Also, why is this restriction only on the API, package search, group index and tags and agroup index? I'm guessing SITE_READ is only for places where other authz don't apply, but maybe it should not be called 'SITE_READ' but 'OTHER_READ' or something?)"	1300358919000000	1302096155000000
1122	enhancement	pudo	dread		closed	wontfix	JSON Extra data not searchable	"It is possible to use the CKAN API to insert JSON format data into package extra values, but this data is not found on searching.

Full text from Pascal:
{{{
we encountered a Problem concerning accessing Arrays/Lists.

curl -XGET 'http://ckan.net/api/rest/package/hbz_unioncatalog'

will get you amongst others:

 ""extras"": {""publishingInstitution"":
""[u'http://lobid.org/organisation/DE-605',
u'http://lobid.org/organisation/DE-290',
u'http://lobid.org/organisation/DE-38M',
u'http://lobid.org/organisation/DE-98',
u'http://lobid.org/organisation/DE-38',
u'http://lobid.org/organisation/DE-Kn41',
u'http://lobid.org/organisation/DE-82',
u'http://lobid.org/organisation/DE-107',
u'http://lobid.org/organisation/DE-929',
u'http://lobid.org/organisation/DE-Zw1',
u'http://lobid.org/organisation/DE-832']""}

but if I try to query this:

wget
'http://ckan.net/api/search/package?q=lobid&publishingInstitution=""http://lobid.org/organisation/DE-605""'

I get only two packages, among the package ""hbz_unioncatalog"" is
missing. (These two packages have only one value for
""publishingInstitution"").

The ""extra/publishingInstitution""-Array was uploaded through a ""curl
-XPUT ...
 ""extras"": {
   ""publishingInstitution"":[
     ""http://lobid.org/organisation/DE-605"",
     ""http://lobid.org/organisation/DE-290"",
     ""http://lobid.org/organisation/DE-38M"",
     ""http://lobid.org/organisation/DE-98"",
     ""http://lobid.org/organisation/DE-38"",
     ""http://lobid.org/organisation/DE-Kn41"",
     ""http://lobid.org/organisation/DE-82"",
     ""http://lobid.org/organisation/DE-107"",
     ""http://lobid.org/organisation/DE-929"",
     ""http://lobid.org/organisation/DE-Zw1"",
     ""http://lobid.org/organisation/DE-832""
   ]
   },
...
}}}"	1304367510000000	1306747714000000
1155	enhancement	pudo	pudo	pdeu-1	closed	fixed	Harvester for data.london.gov.uk	Write a harvester for data.london.gov.uk to import catalogue metadata into PDEU. API (or at least documentation) is available at: http://sourceforge.net/projects/londondatastore/files/ 	1306337318000000	1306773174000000
1156	enhancement	pudo	pudo	pdeu-1	closed	fixed	Scraping harvesters for Paris and Vienna Catalogues	"Import metadata from both sources into PDEU via the Harvesting framework but by scraping their respective catalogue pages. 

 * http://data.wien.gv.at/katalog/
 * http://opendata.paris.fr/opendata/jsp/site/Portal.jsp"	1306337428000000	1306855111000000
1157	enhancement	pudo	pudo	pdeu-1	closed	fixed	A simple theme for publicdata.eu, CKAN	"Change the visual style of CKAN to be more like these sites: 

 * GitHub
 * Quora 
 * Google Projects

This does not include major UX work. "	1306337573000000	1309804029000000
1158	enhancement	pudo	pudo	pdeu-1	closed	fixed	Put PDEU into read-only mode and remove unneeded functions	"PDEU should be in a mode where: 

 * Users cannot edit or create packages
 * Users cannot sign up on their own
 * AuthorizationGroups and RevisionHistory is completely gone from the UI 
 * Groups have been repurposed as Subjects

"	1306337786000000	1309804060000000
1159	enhancement	pudo	pudo	pdeu-1	closed	fixed	Add RDFa to CKAN package pages	"RDFa can be used as a simple way to expose linked data or at least sameAs the API version of the data but we need to make sure we do not expose anything different from the ""official"" representation."	1306337907000000	1307615133000000
1161	requirement	pudo	amercader	pdeu-1	closed	duplicate	Build a simplified theme for PDEU	"We need to offer a strongly simplified version, read-only of CKAN under publicdata.eu, with a focus on its role as search engine instead of a data catalogue.

This ticket relates to work on the PDEU theme only!"	1306407835000000	1306408026000000
1174	enhancement	pudo	pudo	pdeu-1	closed	wontfix	API Representation Registry	"As CKAN grows, more and more optional representations of packages will become used. Besides RDF (which is the motivation for this ticket), support for DSPL, home-grown XML, or CSV listings is imaginable. 

To properly support this CKAN should have an extensible representation registry that can be extended when new output types become available (and without changing the API in the process). This needs to integrate in two places: 

 * We need to add support for file format extensions to the package handler and REST API. 
 * We need to add HTTP content type negotiation (Accept headers).

To support this we should have a registry with two registers: 

 * to map format extensions to mime types (e.g. ""json"" -> ""application/json"")
 * to map mime types and entity types to converter functions (e.g. 8""application/json"", ckan.model.Package) -> func(obj, mime_type) )

This should be set up on load_environment so that IConfigurer plugins can feed into it. 

The registry should then first be added to _finish in the REST API (which needs to be refactored to be passed the {format} part of the URL if one is given. Based on the format part and HTTP headers, an appropriate representation can be generated by the registry and then be returned to the user. 

Forwarding of requests to the regular WUI controllers with Accept headers set or a format specified can be implemented in a separate effort. "	1307050992000000	1307615200000000
1199	enhancement	pudo	pudo		closed	fixed	Implement an email function for users.	"We need an email function in CKAN to accept messages sent to users. The basic signature will be: 

 * mail_user(user_obj, subject, body, mime_type='text/plain', headers={})

This has a number of use cases: 

 * Retrieval of lost passwords
 * E-Mail confirmation 

Finally, the mail function should be exposed in the API for sysadmin clients. This way we can have scripts traverse CKAN for 404s, invalid data or missing fields and ping users about that automatically (requires traversal by revision, not package, to get the associated users). 

== Implementation ==

Note we have already written code like this (*and* tested it) in isitopen:

 * https://bitbucket.org/okfn/isitopen/src/bb2cbd146fa5/isitopen/lib/mailer.py
 * https://bitbucket.org/okfn/isitopen/src/bb2cbd146fa5/isitopen/tests/lib/test_mailer.py
"	1308821986000000	1310555589000000
1223	enhancement	pudo	pudo		closed	fixed	Caching of static files	StaticURLParser can have caching - use it	1310573854000000	1310573893000000
1246	defect	pudo	adrian.pohl@…	ckan-sprint-2011-10-28	closed	fixed	Search results on ckan.net are mistakenly all 'open'	"All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: http://ckan.net/package
"	1311863353000000	1311892816000000
1249	enhancement	pudo	rgrp	ckan-v1.5	closed	fixed	Exclude script tag from extraction for i18n	"Currently have this script section put in for i18n. It shouldn't be. 

{{{
<script type=""text/javascript"">
//<![CDATA[
(function($){
    $.fn.ajaxCreateSlug = function(name, url) {
        var title = this;
        var updater = {
            init: function(title, name) {
                // Add a new element where the validity of the package name can be displayed
                this.name_field = name;
                this.title_field = title;
                this.name_field.parent().append('<div id=""package_name_valid_msg""></div>');
                this.title_field.blur(this.title_change_handler())
                this.title_field.keyup(this.title_change_handler())
                this.name_field.keyup(this.name_change_handler());
                this.name_field.blur(this.name_blur_handler());
                this.url = url;
            },
....
}}}"	1311958536000000	1311959356000000
1250	enhancement	pudo	rgrp	ckan-v1.5	closed	fixed	Search results should be sorted by score rather than alphabetical	"At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases.

Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard).

Options:
 * Default this in solr (no need to touch code) but fragile and affects everything ...
 * Do it in code and default to score
 * Do it in code and have alphabetical (on name or title?) when no criteria otherwise score

Aside: may also wish to support search in query api but that is for later!"	1311964752000000	1312370201000000
1281	defect	pudo	pudo	ckan-sprint-2011-10-28	closed	fixed	Locale change flash message in wrong language	"When you change the locale, it displays a flash message ""Language has been set to: English"" in the original language, not the one we have just changed to."	1314021656000000	1314021919000000
371	requirement	nils.toedtmann	johnbywater	ckan-v1.4	closed	fixed	The system shall monitor QoS against SLA	Requested by DGU.	1279885622000000	1300217820000000
659	enhancement	nils.toedtmann	dread		closed	fixed	Check CKAN instance works automatically	Auto way to check web and API interface of a CKAN instance basically works. Several gotchas can be quickly determined, such as logging in, search not working. Needs to be configurable per site basis.	1285348333000000	1311183031000000
1305	defect	nils.toedtmann	amercader	ckan-backlog	closed	fixed	SMTP config for thedatahub.org and IATI	"The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one.

Could not send reset link:
SMTPRecipientsRefused({u'amercadero@gmail.com': (550, 'relay not
permitted')},)

As I said, adria.mercader@okfn.org works fine.
The SMTP server used mail.okfn.org"	1314956657000000	1315317033000000
89	enhancement	nickstenning	rgrp		closed	fixed	Tag cloud for package tags	Should not be hard to do (lots of existing libraries) but not sure that this is very important.	1248693939000000	1265892698000000
90	enhancement	nickstenning	rgrp	v0.11	closed	fixed	Link to RDF versions of CKAN data	"We should link to representations on semantic.ckan.net

  * Each package page could link to its RDF representation
    * ""Alternative formats"" near star rating. Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f).
    * Some instances of ckan won't need this, so have option in ini file for this. rdf_store_url=xyz
    * Include machine-readable link to the RDF: 
{{{
<meta http-equiv=... />
}}}
    * 303 redirect to RDF when content negotiation requires RDF. Test:
{{{
curl -I -H ""Accept: application/rdf+xml"" http://localhost:5000/package/warandpeace
}}}

Cost: 3h

== Future ==

  * Search page to link to SPARQL interface: http://semantic.ckan.net/sparql
  * License/dump page could link to full RDF dump.
"	1248943742000000	1265890334000000
206	enhancement	nickstenning	nickstenning	v1.0	closed	fixed	UI review	"Investigate and list (on the wiki at wiki:UiReviewNotes) CKAN's UI issues, in particular:
 
 * Form usability
   * best example: http://ckan.net/package/new
   * ""Extras"" fields
   * Adding to groups (from group page or from package page?)

 * Overview pages: reduction in visual noise and unnecessary elements.
   * best example: package overview page
   * consider openness/downloadability icons
   * display common metadata fields in a more structured fashion?

Est cost: 2 hrs."	1260199941000000	1260881123000000
214	enhancement	nickstenning	dread	v0.11	closed	fixed	UI Review - General template	" * Set hover names on navigation links.
 * Do we really need to display ""About CKAN"" everywhere? Just on home page.
 * Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions.
"	1260879581000000	1263406056000000
216	enhancement	nickstenning	dread	v1.0	closed	fixed	UI Review - Search	" * allow clicking on 'only openly licensed'/'only downloadable' to select/deselect checkboxes: make them <label>s.
 * cut 'only's -- use word 'filter' instead
 * doesn't appear to match substrings!? A search for 'anna' doesn't return match the 'annakarenina' package.
 * autocomplete package names & tags
"	1260879802000000	1273596102000000
217	enhancement	nickstenning	dread	v0.11	closed	fixed	UI Review - Search results	"Package search results page [/package/search?q=something]
 * need clearer delineation between search entry form and results.
 * 0 results needs to be clearer, and the alternative suggestion: 'would you like to create a new package' needs to not look like a search result.
 * ""N tags found"" -- why do I care? this should be presented as ""filter results by tag"" -- very unlikely that I'm looking for a tag: I'm looking for a package, but tags might help me narrow down my search.
 * tags should be displayed more like the way they're displayed when you add them to a package (i.e. blue 'tag' background)
 * filter out particular tags
 * tag cloud?
 * copywriting: ""Packages - Search"" -> ""Search packages""
"	1260879870000000	1263406092000000
218	enhancement	nickstenning	dread	v0.11	closed	fixed	UI Review - Package listing	" * openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS.
 * maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence.
 * results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise.
 * maybe links should go on the full titles? Get rid of package names.
 * Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS?
 * same comments as search listing (ticket:217)
"	1260879903000000	1263406033000000
219	enhancement	nickstenning	dread		closed	wontfix	UI Review - Package	" * Propose three-column package listing: two lists of highest-scorers on some metric and last column is an invitation to register a new package, with a little copy explaining what's involved and why you'd want to.
 * Add in search widget
"	1260879947000000	1294914901000000
220	enhancement	nickstenning	dread	v0.11	closed	fixed	UI Review - Package read	"Package page [/package/mypackagename]
 * Design
  * In general: a huge amount of visual and typographic noise: ""Package:"" is redundant.
  * ""{edit}"", ""{history}"" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is ""make an enquiry with Is It Open?"" bigger than these?
  * Openness/downloadability icons: need help, as discussed earlier
  * ""Rating:"" redundant, as is the second set of stars -- why can't both the display and rate widget
    * Rating - alternative to GET on set_rating.
  * ""Metadata:"" technospeak, probably redundant.
  * <li> bullets add nothing.
  * Title, Version, URL can probably go in one line.
  * Resources can be made more prominent.
  * Author, author email and Maintainer, maintainer email can be merged.
  * Groups, tags in little boxes?
  * Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface?
  * Don't show extras unless there are any.
"	1260880136000000	1265286499000000
222	defect	nickstenning	dread		closed	fixed	UI Review - Package history	"Package history page [/package/history/mypackagename]
  * ""Revisions"" is redundant
  * ""Compare"" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it!
  * Ditto to overall ""Recent Changes"" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links.
"	1260880198000000	1282909280000000
223	enhancement	nickstenning	dread	v1.0	closed	fixed	UI Review - Tag	"Tags landing page [/tag]
 * Both search for tags *and* browse the list of tags right here
 * Big cloud of tags (sorted by popularity/usage?) filterable with a search box.
 * Tags search page [/tag/search] and list [/tag/list] are both redundant and should be merged straight into [/tag].
"	1260880276000000	1273596111000000
224	enhancement	nickstenning	dread	v0.11	closed	fixed	UI Review - Groups	"Groups landing page [/group]
 * Yeesh! What's up with the typography for ""Just login...""?
 * Truncated description along with group name.
 * Explanation text about what groups are for (RHS)
 * Another case of <li> bullets adding nothing.
"	1260880360000000	1267100560000000
226	enhancement	nickstenning	dread	v1.1	closed	fixed	UI Review - History	"Repository History [/revision]
 * What is this page called? ""Recent Changes"", ""Repository History"", ""/revision"": standardise. Between the link in the nav and the page <title> particularly, but the route is also important. Perhaps /changes or something similar?
   * Will change page title on /revision/ to Revision History. Will not change route for the time being.
 * ~~needs copyediting.~~
 * ~~Pagination has similar issues to elsewhere. Also, most obvious here, is the fact that we don't need to display a link to every possible page. Please can we limit it to, say, a dozen nearby pages and an ellipsis.~~
 * ~~Without looking at dates, its not clear whether I'm seeing most recent or oldest changes. Change pagination to say ""Older""/""More recent"" rather than ""Previous""/""Next"".~~ (wontfix: now have text saying we are showing most recent changes)
 * Table layout is pretty ugly (yes, I'm aware this is my fault).
   * wontfix - nothing better at the moment
 * Timestamps are horribly unreadable. At the absolute minimum get rid of the micros. Hover for more detail? 
 * Can we link to an author page? Yes!
 * ~~Atom feed should have a feed icon!~~
 * Why are we adding another <link rel=""alternate""> to this page? We already have one for recent changes on every page in the site and the one we're adding has a less descriptive title. Which is correct? Use that one as a feed for every page.
"	1260880515000000	1279130390000000
227	enhancement	nickstenning	dread	v1.1	closed	fixed	UI Review - General checks	" * There is huge inconsistency in the titles of pages across the site: to give one example ""Edit Package:"" vs ""Package: mypkg [not linked]"" vs ""History - mypkg [linked to package page]""
   * Normalize and use terminology: ""Data Package"" - also change in navbar (but nowhere else for the time being). So hvae Data Package - mypkg, Data Package - mypkg - Edit etc
 * pagination is currently lost in amongst the results list. needs to be *much* more obvious, and should appear top and bottom, or at the very least at the bottom!
"	1260880552000000	1279192621000000
234	enhancement	memespring	rgrp		closed	wontfix	UI Review - Autocomplete package names & tags in search	Broken out from ticket:216	1263056678000000	1338206486000000
447	task	memespring	johnbywater		closed	fixed	Presentation on Package edit form (labels IE7, horiz alignment in chrome)	Follow up with NS and RP.	1282299423000000	1311182895000000
838	enhancement	memespring	memespring	ckan-v1.3-sprint-1	closed	fixed	UI improvements December 2010	"This is a meta ticket for a bunch of UI changes: 

 * #839 Package page redesign
 * #846 Merge css files
 * #850 Show welcome message
 * #863 Prompt users to enter missing info 
 * #864 Search results changes
 * #866 Add download formats to search results"	1291296811000000	1295259773000000
839	enhancement	memespring	anonymous	ckan-v1.3-sprint-1	closed	fixed	Package page redesign	as per http://ckan.org/wiki/UIRedesignPackage	1291299579000000	1292239918000000
846	enhancement	memespring	memespring	ckan-v1.3-sprint-1	closed	fixed	merge style.css and ckan.css	no need for separate files	1291635478000000	1291719074000000
850	enhancement	memespring	memespring	ckan-v1.3-sprint-1	closed	fixed	Show welcome message on first visit to package page	Control using cookies.	1291721923000000	1291721964000000
863	enhancement	memespring	memespring		closed	wontfix	prompt users for missing package information	e.g. license, owner etc	1291729787000000	1338206455000000
864	enhancement	memespring	memespring	ckan-v1.3-sprint-1	closed	fixed	Search results UI changes	as per http://ckan.org/wiki/UIRedesignSearch	1291736441000000	1291741028000000
866	enhancement	memespring	memespring	ckan-v1.3-sprint-1	closed	fixed	Include download formats in search results	as above	1291812213000000	1292240191000000
918	defect	memespring	dread		closed	invalid	Package preview doesn't show tags or groups	"1. Edit any package
2. Hit 'preview'
Notice that the preview at the bottom doesn't show what tags or groups the package is in. 

This is because the new design of the package view only has this info in the sidebar, which is not displayed in the preview.

One might argue that this is ok because the tag is not a central part of the metadata, only used for searching, so it is ok not to preview it. But I wanted to check that this change was a conscious decision."	1295346496000000	1315911507000000
2773	enhancement	markw	markw	ckan-v1.9	new		About page needs improving	The about page for the DataHub (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.	1343646795000000	1343646795000000
1581	enhancement	mark.wainwright@…	johnglover	ckan-future	new		Blog post about Google Analytics extension for CKAN	"The CKAN Google Analytics extension has been updated to work with the latest version of CKAN, could make for a nice blog post.

Can ping John Glover in January for any details required.

Key link is: http://thedatahub.org/analytics/dataset/top though this should probably move to be under stats (e.g. http://thedatahub.org/stats/usage)"	1324402800000000	1325474274000000
1171	enhancement	mark.wainwright	dread	ckan 2.0	assigned		Citation instructions on dataset and resource view pages	"Some sort of citation helper. Something small on the dataset and resource page that would show how to cite.

wwaites: Some related thoughts on this from opb: http://homepages.inf.ed.ac.uk/opb/papers/ssdbm2006.pdf

timclicks: I'm looking at Dataverse for the first time[0]. It seems very popular in the social sciences. I noticed that there is a recommended citation for each dataset. For example, [1] is has this one: ""Targeted Input Programme (TIP) 2000-01"", http://hdl.handle.net/1902.1/SSC-MWI-TIP2000-01-M1 V1 [Version]""

== Implementation ==

Add a small box at bottom of dataset / resource page (or in sidebar on dataset page) with title ""Cite this"" with contents like:

%title. %author. Retrieved %date. %site_title.

For resource: %title = %dataset_title. %resource_name.

Could also add export to ref managers (e.g. to bibtex) but that is for later."	1306920799000000	1347358705000000
2796	enhancement	mark.wainwright	ross		new		Need a datahub one-pager	"A one-pager explaining what the datahub is and with howto/examples for new users. This would make it much easier to explain the value in using the datahub for storing data.

"	1343924916000000	1345129495000000
1111	task	lucychambers	lucychambers		closed	fixed	FAQ - For CKAN	"Write CKAN FAQ (Basis can be: http://wiki.ckan.net/FAQ) 

Post preliminary questions on: 

http://notebook.okfn.org/"	1303906561000000	1305881039000000
1139	enhancement	lucychambers	lucychambers	ckan-backlog	closed	fixed	Create CKAN Theme Gallery	"Take screenshots of existing ckan instances esp those mentioned
<http://wiki.ckan.net/Theming> and put on flickr in ckan or ckan-theme
group so we can create a gallery ... (both to illustrate theming but
also to show ckan instances that are around -- could add to
http://wiki.ckan.net/Instances)"	1305645859000000	1306941356000000
1153	enhancement	lucychambers	lucychambers	ckan-backlog	closed	fixed	Update CKAN wiki front page	"Update CKAN wiki front page - a la OpenSpending: http://wiki.openspending.org/Main_Page

Sections should relate to different types of people using the site: 

Developers, Users etc.."	1306155211000000	1306941386000000
84	enhancement	kindly	rgrp	ckan-future	assigned		Revert support on versioned objects	"Basic revert in the classic wiki form is already support by purging a Revision. However may wish to support:

  1. Cases where multiple objects changed in a revision but only want to revert 1 (low priority)
  2. Want to revert but have reversion as a new revision of that object.

Seems low priority at present.

Cost: ?"	1248339543000000	1340626385000000
103	enhancement	kindly	rgrp	ckan-v1.5	closed	fixed	View a package at a given revision	"As a user I want to view a package at a given revision:

  * When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
  * package history page should provide links to these pages

Cost: 2h"	1251289897000000	1311180850000000
363	defect	kindly	dread	ckan-backlog	closed	wontfix	Blank revisions	"Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they?

They appear on the 'Recently changed' list on the home page with an empty 'Packages' column."	1278947772000000	1310125872000000
515	defect	kindly	dread	ckan-v1.4-sprint-6	closed	fixed	Inconsistent use of 'location' header in API	"When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all.

I've removed the info about the header in the docs in the meantime."	1282757360000000	1302774268000000
560	defect	kindly	rgrp	ckan-v1.4	closed	fixed	Deleted packages showing up in tag lists	"E.g. http://ckan.net/tag/lod contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins).

Also suggested that these are showing up in the API (courtesy of Anja)

Cost: 2h"	1283700001000000	1297084192000000
663	defect	kindly	dread	ckan-v1.4	closed	fixed	Duplicate sibling relationships displayed	"Looking at:
http://ckan.net/package/rkb-explorer-acm

'rkb-explorer-resex' gets display multiple times and shouldn't"	1285576369000000	1298913603000000
664	defect	kindly	pudo	ckan-v1.4	closed	fixed	Sibling relationships shown for deleted packages		1285576600000000	1300371645000000
738	requirement	kindly	johnbywater	ckan-v1.4-sprint-4	closed	fixed	Store revisions of harvested documents		1287775892000000	1299761436000000
826	enhancement	kindly	dread	ckan-v1.4-sprint-1	closed	fixed	Resource 'extra' fields	"There are a number of extra properties we want to store in a resource: 

 * resource size (in bytes, no. triples etc.) - https://trac.dataco.coi.gov.uk/projects/datagov/ticket/673
 * resource hash is 'static' (default) or 'changeable' - https://trac.dataco.coi.gov.uk/projects/datagov/ticket/674
 * alternative link (e.g. TNA Web Archive) to use the link is broken - https://trac.dataco.coi.gov.uk/projects/datagov/ticket/173

i.e. this is a much needed enhancement

Need to be able to read/write these values in web interface and API.

(At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.)

= Implementation =

I suggest these are arbitrary key/value pairs, just like package extras, reusing PackageExtra code as much as possible.
(Other much more radical alternative is to move model to RDF.)"	1290611905000000	1306766057000000
841	enhancement	kindly	dread	ckan-v1.4-sprint-4	closed	duplicate	Caching docs (as a whole)	"Documentation article on caching / improving performance. (To complement configuration docs.)

 * Different sorts of cache - beaker style, etags, package_dict in search results(?)
 * How each one affects performance
 * How to turn them on/off and configure them
 * Is it possible to bypass each of them in the browser or with wget/curl?"	1291308879000000	1300364333000000
890	enhancement	kindly	pudo	ckan-v1.6	closed	invalid	Introduce timed actions into ckanext-queue	"The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows: 

 * routing_key: Package 
 * operation: daily 
 * payload: pkg.as_dict() 

Where operation is one of daily, weekly, monthly or any other interval. "	1294050769000000	1318599247000000
917	enhancement	kindly	kindly		closed	fixed	Turning off Autoflush in SQLAlchemy	"David Raznik: I propose we remove autoflush by default in ckan.  It caused a lot of bugs on the version upgrade due to its magical behavior.   

I have a patch in https://bitbucket.org/kindly/ckan/    on branch   feature_switch_autoflush_off_by_default.

This change simplifies code.  It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet).

Things to know about using flush:

* Most of the time nothing will change.   A commit implies flush.
* You only need to flush if you expect a query to return an object you just saved.    i.e  if you  session.add(obj)  and you expect obj to be in a session.query.
* A flush is handy when you want to get the primary key out before you do a commit.  i.e   session.add(obj);  session.flush();  obj.id  is now generated.
* You can use them as a limited form of nested transaction.

It increased the performance on the tests by about 5%.  There are less flushes altogether so is faster.

I do not see any downsides."	1295279694000000	1295280232000000
922	enhancement	kindly	kindly		closed	duplicate	Make Resources first class entity.	"Why?

 * Will make it easier to give resources their own form, independent of packages.
 * Makes uploading, harvesting and previewing data cleaner as its separate from the packages.
 * Gives the option of sharing a resource between packages.
 * Leads the way to supporting storing richer semantic information against them."	1295713858000000	1320664187000000
945	enhancement	kindly	kindly	ckan-v1.6	closed	fixed	[super] Richer resources - Resource Groups, new fields, improved UI	"Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

 * New resource group table.  #956
 * New kind field in resource.  #957
 * UI for new kind field. #958
 * Resources in REST API ticket:358
 * Resources in WUI #1445
 * Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

 * http://ckan.okfnpad.org/resources - discussion etherpad
  * Contains commented version of James Gardner's main email: http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html
 * UseCasesResources

"	1296475283000000	1325259350000000
954	enhancement	kindly	rgrp	ckan-v1.5	closed	fixed	[super] API version 3	"Child tickets:

 * #1107 Move package autocomplete from package controller and move to API
 * #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

=== A standard package response ===

{{{
{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}
}}}

On error:
{{{
{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}
}}}

=== A search query ===

Based directly on solr.

{{{
{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}
}}}

== Issues ==

This is a breaking change for clients

== References ==

 * http://wiki.apache.org/solr/SolJSON
 * http://www.flickr.com/services/api/response.json.html

"	1296811899000000	1320142744000000
956	enhancement	kindly	kindly	ckan-v1.4-sprint-2	closed	fixed	Add a resource group table.	"We need a way to group resources by various different factors. The most important of which is timeseries.

The model will look like

Package  <o2m>  resource_group <o2m> resource

The use cases to this are outlined in this email. http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html
"	1296821112000000	1299489084000000
957	enhancement	kindly	kindly		closed	wontfix	"new ""kind "" field added to resources"	"The new field was outlined in 

http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

it says

'''Can be ""data"", ""metadata"", ""service"", NULL. There is a need to do this for UKLP where some of the data associated with a   package is metadata, some is data and some could be a service.'''"	1296821398000000	1314218701000000
961	enhancement	kindly	rgrp	ckan-v1.5	closed	fixed	[super] Refactoring of forms, validation and model synchronization	"This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

== The Issue ==

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... !FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

 1. Generating (or just filling) a form template with 'form data' (and errors)
 2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
 3. Converting form data to model data (and validating) (inverse of previous step)

== Related Tickets ==

 * #926 - Pick a simpler form framework
 * #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
  * #1079 Refactor API to use new logic layer and dictization
  * #1078 Refactor WUI controllers and forms to use logic layer
  * cf existing dumper and importer code
  * This will fix #662
 * [not ticketed yet] - validation layer (should work on serialized objects?)
 * #662 - Can't put entity that is returned by posting to package register (Defect)
 * #972 - Merge 'extras' into main package dict rather than having separate key
 * #1035 - Form impressions are given IDs
 * #810 - Move ""add packages"" field up in group form (easier to do this once forms are done)"	1297069849000000	1310126100000000
965	enhancement	kindly	kindly	ckan-v1.4	closed	fixed	make migration testing plugin to nose	"In order to test migrations properly, we need to test the upgraded database against *all* the tests.

To do this a special flag to nosetests should be added. nosetests --migrations

This flag will make sure that the database is recreated from scratch each time using the migrate repository, bypassing init_db."	1297213492000000	1298280637000000
979	enhancement	kindly	dread	ckan-backlog	assigned		Edit Resource extras in the API	Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API. 	1297429777000000	1315222244000000
981	defect	kindly	kindly	ckan-v1.4-sprint-2	closed	fixed	fix authorization_group_user.id so it's added in migrate	This was missed out the migrate script. It should be to be added as a conditional script at the end of the repository so that databases can catch up.	1297434480000000	1298278654000000
984	defect	kindly	kindly	ckan-v1.4-sprint-2	closed	fixed	Fix migration, where upgrade diverges from actual model.	"The migration scripts have not been inline with the upgrade scripts. There are lots of discrepancies.

"	1297627447000000	1298280689000000
989	enhancement	kindly	pudo	ckan-future	new		Extending the model from plugins	"We need to support extending the model from plugins. This could involve: 
 
 * Adding a plugin hook to extend the mapper
 * Adding an upgrade hook for plugin schema migrations
 * Documenting how this is to be done
 * Find a way to avoid conflicts "	1297689724000000	1340034311000000
994	defect	kindly	kindly	ckan-v1.4-sprint-3	closed	fixed	document a new migration methodology	It is too easy to make a mistake with the migrations at the moment.  A more systematic way of doing them is needed and this needs to be documented.	1298281192000000	1298912842000000
995	defect	kindly	kindly		closed	wontfix	consolidate caching	Try and minimise the amount of different caching methodologies, especially for api calls.	1298283499000000	1311179009000000
996	task	kindly	kindly	ckan-v1.4	closed	fixed	Run some basic load testing.	This will involve running a sample of real requests synchronously against real data.	1298283994000000	1300364398000000
997	defect	kindly	kindly		closed	duplicate	change create on cli to upgrade	When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.	1298287981000000	1298288665000000
998	defect	kindly	kindly		closed	fixed	change create on cli to upgrade	When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.	1298287981000000	1300364423000000
1000	enhancement	kindly	rgrp	ckan-v1.4-sprint-3	closed	fixed	Remove call timings code	"Not used and extra overhead. Very useful to strip this out and low cost.

Places where it seems to be:

  * lib/base.py: c.time_call_started
  * ...."	1298489643000000	1298912726000000
1006	enhancement	kindly	rgrp	ckan-v1.4-sprint-4	closed	fixed	Deprecate stable branch	"Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time).

Cost: 10m (giving high priority because of low cost)

(Assigning to dread as he has been managing the stable branch)."	1298624996000000	1300372286000000
1012	defect	kindly	kindly	ckan-v1.4	closed	fixed	Add package revision history to api	Revision history information is not accessible, dgu want this. 	1298887351000000	1301943113000000
1015	defect	kindly	dread	ckan-v1.4-sprint-3	closed	fixed	Editing group gives exception	"Editing a group properties on ckan.net gives an exception when you submit the form. (Is this related to db migration issues?)

Example page: http://ckan.net/group/edit/civil-society

Exception:
{{{
WebApp Error: <class 'sqlalchemy.exc.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint ""group_revision_pkey"" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'description': u""A group for open data related to civil society supported by members of the interest group at: http://okfn.org/groups/civil-society\r\n\r\nCivil society is composed of the totality of voluntary civic and social organizations and institutions that form the basis of a functioning society, as distinct from the force-backed structures of a state (regardless of that state's political system) and commercial institutions of the market. \r\n\r\nhttp://en.wikipedia.org/wiki/Civil_society"", 'created': datetime.datetime(2011, 2, 24, 14, 30, 53, 334842), 'title': u'Civil Society', 'state': u'active', 'continuity_id': u'f4f9f09
}}}"	1298898588000000	1299788821000000
1046	enhancement	kindly	thejimmyg	ckan-v1.4-sprint-5	closed	fixed	Dictization and the new logic layer	"The stages involved with doing this.

 * Convert model objects to standard dict format (DONE)
 * Convert standard dicts to current api formats (DONE)
 * Make standard dicts savable (DONE)
 * Validate standard dict format. (DONE)
 * Authorize actions"	1300364694000000	1302777668000000
1054	defect	kindly	dread	ckan-v1.4-sprint-4	closed	fixed	Ordering of resources	"Changing an old resource and creating a new resource on the end results in the old resource moving to the end in the ordering.

This breaks tests:
 * (ckanext-dgu) ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadBasic.test_fields
 * (ckanext-importlib) ckanext/importlib/tests/test_loader.py:TestLoaderInsertingResources.test_0_reload

You can make a ckan test break with this patch:
{{{
diff -r e6643cf1324c ckan/tests/models/test_resource.py
--- a/ckan/tests/models/test_resource.py        Wed Mar 23 13:25:52 2011 +0000
+++ b/ckan/tests/models/test_resource.py        Wed Mar 23 19:22:35 2011 +0000
@@ -297,6 +297,8 @@
                'url':self.urls[1], 'format':u'OTHER FORMAT',
                'description':self.description, 'hash':self.hash,
                'id':original_res_ids[2]},
+            { #new
+                'url':'new'},
            ]
        pkg.update_resources(res_dicts)
        model.repo.commit_and_remove()
}}}

There seems to be a problem with vdm creating a replacement Resource for the old resource - because it has a duplicate position it is put to the end by the SQLAlchemy ordering_list function."	1300969236000000	1301305615000000
1077	enhancement	kindly	rgrp	ckan-backlog	new		Move to simpler vdm system	"== Option 1: 'Changeset' Model ==

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new ""changeset"" based model for revisioning in vdm. 


=== Implementation ===

 * The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

{{{
Revision -> Changeset

for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))
}}}

Question:
   * does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

== Option 2: Simplify Revision Object Model ==

Just use a simpler vdm, see ticket:1136 (move to !SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

== Discussion ==

Advantage of Option 1 versus 2:

 * Easier support for pending state and similar behaviour
 * No need to introduce new tables (and hence migrations) when making something revisioned (or not).

=== Disadvantages ===

 * Migration is required
 * More difficult to query revision history.
  * Could be addressed by having !ChangeObject have separate cols for table name and id but would likely be more difficult.
 * Performance (?)
  * Have one big !ChangeObject table to query when looking at changed objects rather than many revision tables.
   * Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the !ChangeObject table

== Conclusion ==

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date."	1302304464000000	1340034345000000
1078	enhancement	kindly	rgrp	ckan-v1.5	closed	fixed	Refactors WUI controllers and forms to use logic layer	" * Deserialize forms to new dict format.
 * Replace controllers/forms to use dictization.
"	1302509347000000	1305828973000000
1079	enhancement	kindly	rgrp	ckan-v1.4-sprint-5	closed	fixed	Refactor API to use new logic layer and dictization	" * Convert current api saves to the new standard dict format.
"	1302509530000000	1302777504000000
1092	defect	kindly	kindly	ckan-v1.4-sprint-6	closed	fixed	refactor logic layer to seperate out api, form logic	"The logic layer is a bit too api centric. Make the reusable parts separate in preparation for the wui refactor.

"	1302777929000000	1305570822000000
1095	defect	kindly	kindly		closed	fixed	add way to pass in schema to logic layer.	We need a way to pass in schemas to the logic layer to deal with edge cases. 	1303221854000000	1310134959000000
1109	defect	kindly	kindly		closed	fixed	When extras has a value other than a string an integrity error occurs in the api.	"This is a regression that happened after refactoring the api.

It was shown by 

http://pastebin.com/2v7QasZy"	1303839943000000	1305124697000000
1110	enhancement	kindly	kindly		closed	wontfix	profile ckan	We need to see what areas of ckan are slow.	1303840041000000	1340034394000000
1113	defect	kindly	kindly		closed	fixed	lists in extras serialized wrongly on get with the api.	Lists are being converted to unicode and then translated into a json when getting from the api.	1304017353000000	1304024611000000
1135	enhancement	kindly	rgrp		assigned		Changeset model for vdm	"Move to Changeset model for vdm.

A changeset model is like an Audit-Log model in which we just record Changesets with Change-Objects rather than have Revision-Objects for each Object that is revisioned.

This change would also incorporate significant simplication of vdm."	1305209986000000	1340632267000000
1136	enhancement	kindly	rgrp		assigned		Move to SessionExtension in vdm	When vdm was created there was no !SessionExtension so we use !MapperExtension for doing revisioning. Now that SessionExtension exists we should use it. We can also follow the existing SQLAlchemy recipe: <http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects>	1305210855000000	1340632980000000
1137	enhancement	kindly	rgrp		assigned		Remove need for statefulness in vdm	"Statefulness, especially statefulness for relation (esp m2m) is cause of most of the complexity in vdm. It is ''required'' because, atm, revision objects have FKs to continuity objects.

This ticket proposes the following changes:

NB: this could be limited just to case of join tables (leaving state stuff on other tables)

 * Remove FKs from revision to continuity (or allow for them to be nullable).
  * We could just limit this to m2m stuff
 * Delete of an object leads to:
  * Deletion of continuity object
  * Adding an entry in revision table with state set to deleted (we retain state on revision table)

If this is done we will no longer need to worry about filtering on state on relationships as join table will only contain ""active"" relationships.

If we do this on all tables we remove need for any state awareness in client (e.g. no need to filter tables on active state).

The only disadvantage of this change is that undeletion becomes more problematic (we have to recreate some continuity objects)."	1305211628000000	1340631974000000
1146	enhancement	kindly	kindly	ckan-v1.5-sprint-1	closed	fixed	make logic layer control its own state	Logic layer should not use any vdm defined state and should manage it itself.	1305829117000000	1307957527000000
1147	enhancement	kindly	kindly	ckan-v1.5-sprint-1	closed	fixed	Add expired_id to all revision tables.	"Revision tables need expired_id to make querying history AND pending changes more efficient.

This involves making a session extension and a large table migration."	1305839833000000	1307957556000000
1148	refactor	kindly	kindly	ckan-v1.5-sprint-1	closed	fixed	test speed improvements and cleanup	The tests have been running slower recently and need fixing.  They also could do with a bit more consistency to them.	1305969223000000	1305969925000000
1149	enhancement	kindly	kindly	ckan-v1.5-sprint-1	closed	fixed	Change domain object modification plugin to use Session extension.	This should make it more efficient as it currently does a lot of repeating work.  i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1.   	1305969863000000	1306090663000000
1193	enhancement	kindly	kindly		closed	fixed	bug with new formalchemy causing all resouces to be deleted	"The new formalchemy stores dict in a new format.

"	1308525655000000	1309768960000000
1205	defect	kindly	dread		closed	fixed	db upgrade problem on cz.ckan.net	"When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below).

cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan.

In the meantime I've reverted the db to the old version (pre-upgrade). 

cz.ckan.net is not working properly until this is fixed.

{{{
okfn@eu19:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://<user>:<pass>@<host>/<dbname>
  module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects
2011-06-29 09:07:41,294 INFO  [migrate.versioning.api] 28 -> 29... 
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname
  ret = fn(self, con, *args, **kw)
Traceback (most recent call last):
  File ""/usr/bin/paster"", line 18, in <module>
    command.run()
  File ""/usr/lib/pymodules/python2.6/paste/script/command.py"", line 84, in run
    invoke(command, command_name, options, args[1:])
  File ""/usr/lib/pymodules/python2.6/paste/script/command.py"", line 123, in invoke
    exit_code = runner.run(args)
  File ""/usr/lib/pymodules/python2.6/paste/script/command.py"", line 218, in run
    result = self.command()
  File ""/usr/lib/pymodules/python2.6/ckan/lib/cli.py"", line 84, in command
    model.repo.upgrade_db()
  File ""/usr/lib/pymodules/python2.6/ckan/model/__init__.py"", line 163, in upgrade_db
    mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py"", line 185, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File ""<string>"", line 2, in _migrate
  File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py"", line 160, in with_engine
    return f(*a, **kw)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py"", line 364, in _migrate
    schema.runchange(ver, change, changeset.step)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py"", line 83, in runchange
    change.run(self.engine, step)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py"", line 140, in run
    script_func(engine)
  File ""/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py"", line 152, in upgrade
    migrate_engine.execute(q)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1788, in execute
    return connection.execute(statement, *multiparams, **params)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1191, in execute
    params)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1271, in _execute_clauseelement
    return self.__execute_context(context)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1302, in __execute_context
    context.parameters[0], context=context)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1401, in _cursor_execute
    context)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1394, in _cursor_execute
    context)
  File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py"", line 299, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table ""package_group_revision"" violates foreign key constraint ""package_group_revision_continuity_id_fkey""
DETAIL:  Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table ""package_group"".
 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'}
}}}"	1309339700000000	1310568631000000
1211	enhancement	kindly	kindly	ckan-v1.5	closed	fixed	[super] Drupal integration	"Sub-tasks / tickets:
 * #1335 Action API - access to any logic layer function
 * Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
 * Add package_create_validate & package_update_validate logic functions
 * Allow extensions to provide logic layer functions (IActions)
"	1309958442000000	1315948703000000
1231	requirement	kindly	thejimmyg	ckan-backlog	closed	wontfix	[super] Management Information Reporting	"Child tickets:

 * #1101 Integrate stats and googlanalytics into site nav

We have a spreadsheet from UKLP of statistics we'd like to generate"	1311173919000000	1325474447000000
1238	defect	kindly	dread	ckan-sprint-2011-10-28	closed	invalid	Viewing a package @date misses non-PackageRevisions	"There's a problem with getting old versions of packages using @date. The query looks for the PackageRevision related to the date, but ignores TagRevisions etc.

So for example if you compare this table:
http://ckan.net/package/history/osm
with this view:
http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830
you actually see the 2010-01-13 11:13 revision - the wrong revision.
"	1311351303000000	1311759784000000
1240	enhancement	kindly	rgrp	ckan-backlog	assigned		[super] API v4	"(Just creating this ticket as somewhere to keep notes)

 * Decide on REST api versus action API
  * Do we want to support both?
 * Tidying
  * Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
  * extras merged into normal field list in package
  * Get rid of /rest/ so just have api/v1/package
  * Get rid of separation of search api from 'rest' api
   * Propose that GET on REST index is search e.g. /package/?q=...
    * This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
 * Resource read/write in API (separate from package)
  * Does this need authorization work?
 * user/account API - read/write
 * Remove autocomplete -- can just use search
  * Do not worry about backwards compat as should only be used in our js (if others using it too bad!)
"	1311525660000000	1325473312000000
1255	enhancement	kindly	kindly	ckan-backlog	assigned		Drupal consistancy checks.	Make a robust way to make sure the drupal database is consistent with the ckan data.	1312219968000000	1313400054000000
1256	defect	kindly	kindly	ckan-sprint-2011-10-28	closed	fixed	fix modification extension for moderation	Solr is not working for moderation, pending changes are not being indexed.	1312280026000000	1319812556000000
1258	enhancement	kindly	kindly	ckan-sprint-2011-10-28	closed	fixed	Bring purge revision into ckan repo from vdm	In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm.  This is best modified in ckan so we will override the vdm one in the reposotory.	1312289539000000	1319812452000000
1268	enhancement	kindly	rgrp	ckan-sprint-2011-10-10	closed	fixed	Speed up listing users by revisions by adding db index	"At the moment this appears to be slow (kindly suggests because missing index in db). Page is /user on a ckan instance.

Would this also speed up user page where we recent changes?"	1312906592000000	1318279592000000
1291	enhancement	kindly	kindly	ckan-sprint-2011-09-12	closed	fixed	Add activity and activity detail tables.	"Add tables described in http://ckan.okfnpad.org/notifications
"	1314267572000000	1315950394000000
1298	enhancement	kindly	kindly	ckan-sprint-2012-01-09	closed	fixed	Generate activites to be put into activities table.	This should be done from the logic layer or automatically from a session extension.	1314696442000000	1325591582000000
1306	enhancement	kindly	rgrp	ckan-backlog	closed	wontfix	Tests for the logic layer	AFAICT there are no tests for the logic layer at the moment. I imagine this is an issue (if it is not please explain and close as wontfix).	1315153267000000	1319797204000000
1316	defect	kindly	dread		closed	fixed	/api/action uncaught exception if missing necessary parameter	"{{{
curl http://ckan.net/api/action/package_show -d '{""name"": ""uk-quango-data""}'
}}}
This gives 500 error due to KeyError (it looked for the 'id' param), rather than 400."	1315397224000000	1338193724000000
1324	enhancement	kindly	kindly	ckan-sprint-2011-09-12	closed	fixed	Add extra fields to resource table	"add extra fields to resource as outlined in

http://wiki.ckan.net/Domain_Model/Resource

Specifically (all unicode strings except for those noted):

 * ''name'': a name for this resource (could be used in a ckan url)
 * ''type'': the type of the resource. One of: file | api | service | listing
 * ''mimetype'': standard mimetype (e.g. for zipped csv would be application/zip)
 * ''mimetype_inner'': mimetype of innermost object (so for example would be text/csv)
 * ''size'': [integer] size of the resource (content length) in bytes. Usually only relevant for resources of type file.
 * ''last_modified'': [datetime] the date when this resource's data was last modified (NB: ''not'' the date when the metadata was modified).
 * cache_url: url for cache of object in ckan storage
 * cache_last_updated: [datetime] timestamp when cached version was created
 * webstore_url: 
 * webstore_last_updated: [datetime] timestamp when webstore was last updated"	1315578976000000	1315948336000000
1335	enhancement	kindly	dread		closed	fixed	Action API - basic	Provide an API to the logic layer at /api/action.	1315910021000000	1315910117000000
1339	enhancement	kindly	rgrp	ckan-backlog	closed	fixed	Issues / question re navl and data conversion	"I ran into a bug with the size field on resources.

 * It would not accept an empty value from form (IMO this clearly equates to null/None)
 * This could be fixed via using ignore_empty instead of ignore_missing
 * However using this means there was no way to empty the field (e.g. i may just want to set the size field back to null not just change to another value)
 * similar issues could arise around other fields (such as last_modified ...)
  * cf cset:645031d07b60

To solve this (cset:58acdcfe6d4e) i created an int_converter temporarily in logic/schema.py (this is almost certainly the wrong place). But I think it raises a bigger issue about the conversion layer and how it works."	1315947022000000	1316014954000000
1341	enhancement	kindly	kindly	ckan-backlog	reopened		Delete spam users from ckan	Spam users where added to thedatahub and we need to clean them.	1315995034000000	1320141540000000
1344	enhancement	kindly	kindly	ckan-sprint-2011-10-10	closed	fixed	datetime error json conversion on search	Json decoding error on search, due to date in resources.	1316020195000000	1317211299000000
1345	enhancement	kindly	nils.toedtmann		closed	fixed	Investigate possible memory leak	"There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) ([http://trac.okfn.org/ticket/882 ticket]) we experienced extraordinary memory usage peaks ([http://trac.okfn.org/ticket/904 ticket]). Here are the observed value with Apache default settings:
 * eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
 * s055, mpm-prefork: base level ~1GB, peaks up to 4GB
 * s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests ([https://bitbucket.org/okfn/sysadmin/changeset/126290aead28 changeset]). This (together with two other tweaks) changed the situation drastically:

 * s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to ''processes=3''), and then observing the relation between ''maximum-requests=25...500'' and memory consumption.


On 14/09/11 17:49, David Read wrote:
> Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1"	1316041450000000	1343215489000000
1356	enhancement	kindly	amercader	ckan-sprint-2011-10-10	closed	fixed	Can not recreate a deleted extra	If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).	1317034180000000	1318279617000000
1364	enhancement	kindly	kindly	ckan-sprint-2011-10-10	closed	fixed	If package in index that is not in ckan error is caused.	We should ignore these but log the packages that are causing this to investigate.	1317222206000000	1318279636000000
1376	enhancement	kindly	kindly	ckan-sprint-2011-10-10	closed	fixed	404 raised when approving package	Regression found by updating datacatalogs.	1318088589000000	1318279651000000
1383	enhancement	kindly	kindly	ckan-sprint-2011-10-24	closed	fixed	Add IResourceChange IResourceUrlChange Interfaces.	Need to add way for extensions to trigger based on resource change and just resource url change. 	1318342153000000	1320142305000000
1397	enhancement	kindly	rgrp	ckan-v1.7	closed	fixed	[super] Resource archiving	"We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: http://ckan.okfnpad.org/queue (most relevant parts inlined here)

== Preliminaries ==

 * Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)
===  Configuration setup for daemons ===

Pass config through to workers i.e site_url, user, api_key.
Need to make site user account. #1408

celeryd config:

All providers of tasks will add an item to the following entry point:

{{{
[ckan.tasks]
name = ckanext.{name}.tasks:....
}}}

celeryconfig.py

{{{
from pkg_resources import iter_entry_points
for entry in iter_entry_points:
     celeryimports.appen(....)

CELERY_IMPORTS = celeryimports
}}}

== Work Items ==

 * Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] -  #1383
 * Generate archiving request on resource url change [0.25d][0.25d] - #1399
 * Make site user account.
 * Make entry point system for celery config
 * Archiver daemon #891
   1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
   2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
 * Write to resource and task status table.[1d][0.75d]
 * [Required?] Make archived data available in WUI - #892
 * Documentation - #1400
"	1318529593000000	1338202920000000
1398	enhancement	kindly	rgrp	ckan-sprint-2011-11-21	closed	fixed	Automated conversion of resource data into webstore	"
 * Possible sibling / requirement: #1397 (resource archiving)

 * Setup test environment. 1d
 * Make task to upload files to webstorer. 3d
   * This includes writing back webstore_url and updated date. 0.5d
   * Type guessing of file format. 1d
   * Make cron to upload all un webstored resources. 0.5d
 * Testing for errors by running cron 2d
 * Make nice way to deploy celery 1d.
 * Documentation. 1d
 * Deployment. 1d
"	1318529724000000	1321827970000000
1402	enhancement	kindly	rgrp	ckan-v1.6	closed	fixed	Migrate repository from mercurial to git	"Plan to migrate from mercurial git

 * Convert from mercurial to git
  * Map named branches to normal branches
  * Also, if possible map usernames for RP: rgrp <http://rufuspollock.org/> (mercurial) -> rgrp <rufus.pollock@okfn.org>
 * Place at http://github.com/okfn/ckan (can delete current repo there as an out of date mirror)

Process:

 * Do trial run
 * Announce conversion date / time
 * Require everyone to have pushed '''all''' outstanding changes at that time
 * Do conversion
 * Test
 * Announce on list and ckan.org/
"	1318811651000000	1324334011000000
1408	enhancement	kindly	kindly	ckan-sprint-2011-10-24	closed	fixed	Make site user, so tasks can post information back to ckan.	Give site user system administrative rights.	1319019626000000	1320141847000000
1433	enhancement	kindly	rgrp	ckan-sprint-2011-11-21	closed	fixed	Support SQLAlchemy 0.7	"Why: current stable version of sqlalchemy. geoalchemy stuff required 0.7 and likely that some other things will require it soon.

Probably requires work on vdm https://bitbucket.org/okfn/vdm

NB: should have discussion before making 0.7 the default required version in CKAN core."	1320143453000000	1324472583000000
1447	defect	kindly	dread	ckan-backlog	assigned		disk space leakage	"Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it.

One problem in the past has been file handles running out when creating lots of tiny files in the data directory.

Another problem has been several enourmous backups being created every day - pdeu on eu25."	1320666843000000	1340727330000000
1448	enhancement	kindly	kindly		closed	fixed	Set up nice way to do celery deployment.	Celery is awkward to deply, need to find a way to do it more simply.  i.e using celery-pylons and supervisor.  A modified version of celery-pylons may be the best solution. 1d	1320666977000000	1325774155000000
1475	enhancement	kindly	kindly	ckan-sprint-2012-01-23	closed	fixed	Create documentation showing how to make an extension that use celery.		1321826681000000	1327576068000000
1477	enhancement	kindly	kindly	ckan-v1.6	closed	fixed	[Super] Group refactor	"This is the implementation of the final draft in

http://wiki.ckan.org/Groups_Refactor

Steps to complete this will follow (somewhat) in the following order.

Change package_group table to membership. (#1478) 3d
  * model migration. (done)
  * simplify revisioning  (done)
  * Remove any sqlalchemy releations from groups. (done)
  * change logic functions (done)
  

Single type per group (#1531)
   * migration add type to group table. (done) 1d
  
Add capacity table (#1522)
  * model migration. (done)
  * add to logic functions.
  
Allow other objects to be added to groups. (#1531)
  * change logic functions (done)

Adapt thedatahub.org to use new model.
  * move auth groups over to groups/members
  * change user_object roles to use groups? keep user object roles for time being.
  * make create update package/group put the correct items in the group/members/capacity tables.
  * work out how ui works.


--not doing the below as part of this super ticker now #1669.

Create model for dgu.
  * adapt auth functions to use auth.
  * adapt package_create/update group_create/update to fill in the correct capacties/members.



Adapt Iati, and potentially other custom instances.

Sub-tickets:
 * #1279 Deprecate Authorization Groups
 * #1198 Publisher hierarchy
"	1321959320000000	1328016209000000
1478	enhancement	kindly	kindly	ckan-sprint-2011-12-05	closed	fixed	change package_group to members table	"This will comprise of the following steps.

  * model migration.
  * simplify revisioning  
  * Remove any sqlalchemy relations from groups.
  * change logic functions

estimate 2d"	1321959973000000	1323172654000000
1485	enhancement	kindly	kindly		closed	fixed	Package/Group form extension mechanism so you can add forms for particular package_types	"We want to be able to change form depending on package type or group type.

This is dependent on a type field being added to the Package and the Group."	1322059169000000	1340034422000000
1492	defect	kindly	dread		closed	wontfix	Interference between extra and relationship fields in API	"From Jan Kučera <elquenor@gmail.com>

 I have also tried to use extras fields to store information about relationships between datasets. I found out that if you first add relationship using CKAN API into dataset relationships attribute and than you add extras field for the same relationship with key in format ""parent_of:dataset_name"" or ""child_of:dataset_name"", storing this extras field will clear all relationships in relationships attribute stored earlier.
 
 I might be some kind of bug but I suppose you will know better. Is it possible to use relationships attribute and extras field for relationships at the same time?
"	1322481496000000	1340034528000000
1495	enhancement	kindly	seanh	ckan-sprint-2012-01-23	closed	fixed	API call for getting a user's public activity stream	An API call that returns a user's public activity stream in JSON format.	1322495549000000	1327576099000000
1511	enhancement	kindly	seanh	ckan-sprint-2012-01-23	closed	fixed	Logic function for getting a user's public activity stream	Add a function to logic.action.get that returns a user's public activity stream as a list of dicts.	1323094207000000	1327576134000000
1522	enhancement	kindly	kindly	ckan-sprint-2011-12-19	closed	fixed	Add capacity to member table.	Need to add capacities to member tables.	1323172610000000	1324333827000000
1531	enhancement	kindly	kindly	ckan-sprint-2012-01-09	closed	fixed	Update group create/update so you can add capacities and group types.	"The new members table needs a way so you can add arbitrary domain objects against them.

We need to extend the group schema to accept types, and instead of just being able to add packages to groups add members with their capacities that associated with different table rows.

4d"	1323272500000000	1326155226000000
1595	defect	kindly	rgrp	ckan-v1.7	closed	wontfix	Editing of a Group leads to recording a change against all associated datasets	"Or at least it appears that way from revision log which displays all Group datasets against the changelog, see http://thedatahub.org/revision/4cdeeb42-3281-4f53-a29d-c694e1fd9217 -- not apparent from that page but see my user page: http://thedatahub.org/user/rufuspollock

This may turn out to be a UX bug rather than a real bug ...

Assigning to kindly for review and comment."	1324504875000000	1338203959000000
1605	enhancement	kindly	rgrp	ckan-v1.7	closed	fixed	[super] Multilingual support in CKAN	"Multi-language:

 * dataset and resource metadata (and other objects such as groups?)
   * Options for implementation in http://ckan.okfnpad.org/multilingual
 * Field values in taxonomy (e.g. country names - Eurovoc)
   * #1665 Research into Eurovoc
   * Display Taxonomy in different languages
 * Field values not in taxonomy (e.g. title & description)
   * Use extra fields e.g. _i18n_title_fr = Le data.
   * (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)
 * EC extension (templates, form)
   * Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
"	1325473367000000	1338193641000000
1612	enhancement	kindly	kindly	ckan-sprint-2012-01-09	closed	fixed	Group view page slow	Group show that lists packages is slow due to not using query in pagination.	1325633737000000	1325688886000000
1614	enhancement	kindly	kindly	ckan-sprint-2012-01-09	closed	fixed	remove po files from git diff	Its a pain to see the difference between branches as there are normally a lot of po file transaction. Make the default be see that they have changed without actually show the diffs themselves.	1325686639000000	1325689136000000
1652	enhancement	kindly	ross	ckan-backlog	assigned		How we intergrate with Drupal Multiligual?	"
* Analysis of how/where we can integrate with Drupal Multilingual

  * Is it likely to access CKAN via /en/data/... or /data/en/... (not sure, probably former) and we need to make sure that it ends up at /en/...

  * Check with Ian for other issues.
"	1326709894000000	1340187535000000
1689	enhancement	kindly	dread		assigned		List deleted datasets in API	"The admin extension allows deleted datasets to be viewed, but there is no equivalent in the API. The package_list API call filters to just 'state=active' datasets. 

'state' could be a parameter on the package_list call
 * Easy to call and implement
 * But getting permissions right for the different state values may not be easy - maybe a separate logic function is better?
"	1327062214000000	1340190040000000
1715	enhancement	kindly	kindly	ckan-sprint-2012-02-20	closed	fixed	Search index multilingual	Need to make solr schema work for many languages. Get stopwords and choose correct analysis for each.	1327598884000000	1329393450000000
1736	enhancement	kindly	kindly	ckan-v1.7	closed	fixed	[Super] Multilingual Metadata	"Translate all areas of metatdata to selected language. This includes finding datasets in all languages.

Multilingal Metadata:
Tasks:
 * Add translation table. 2d
 * Api to add translation 3d
 * Add extention point to dataset view. 3d
 * Add extension point to search index to add translations. 2d.
 * Add all eurovoc data to translations. 1d.
 * Make multilingual solr schema 2d #1715

"	1328012697000000	1338205016000000
1738	enhancement	kindly	kindly	ckan-sprint-2012-02-06	closed	fixed	Add extension point to search index, so can modify dataset dict before index.	It is useful for extensions to add or change items that go into the search index. Add an extension point for this.	1328031572000000	1330086105000000
1739	enhancement	kindly	kindly	ckan-sprint-2012-02-06	closed	fixed	Fix issues where tests where not run.	Test failures in master due to tom and rufus not running tests. 0.3d.	1328047129000000	1328495826000000
1741	enhancement	kindly	kindly	ckan-sprint-2012-02-20	closed	fixed	Add extention point to dataset view.	"Need to add extension point to change the data_dict that passed to the dataset view template. This extensions modify what data gets shown.

Tasks include:

 * Fix up data_dict and templates so that the pkg objects do not get used in the templates.  This is to assure that all data passed is modifiable. (2d) (complete)
 * Add extension point to Ipackage controller (1d)
 * Test (1d)"	1328105709000000	1329750838000000
1753	enhancement	kindly	rgrp	ckan-sprint-2012-03-05	closed	fixed	Last fixes of Webstorer + Deploy on the DataHub	" * Upgrade to new DataStore system
 * Better support for unicode etc
 * ? Rename to DataStorer
 * Deploy"	1328527932000000	1330990061000000
1777	enhancement	kindly	kindly	ckan-sprint-2012-02-20	closed	fixed	Write about multilingual features for ec.	Basic description of multilingual features intended to be supported.	1328542835000000	1329393295000000
1779	enhancement	kindly	kindly	ckan-sprint-2012-02-20	closed	fixed	Add multilingual translation table.	A new table with 3 columns should be added. term, term_tranlastion, language_code.  This table will be used for all translations, including tags.  The table should have indexs on both the term and (term, language_code) combination.	1328548631000000	1329393759000000
1780	enhancement	kindly	kindly	ckan-backlog	closed	duplicate	Api to add translations.	"Add api to translation to the term_translation table.

* add to logic 2d
* test 2d"	1328548908000000	1328578479000000
1781	enhancement	kindly	kindly	ckan-sprint-2012-02-20	closed	fixed	Api to add translations.	"Add api to translation to the term_translation table.

 * add to logic 2d
 * test 1d"	1328549031000000	1329393822000000
1819	enhancement	kindly	kindly	ckan-sprint-2012-04-02	closed	fixed	change search index code to use package_dictize instead of obj.to_dict	Search index code needs to use package_dictize so that it can get the new vocablurary information and is more consistant with the rest of the system.  This is in preperation for their translation being put in the search indexer.	1329770555000000	1338193886000000
1820	enhancement	kindly	kindly	ckan-sprint-2012-03-19	closed	fixed	Index multilingual data when mulilingual extension is added.	All translated fields and vocabularies need to be added to search index in the correct fields.	1329770903000000	1332163374000000
1821	enhancement	kindly	kindly	ckan-sprint-2012-03-05	closed	fixed	create internal multilingual extension that brings together all multilingual features	"This extension point needs to combine.

 *  Making sure the correct data get put in search index.
 *  Make the search queries are modified to weight multilingual search correctly.
 *  View pages, including search results are translated.
 *  Bring together all search schema modifications."	1329771267000000	1330990111000000
1822	enhancement	kindly	kindly	ckan-sprint-2012-03-05	closed	fixed	Weight mulilingual searches correctly	Dismax query across languages. Titles and selected languages, weighted more highly. 	1329772080000000	1330990127000000
1826	enhancement	kindly	kindly	ckan-sprint-2012-03-05	closed	fixed	weight queries so that title is more important than rest of body	Currently everything is considered equally when doing a normal search. Use dismax when there is not a fielded search to get round this. This is part of the work for #1822 as the dismax option is needed for multilingual search.	1329924260000000	1330990247000000
2203	enhancement	kindly	rgrp	ckan-v1.9	assigned		Data Viewer support for PDF	" * Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf
 * propose we use pdf.js - https://github.com/mozilla/pdf.js
 * We could also try iframing for browsers with native support but this seems more complex"	1330860293000000	1348568961000000
2229	enhancement	kindly	kindly	ckan-sprint-2012-03-19	closed	fixed	Cleanup plugin system after some test failed to run.	The logic test did not have __init__. This caused lots of tests to fail because there were mock extensions that ran automatically in them. Fix plugin system so this can work.	1331721611000000	1332163408000000
2233	enhancement	kindly	kindly	ckan-sprint-2012-03-19	closed	fixed	Make package show, use custom json schema	There is a disrepency between what we put in and get out when using a custom form/api. Narrow this gap by making sure package_show shows the correct serilization.	1331865349000000	1332178421000000
2237	enhancement	kindly	kindly	ckan-sprint-2012-03-19	closed	fixed	Make way to split up tests and run tests with different db.		1331938496000000	1332178352000000
2249	enhancement	kindly	kindly	ckan-sprint-2012-04-02	closed	fixed	Fix google spreadsheet to work with DataStore.	Need google spreadsheet app to work with the new datastore.  This should simplify it a lot.	1332330242000000	1338205316000000
2250	enhancement	kindly	kindly	ckan-sprint-2012-04-02	closed	fixed	Setup nginx and datastore on ecportal		1332330570000000	1338205361000000
2291	enhancement	kindly	rgrp	ckan-v1.9	new		DataStorer incorrectly attempts to store JSON in DataStore	"See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review."	1334431147000000	1340636737000000
2294	defect	kindly	rgrp	ckan-v1.9	assigned		Unauthorized action on Data API gets 302 rather than 401 or similar	"Well known issue that 401 *in API* end up with 302 redirect to login page (which makes no sense and is very confusing for clients).

Fix this by ensuring that error messsages on API get passed through correctly.

Minor because affect is annoying but not terminal"	1334432104000000	1340624003000000
2327	enhancement	kindly	kindly	ckan-sprint-2012-04-30	closed	fixed	Upgrade solr schema to version 1.4	" * Add Ascii folding filter to text fields.
 * Add capacity field for public, private access.
 * Add title_string so you can sort alphabetically on title.
 * Fields related to analytics, access and view counts.
 * Add data_dict field for the whole package_dict.

Actual implementation of data_dict adding will be added later."	1335374973000000	1338205397000000
2331	defect	kindly	rgrp	ckan-sprint-2012-05-29	reopened		Search should AND terms not OR terms	"Appears current default search in CKAN ORs terms rather than ANDing them (i.e. adding more terms increasing number of items found rather than reducing it).

Not sure when this crept in or if it has been there for a long time."	1335637485000000	1356474344000000
2334	enhancement	kindly	rgrp	ckan-v1.9	assigned		Improved data import	"Placeholder for thinking about this ...

 * Allow users to configure info for CSV import (e.g. separators, quotes etc)"	1335651662000000	1340624538000000
2402	enhancement	kindly	kindly	ckan-sprint-2012-05-29	closed	fixed	search result speedup.	Search results are slow as we do many package_dictize.  Store the package_dict in the search index so that we do not have to hit the database for each of them.	1337300201000000	1337781991000000
2403	enhancement	kindly	kindly	ckan-sprint-2012-05-29	closed	fixed	home page speed improvements	Home page loads *all* groups and dictizes them, it should only dictize the top 10 or so from the search facets.	1337302582000000	1337782037000000
2406	defect	kindly	markw	ckan-v1.9	new		Errors in source install doc	"There are a few errors or unclarities in the source install doc - see

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html
http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html
"	1337595093000000	1340636804000000
2415	defect	kindly	markw	ckan-v1.9	new		Confusing message on attempting to register new account with upper case	"On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

""The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_""

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.
"	1337779189000000	1341268304000000
2422	defect	kindly	ross		closed	invalid	Paster rights command appears broken	"Reported in IRC by @floapps


In previous ckan versions i could use paster commands to remove reader rights from visitors on all packages
and this would then throw the user to login screen when trying to see a dataset but, when i did this in ckan 1.7, i could still see all datasets as a visitor

Using http://docs.ckan.org/en/latest/authorization.html#permissions-publisher-mode

    paster rights remove visitor reader package:all


Also has an empty list of roles in config for default visitor roles

I tested on my local install and could replicate with that command doing nothing. Neither affects existing or newly created datasets."	1337937993000000	1337945204000000
2425	enhancement	kindly	seanh	ckan-sprint-2012-06-25	closed	fixed	Get rid of CKAN's flup dependency		1337946421000000	1340117561000000
2426	enhancement	kindly	seanh	ckan-v1.9	new		Get rid of CKAN's formalchemy dependency		1337946462000000	1340616095000000
2427	enhancement	kindly	seanh	ckan-v1.9	new		Reduce cruft installed into CKAN's virtualenv due to Pylons		1337946507000000	1341268323000000
2473	enhancement	kindly	kindly	ckan-v1.9	assigned		Make datstorer store field ordering in _meta field	Recline views should have a default table order, being the same as the csv that was imported.	1338293492000000	1340636871000000
2524	enhancement	kindly	kindly	ckan-ecportal	new		If there are no translation files for selected language fall back to default lang.	If a user selects a language there are no mo files for then an error is raised.  Revert to default language instead.	1339609048000000	1340117608000000
2530	enhancement	kindly	rgrp	ckan-v1.9	new		DataHub purge fails on some revisions	"See http://datahub.io/ckan-admin/trash and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like:

Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (IntegrityError) update or delete on table ""revision"" violates foreign key constraint ""group_extra_revision_revision_id_fkey"" on table ""group_extra_revision"" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table ""group_extra_revision"". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'}"	1339750498000000	1341268280000000
2548	enhancement	kindly	ross	datahub-july	assigned		Object ownership for groups/package	"
== Requirements ==

We need to be able to easily determine who the owner of a dataset or group is.  Datasets and Groups should have an Owner, who may change over time but is a specific user within the CKAN instance.  It should be easy for CKAN components to determine the user and for the initial version we should ignore the can of worms labelled 'ownership transfer'.

At this point migration is likely to be the biggest issue, and would suggest that it is acceptable that the last user to edit a dataset be set as the current owner.

More tickets should arise as a result of this work where we may be able to optimise some queries to use the new feature.


== Interface ==

None


== User Stories ==

None


== Tasks ==

[ ] Analysis/Clarification

[ ] Tests

[ ] Migration

[ ] Code/Schema changes

[ ] Documentation


== Estimates ==


"	1340017331000000	1340706539000000
2571	enhancement	kindly	toby	demo phase 1	closed	fixed	implement logic.action.create.resource_create()	"This function is incomplete

If you want I can do this but I need guidance over how to do this properly. ie would I go through lib.dictization.model_save.package_resource_list_save() which looks wrong to me as I'd need the package.  resource_dict_save() seems better but to be honest the total lack of docstrings etc makes me feel ill."	1340102259000000	1340369783000000
2574	enhancement	kindly	toby	demo phase 1	closed	fixed	look at package states	"we want to be able to add new datasets but not have them indexed or seem by people

is the vdm.state the place to do this"	1340115771000000	1342092416000000
2580	enhancement	kindly	toby	demo phase 1	closed	fixed	create resource validation	"david I've got the add resource working but it is happy to add empty resources which implies that it isn't doing validation correctly.

I'm not sure what the validation rules are so hard to test this - I'm coming through as an api call

let me know if i can help"	1340285208000000	1342092353000000
2674	defect	kindly	shevski	demo phase 5	assigned		Data preview not loading on s031	"Not loading for all resources as far as I can tell; e.g. http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360

Fine on demo.ckan.org"	1342435102000000	1344349324000000
2708	enhancement	kindly	toby	ckan-v1.9	new		limit extra data for package/group show	"    context['package_limits'] = { 'tags': 5,     <- get first 5
                          'extras': 0,   <- get all
                         }
only get what you ask for have to be explicit

    context['group_limits'] = {}   only main item


start with datasets/groups expand if we like it"	1342622420000000	1342622420000000
2729	enhancement	kindly	shevski	ckan-backlog	new		searching for tags:[tag] works but tag:[tag] doesn't	"which is confusing since you can only search for one tag like this at a time. I.e.
tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work!

http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc"	1342951109000000	1342951176000000
2790	enhancement	kindly	toby	demo phase 4	new		logic.action.user_show is slow	"This is a very slow call it would benefit from the sort of speed-ups that package-search received

for me locally this is taking 6 seconds for rufus using the datahub data I have.  I think a lot of this is the dataset retrival/dictization

can we just grab json blobs from solr?

also is it possible to specify a sort order/paging?

I've put this as a demo-theme ticket as it is an big issue on the demo we are at 25 second page loads - which i can get down to about 8.8 secs so this is the main pain point now"	1343852483000000	1345023734000000
2877	enhancement	kindly	rgrp		assigned		Bugs with datastore v2	"In progress

 1. ~~[major] q does not seem to work reliably. e.g. using the setup from this gist https://gist.github.com/1930806 and doing a ?q=DE yields no results (does not work with ""q=de"" either)~~
  * ~~q=second does work ...~~
 1. [major] q does not work with 2 values (see below)
 2. Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT
   * Also weird fact that limit is returned but as as as string - should it not be an integer
 3. Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...)

=== Multiple query values ===

Try a query such as: ""second UK"" and you will get 500 error:

http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&q=second%20UK

{{{
ProgrammingError: (ProgrammingError) syntax error in tsquery: ""second UK"" 'select ""_id"", ""id"", ""date"", ""x"", ""y"", ""z"", ""country"", ""title"", ""lat"", ""lon"", count(*) over() as ""_full_count""\n from ""4f1299ab-a100-4e5f-ba81-e6d234a2f3bd"" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',)
}}}

=== Suggestions ===

Filter support: should think in more detail about this (may want to follow recline style)

Simple filters in query parameters would be nice too ..."	1345250002000000	1346320395000000
2935	enhancement	kindly	kindly	ckan 2.0	closed	fixed	Make recline work with new datastore		1348042459000000	1350578067000000
3027	enhancement	kindly	kindly		new		solr for 2.0	"change mm
support solr 3 and 4
add *_date field"	1354587729000000	1354587729000000
2446	enhancement	johnmartin	aron.carroll	demo phase 4	assigned		Create demo dataset history/comparison page	"This is a low priority for the demo site.

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62414120/comments

Implementation:

http://s031.okserver.org:2375/dataset/history/adur_district_spending"	1338210738000000	1352658819000000
2451	enhancement	johnmartin	aron.carroll	demo phase 5	assigned		Create demo user list page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

http://s031.okserver.org:2375/user"	1338211213000000	1352658902000000
2457	enhancement	johnmartin	aron.carroll	demo phase 4	assigned		Create demo tags list page	"This includes the tag page as well for now.

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments

Implementation:

http://s031.okserver.org:2375/en/tag"	1338211735000000	1352658878000000
2471	enhancement	johnmartin	ross		closed	invalid	ckan.org bug	"If you go to http://ckan.org/features/geospatial/ there is a problem with the pop up tag shown (and zoomed in) - missing status and description.

"	1338229749000000	1352206679000000
2562	enhancement	johnmartin	aron.carroll	demo phase 5	closed	invalid	Demo theme tag improvements	"Tags should be implemented in two sections.

 * Custom predefined tags using vocabularies
 * Provide suggestions, auto completion then fallback to free text"	1340035714000000	1352206599000000
2599	enhancement	johnmartin	aron.carroll	demo phase 4	closed	fixed	Test in IE	can we promote this to phase 3?	1340717662000000	1346756230000000
2616	enhancement	johnmartin	aron.carroll	demo phase 5	closed	invalid	Improve display of split stages in demo	Currently stages of the same colour blur together, these could do with a nice divider.	1340875108000000	1352206282000000
2631	enhancement	johnmartin	aron.carroll	demo phase 5	closed	fixed	Style activity stream in user profile		1341826342000000	1352205778000000
2641	enhancement	johnmartin	amercader	demo phase 5	assigned		Adapt spatial widgets to new theme	Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now	1341846147000000	1352658854000000
2658	enhancement	johnmartin	aron.carroll	demo phase 5	closed	invalid	On Demo change 'Groups' to 'Publishers' everywhere	Can we do this as a localisation?	1342008495000000	1352205894000000
2676	enhancement	johnmartin	shevski	demo phase 5	closed	fixed	no way to browse through related items on dataset page	"Need to add JS (presumably?) to the related app box on dataset page: http://s031.okserver.org:2375/dataset/gold-prices to allow user to scroll through when multiple related items exist
"	1342435678000000	1352205920000000
2697	enhancement	johnmartin	shevski	demo phase 5	assigned		create dataset validation	Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check)	1342620035000000	1346235925000000
2704	enhancement	johnmartin	shevski	demo phase 5	closed	fixed	add list of resources to edit dataset page sidebar	when editing a dataset (e.g. here: http://s031.okserver.org:2375/en/dataset/edit/afghanistan-election-data) add sidebar list of resources for user to click & browse through taking you to the relevant resource edit pages	1342620986000000	1352205939000000
2705	enhancement	johnmartin	shevski	demo phase 5	closed	fixed	add link to edit dataset page from edit review page	"e.g. on http://s031.okserver.org:2375/en/dataset/afghanistan-election-data/resource_edit/f6331f99-51f6-44d9-95b9-b20f3b74f360

& ensure consistant with sidebar of edit dataset page"	1342621122000000	1352205954000000
2711	enhancement	johnmartin	toby	demo phase 5	closed	fixed	bin/less dies for toby	it would be nice if this didn't fall over when the dir is removed maybe a linux only issue?	1342631508000000	1352205968000000
2714	defect	johnmartin	shevski	demo phase 5	closed	fixed	XLS preview not working	"see http://demo.ckan.org/dataset/vif-2012-population-totals/resource/e7b8401a-c410-4902-9e5a-6dbd78834873

what XLS previews do we support?"	1342706454000000	1352206138000000
2741	enhancement	johnmartin	toby	demo phase 4	closed	fixed	admin index styling	"http://localhost:5000/ckan-admin

you need to be an admin

paster sysadmin add <username> -c dev.ini

we need to get links to other admin pages etc - I think this will want to be flexible so extensions can add extra links etc - but that can come later"	1343124308000000	1352205988000000
2742	enhancement	johnmartin	toby	demo phase 5	closed	fixed	admin authz styling	"http://localhost:5000/ckan-admin/authz

you need to be an admin

paster sysadmin add <username> -c dev.ini

like #2741 we need other admin links

you may want to make the inputs use your form macros - I've just done a basic port

also we will need to get all the tables done but looks like we need to get auth groups working for that"	1343124423000000	1352206081000000
2785	enhancement	johnmartin	aron.carroll	demo phase 5	assigned		Allow resources to be re-ordered	"Not sure where this functionality should be added, possibly in one of the sidebar widgets when editing a resource? 

Ira, what are your thoughts?"	1343816523000000	1346235916000000
2806	enhancement	johnmartin	shevski	demo phase 4	closed	fixed	way to read full group description	"e.g. currently no way to read full description for the 'data explorer examples' group here:http://s031.okserver.org:2375/group/data-explorer

Can we add a 'read more' link that expands and contracts the sidebar box or similar?"	1344254292000000	1352206005000000
2903	enhancement	johnmartin	johnmartin	demo phase 5	closed	wontfix	Two search boxes	"I think the architecture on the search page is a bit odd. Not sure I like that there is a doubling up of the search box (e.g. one in the header and one in the body of the search results page)

Needs some thought"	1346853712000000	1352206813000000
2907	enhancement	johnmartin	johnmartin	demo phase 5	closed	fixed	Demo header	The user login / avatar with in the header really should be within another topbar, as it goes against convention to have the user information within the normal header.	1346930925000000	1352206307000000
2916	enhancement	johnmartin	ross	ckan 2.0	closed	fixed	User management in orgs needs autocomplete	"
The user management in organizations ( http://localhost:5000/organization/users/NAME ) should autocomplete names in the dropdown box so that only valid usernames are added to the table."	1347451792000000	1347970727000000
2917	enhancement	johnmartin	ross	ckan 2.0	new		Organization admins can delete themselves	Organization administrators can delete themselves from the user management pages.  We should disable this on the front-end as well as the back-end.	1347451843000000	1347455636000000
2918	enhancement	johnmartin	ross	ckan 2.0	closed	fixed	Can't remove users from organizations	"When you remove someone, without adding them, the text box at the bottom (which should probably autocomplete) is empty, and this causes problems on the server.

Ideally when you add a user (select from the autocomplete) it would add another row to the table, defaulting the user to editor and setting the names to user__{{X}}__name and user__{{X}}__capacity where X is $('tr').size()"	1347455572000000	1347970735000000
2940	enhancement	johnmartin	johnmartin	ckan 2.0	closed	fixed	Edit different resources from edit dataset pag	We need a nice way to be able to get to editing the different resources that are associated to a dataset.	1348580812000000	1350560926000000
2941	enhancement	johnmartin	johnmartin	ckan 2.0	closed	fixed	Add follower support back into CKAN 2.0	Add the views and functionality of following users and groups into 2.0	1348584670000000	1350560915000000
2954	enhancement	johnmartin	johnmartin	ckan 2.0	closed	fixed	Flickering on resource view when changing between views	From the google doc: When I move between grid, graph and map view mode (or click on filters) the whole data explorer flickers (jumps up & down slightly) for a few moments before settling. This is happening on all resources for me and when you initially open launch the page: http://demo.ckan.org/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360. Only in Chrome, working fine in Firefox.	1349268916000000	1350578030000000
3015	enhancement	johnmartin	johnmartin	ckan 2.0	closed	invalid	Dropdowns don't work without JavaScript	Essentially the default bootstrap dropdowns don't work properly without JavaScript which is a little bit of an issue with JS not working.	1352805553000000	1353410072000000
3016	enhancement	johnmartin	johnmartin	ckan 2.0	new		CKAN 2.0 template tweaks	Just a ticket to keep track of a few suggested template changes.	1352813417000000	1352813417000000
3018	enhancement	johnmartin	johnmartin	ckan 2.0	new		Load more in activity streams	Activity streams should be able to load more than 15 items within them. Suggest the default amount of loading is around 30 and then click to load more.	1352900051000000	1355140950000000
3021	enhancement	johnmartin	johnmartin		accepted		Logout doesn't work without JS	"Essentially, the functionality should be as follows:

- Add logout link that has `.js-hide` attached to it within the header that isn't hidden within a dropdown

See http://plus.google.com/ (when logged in) with and without JS to see an example of the actual sign-out working without JS"	1353410350000000	1353411968000000
833	enhancement	johnlawrenceaspden	rgrp	ckan-v1.4-sprint-6	closed	fixed	[super] Administrative dashboard extension	"Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:
  * Purge revisions (or sets of revisions) and purge objects #1076
  * Set roles for users #1075
  * Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
    * WONTFIX
  * Overview of activity
    * WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.

https://bitbucket.org/okfn/ckanext-admin

=== Tickets ===

 * ticket:1031 - user autocomplete api in ckan

=== Notes ===

Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

{{{
# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of 
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system
}}}"	1290765921000000	1303236364000000
854	defect	johnlawrenceaspden	dread	ckan-v1.4-sprint-6	closed	fixed	Tests for authorization_group	"The coverage tool (run by buildbot in the ckan build) reports that only 24% of lines of ckan.controllers.authorization_group are run in tests and 38% of ckan.forms.authorization_group. This suggests a need for more tests.

"	1291723815000000	1304351843000000
1031	enhancement	johnlawrenceaspden	rgrp	ckan-v1.4-sprint-4	closed	fixed	User lookup API	"Add an api for searching users. This is needed for any kind of ajax autocomplete (needed for anywhere we want to add users).

 * API location: /api/util/user/lookup?q=querystr&limit=10
 * Return json objects containing {id: ..., name: ..., fullname: ...}
 * Put in a module called controllers/apiv2/user.py"	1299780419000000	1300101520000000
1036	defect	johnlawrenceaspden	johnlawrenceaspden	ckan-v1.4-sprint-4	closed	fixed	Add tests for three functions in ckan/model/user.py	"create a new test file ckan/model/test_user.py

add tests for the following three functions in ckan/model/user.py

number_of_edits,
number_administered_packages,
search


merged in in changeset 0046f83aedcf
"	1300127840000000	1301304575000000
1075	enhancement	johnlawrenceaspden	rgrp	ckan-v1.4-sprint-6	closed	fixed	Administrative dashboard - Edit Authorization related to System object	"Roles on System object are important because admin role on system equates to being a 'sysadmin' (i.e. able to do anything).

 * Make users sysadmin (either as separate action or as part of editing roles on system object)
 * /authz subpage for editing roles on system object
  * Add and update user roles
  * Add and update authz group roles
  * List actions associated to roles at top of page (extra points for checkbox table with editability)
 * Document on http://wiki.ckan.net/Authorization what roles on System object 'mean' esp sysadmin role on System

== Related Tickets ==
 
 * super ticket: #833
 * authz lib improvement and authztool refactor #1050"	1302279799000000	1303227982000000
1076	enhancement	johnlawrenceaspden	rgrp	ckan-v1.4	closed	fixed	Improve revision and package purge system	"== Purging Revisions ==

 * Delete button displayed on:
  * /revision/list
  * ~~(/package/history)~~
   * /package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison.
  * /revision/{id}
 * Delete button submits to delete action on revision and changes revision state to 'deleted'.
  * undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?)
 * Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: ""Empty trash"" (irreversible). Click button purges all revisions with deleted state.

== Purging Packages ==

 * Put into deleted state.
 * Listed on /ckan-admin/trash
 * Separate Empty trash button which deletes all associated revisions.
  * Should be separate from Empty trash for revisions

== Current system ==

 * Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!)
"	1302283442000000	1303236302000000
1081	defect	johnlawrenceaspden	johnlawrenceaspden		closed	fixed	can't remove user from authz group	I've found that if I make an authorization group I sometimes can't remove myself from it. I've no idea why. I can add and remove other users. I'll investigate, just making a note of it here.	1302541056000000	1303489474000000
1083	defect	johnlawrenceaspden	johnlawrenceaspden	ckan-v1.5-sprint-1	closed	fixed	userobjectroles added twice can't be deleted	"the add_user_to_role/remove_user_from_role functions are asymmetrical
in that the add function is happy to add the same role twice but the remove asserts that it's only in the table once and crashes if that's not true.

an attempt has been made to guard against this, but fails, I think because the add functions rely on the caller committing the change to the db.

same problem affects corresponding authorization_group functions

I'll try to sort this out. Making a note here."	1302550660000000	1305537827000000
191	enhancement	johnglover	dread	ckan-sprint-2011-12-19	closed	fixed	Searching by modification date	"Cost - 2 days

Search interface has new options to filter and sort the results by the date the package has been last modified in ckan. Search options are included in both Web UI and Search API.

The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package. 

Example search parameters:

|| modification-range=5/4/09- || Exclude packages last modified earlier than 5/4/09 ||
|| modification-range=5/4/09-5/12/09 || Exclude packages last modified outside of 5/4/09-5/12/09 ||
|| order_by=mod ||Sort by metadata modification. Defaults to newest first. ||
|| order_by=mod-newest || Sort by metadata modification, newest first. ||
|| order_by=mod-oldest || Sort by metadata modification, oldest first. ||
"	1258387778000000	1330020983000000
318	defect	johnglover	wwaites	ckan-sprint-2011-10-28	closed	fixed	Insufficient validation of resource URIs	"The CKAN instance on data.gov.uk serves invalid URIs out of its API.

For example the following can be found,

http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls]

In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396

Also URIs are not stripped of whitespace at the end.

This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs.

""Be liberal in what you accept and conservative in what you send"" would seem apt.

== Actions ==

  * Validation of urls as part of form entry or data loading
    * Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal
  * This code should support analysis of existing data so we can go through existing database and find invalid urls
    * Also useful to have this so we can do out of band validation "	1274302723000000	1349778662000000
816	enhancement	johnglover	cygri	ckan-sprint-2011-10-28	closed	fixed	Autocomplete for the resource format field	"The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource.

This would help improve data quality and consistency."	1289996574000000	1319812324000000
842	enhancement	johnglover	rgrp	ckan-v1.4-sprint-7	closed	fixed	Todo list CKAN extension	"An extension that provides a todo list feature on CKAN so that people can register and find things to do.

Extension name: ckanext-todo

== User Story ==

=== Package page ===

As a user I come to a package:

 * Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
 * At the bottom is a section of the package display titled ""ToDo"" where I see a list of all toDos for the package most recent at the top
  * If I am logged in 
   * See a form for ""Add to do"" at the top of the todo section and can add one straight away
   * I see a ""now resolved"" button next to each which goes green when you hover.
      When clicked the todo fades away.
   * Not logged in: I see a button that says ""login to add todo""

=== Todo list page ===

When a user comes to todo overview at /todo

At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category.

Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo.


== Implementation ==

The Todo form

 * One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed)
 * Add a description
 * Submit, the todo gets added via AJAX to the list at the top as the most recent todo

Model:

{{{
todo table

  id (autoincrement integer)
  package_id
  todo_category_id (required)
  description (required)
  created=NOW()
  resolved=null (unresolved) or a datetime (datetime of resolution)
  creator=user
  resolver=user

todo_category table
  id
  name

Prepopulate with: broken-resource-link, no-author, bad-format, add-description
}}}

API at /api/2/todo
 * GET / POST / PUT ...

/api/2/todo?package=package_id_or_name&category=...&resolved=0/1
  * support limit (?)

/api/2/todo/category -> return list of todo categories
 * No GET / PUT / POST (these are auto-created by creation of todo)

== Optional Extras (Will not be done atm) ==

 * Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ..."	1291467708000000	1305646487000000
888	enhancement	johnglover	rgrp	ckan-sprint-2011-10-28	closed	fixed	Improvements to the dataproxy and the data API	"First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

  * Testing for dataproxy
    * Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
  * Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
  * Google docs style row/column selections
  * Use the swiss library - https://bitbucket.org/okfn/swiss
    * Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
  * Handle redirects for content-length?
  * Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

  * Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}"	1293649783000000	1311773103000000
891	task	johnglover	pudo	ckan-sprint-2011-11-07	closed	fixed	Resource download worker daemon	"Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository. 

== Questions == 

 * Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
 * Should we have clever ways to dump APIs? ANS: no.
 * Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
 * Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache. 
  * Complete support for ETags
  * Expires, Max-Age etc. 
 * Check 

== Functionality == 

 * Download files via HTTP, HTTPS (will not do FTP)

Process:

 1. [Archiver.Update checks queue (automated as part of celery)]
 2. Open url and get any info from resource on cache / content-length etc
  1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
  2. Check headers for content-length and content-type ...
    * IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
    * ELSE: check content-type.
      * IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
      * ELSE: archive it (compute md5 hash etc)
    * IF: get content-length and content-length unchanged GOTO step 4
 3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
  * Add cache url to resource and updated date
  * Add other relevant info to resource such as md5, content-type etc
 4. Update task_status
  
== Optional functionality ==

 * If result object is HTML, search for references to ""proper data"" (CSV download pages etc.)
 * Download from POST forms (accepting licenses or weird proprietary systems) 
 * Support running on Google Apps Engine to save traffic costs.

== Existing work == 

 * https://bitbucket.org/okfn/ckanext-qa/overview
 * out of date: https://bitbucket.org/pudo/ckanextarchive - Old archiver extension, largely experimental. 
 * out of date: https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65 - Openness scores by ollyc"	1294052979000000	1320149841000000
892	enhancement	johnglover	pudo	ckan-sprint-2012-01-09	closed	fixed	Make stored data available in WUI - 0.5d	"Once we have storage, make the data available in the following ways: 

 * Now have a cached_url field can show in the frontend ...
 * Add a [<a href=""${cached_url}"">cached</a>] link to right of real url on resource listing on dataset page.
 * On resource page: will not add it yet.
   * At the moment no clear place to pu this given nice big download button (could put in list of items on left but that does not seem right and note that it will turn up in big list of info at bottom)
 * Add test (?)
 * Deploy
"	1294053293000000	1324402480000000
936	enhancement	johnglover	rgrp	ckan-v1.4-sprint-6	closed	fixed	Follow / watch package extension	"As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis).

NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'.

Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). '''Decision: use follow'''

== Implementation ==

=== Interface ===

Become a follower:

 * Follow button on packages (if already watching say 'unfollow')

Package-related changes:

 * Show number of followers on a package
 * List followers of a package at /package/{name}/followers
  * On a separate page
a package
User-related changes:

 * List followed packages
  * Either on user's page on a separate 'following' page. (NB: called 'following')
 * Does watching involve notifications (by email)
  * Probably not: you can already subscribe to RSS feed after all and email not that necessary (?)
 * [Future - don't have activity stream yet] Show what packages a user has started/stopped  followed on a user's public activity stream on their user page

=== Nitty-Gritty ===

 * Want to do this in ajax-y manner
 * API endpoint: /api/2/follower
 * Store data in a new follower table

==== API ====

/api/2/follower

{{{
follow => PUT / POST
{
   user_id
   object_type
   object_id
}
}}}

If this is submitted by a user with user.id != user_id => error (401)

{{{
unfollow => DELETE

/api/2/follower/package/{id}
=> list of followers
[
    { safe dictized user }
]
}}}

NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method.

==== Table ====

Called 'follower'

{{{
user_id, table, object_id, created
xxx, package, yyy, ...
xxx, user, yyy, ... [future]
}}}
 
=== Random Extras ===

 * What about following users as well
"	1296339079000000	1303838713000000
1062	defect	johnglover	sebbacon	ckan-backlog	assigned		Data preview encoding error	"The preview of ""Species Misc Turtle Download"" at http://ckan.net/package/taxonconcept results in the following error:

Unable to Preview - Had an error from dataproxy:
Data Transformation Error (Data transformation failed. Reason: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte
"	1301396143000000	1311773731000000
1150	defect	johnglover	timmcnamara	ckan-sprint-2011-10-28	closed	fixed	Non-ASCII chars prevent data preview	"Characters outside of ASCII range are not supported within data previews.

Steps to reproduce:

1. Visit http://ckan.net/package/kivele2010
2. Click on [preview] for any of the resources

"	1306019914000000	1311774141000000
1154	enhancement	johnglover	nils.toedtmann	ckan-sprint-2011-10-28	closed	fixed	Make ckan robust against solr failure	"According to pudo, a ckan with activated solr extension throws a 5xx when solr is unreachable. Instead, it should behave more like a ckan without ckanext-solr when this happens.

"	1306254472000000	1314287519000000
1183	defect	johnglover	nickstenning	ckan-sprint-2011-10-28	closed	fixed	"Downloads ""Preview"" button doesn't preview."	"The ""Preview"" button is a nice idea, but it doesn't seem to actually ""preview"" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file.

This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of `application/x-some-junk-here` in order to force a download."	1307701669000000	1311771069000000
1203	defect	johnglover	rolf	ckan-backlog	new		"Moderated edits: html code shows as ""changed"" although it is not"	"I've installed the Moderated Edits extension (ckanext-moderatededits) and am editing a package imported from IATIregistry.org, with an extra field which contains a bit of HTML.

The editor indicates the field has changed, although the content hasn't (see screenshot). All I can find so far is a minor difference: in the field content, there is a code &#8212 and in the rendered table that is an &mdash;"	1309274970000000	1313401579000000
1259	enhancement	johnglover	pudo	ckan-backlog	new		"""Add a row"" for Extras on Package form"	"The default package form offers 4 empty extras fields. Like the resource section, it should have an ""add more"" button to add another row. "	1312302693000000	1312907056000000
1269	defect	johnglover	rgrp	ckan-backlog	closed	wontfix	User view page shows the packages followed by viewing user not user being viewed	This overlaps with Sean's follow support (for activity streams) and so will be dealt with there.	1312906598000000	1335875028000000
1274	enhancement	johnglover	johnglover	ckan-sprint-2011-10-28	closed	fixed	Testing solr search	Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend.	1313413202000000	1313429049000000
1275	enhancement	johnglover	johnglover	ckan-sprint-2011-09-26	closed	fixed	Move solr search extension to ckan core		1313413597000000	1319812967000000
1276	enhancement	johnglover	johnglover	ckan-sprint-2011-09-12	closed	fixed	Apply weighting to search fields in solr	"For example:

 * name and title should have a higher weight than the notes field"	1313414353000000	1315948394000000
1277	enhancement	johnglover	johnglover	ckan-sprint-2011-09-12	closed	fixed	Use solr query parser for search instead of ckan query parser	See http://ckan.okfnpad.org/search for details/discussion	1313414446000000	1315948417000000
1302	refactor	johnglover	johnglover	ckan-sprint-2011-09-26	closed	fixed	Move Resource and Tag search code to logic layer	Resource and Tag search is currently still using the old ckan query parser which is unnecessary. Refactor this out and move to the logic layer.	1314886102000000	1317131948000000
1336	defect	johnglover	dread	ckan-backlog	new		License fudge	"cset:4b59ab34137d ckan/logic/action/get.py:
{{{
-            isopen = model.Package.get_license_register()[license_id].isopen()
-            result_dict['isopen'] = isopen
+            try:
+                isopen = model.Package.get_license_register()[license_id].isopen()
+                result_dict['isopen'] = isopen
+            except KeyError:
+                # TODO: create a log message this error?
+                result_dict['isopen'] = False 

}}}

This change hides problems with the license server and returns potentially incorrect values for openness. 

This has been noted as 'temporary fix' but seems to be forgotten about, since it has been merged to default and gone into release 1.4.3.

I suggest the licenses are cached (I thought this was already the case when CKAN first requests them after start-up?). I suggest failure would return 503."	1315912057000000	1323173073000000
1342	defect	johnglover	amercader	ckan-sprint-2011-10-10	closed	fixed	Home controller does not capture SearchError exceptions	"If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting http://localhost:5000/dataset).

Steps to reproduce:

* Change the solr_url parameter in the ini file to something non-existant
* Visit http://localhost:5000"	1316001227000000	1318256532000000
1362	defect	johnglover	johnglover	ckan-sprint-2011-10-10	closed	fixed	Clearing the database should also clear the search index	When paster db clean is run, the search index should also be cleared.	1317121861000000	1318256546000000
1363	enhancement	johnglover	kindly	ckan-sprint-2011-10-10	closed	fixed	Add task_status table to store qa/archiever/webstore information that does not need to be versioned.	"{{{
table: task_status

id
entity_id
entity_type
task_type
key
value
last_updated
state

unique(entity_id, task_type, key)

example keys:
last_task_id
}}}"	1317219617000000	1318256600000000
1371	enhancement	johnglover	johnglover	ckan-sprint-2011-10-10	closed	fixed	Make logic layer functions for working with task_status table	The background tasks will use the logic API to access the task_status table so new logic layer functions are needed.	1317808776000000	1324037905000000
1372	refactor	johnglover	johnglover	ckan-sprint-2011-10-24	closed	fixed	Refactor archiver to work as a celery task	Take the resource archiver (currently part of ckanext-qa) and make it into a standalone extension that works as a celery task.	1317808970000000	1319625783000000
1389	defect	johnglover	johnglover	ckan-sprint-2011-10-24	closed	fixed	Update ckanext-datacatalogs for ckan 1.5		1318497052000000	1319625799000000
1390	enhancement	johnglover	johnglover	ckan-sprint-2011-10-24	closed	fixed	Update ckanext-moderatededits for ckan 1.5 on datacatalogs.org		1318497099000000	1319625810000000
1391	enhancement	johnglover	johnglover	ckan-sprint-2011-10-24	closed	fixed	Update datacatalogs.org to ckan 1.5		1318497186000000	1319625819000000
1392	enhancement	johnglover	johnglover	ckan-sprint-2011-11-07	closed	fixed	Move datacatalogs.org to an OKFN server		1318497251000000	1320143150000000
1393	enhancement	johnglover	dread	ckan-backlog	assigned		Don't skip search tests	Now we don't use postgres search, all the tests involving search now don't need to be skipped when running on sqlite. Should help coders spot earlier if these tests break. 	1318505453000000	1320153590000000
1399	enhancement	johnglover	rgrp	ckan-sprint-2011-10-24	closed	fixed	Generate archiving request on resource url change	"Superticket: #1397

Generate archiving request on resource url change by implementing IResourceUrlChange and sending tasks to celery. [0.25d][0.25d]

Will be part of of ckanext-archiver extension.

Process:

 0. A resource is added to CKAN
 1. IResourceCreate event generated
 2. IF: resource url points to ckan storage or falls within some other set of exclusion conditions then END else continue
 3. Generate a Archiver.Update task with resource.id"	1318600123000000	1319625829000000
1400	enhancement	johnglover	rgrp	ckan-sprint-2012-03-05	closed	fixed	Document Archiver Extension	" * Create page on http://wiki.ckan.org/ about setting up a queue (this will merge into core docs when this is stable) - having a queue will now become a generic requirement for a bunch of extensions
 * README with install instructions in ckanext-archiver
 * Create an entry in http://wiki.ckan.org/Extensions
 * Write a blog post"	1318600705000000	1331544816000000
1420	enhancement	johnglover	johnglover	ckan-sprint-2011-11-21	closed	fixed	Refactor ckanext-qa to work as a celery task	"Currently can run as a celery task launched by paster command. 

Still have to add a CKAN plugin so that QA will run when a resource is added or URL changed, and then update the extension controller/templates to look for QA data from the task_status table instead of directly on the resource."	1319626138000000	1320843789000000
1436	defect	johnglover	johnglover	ckan-sprint-2011-11-07	closed	fixed	Packages with a 'pending' state not listed on groups page		1320155227000000	1320243278000000
1449	enhancement	johnglover	johnglover	ckan-sprint-2011-11-21	closed	fixed	Improved resource listing on dataset view page	Should list resources in the form of search results rather than as a table.	1320677693000000	1321873008000000
1451	enhancement	johnglover	johnglover	ckan-sprint-2012-01-09	closed	fixed	Reintegrate download stats on dataset and resource view page - 0.5d	" * css class: resource-url-analytics
 * Assign to the link tag (a) everywhere we want to count (dataset, resource view) - 0.25d
 * Display counts in same place ...

And deploy on http://thedatahub.org/ - 0.25d

Possible: Also move analytics extension into core (decided not to)."	1320677859000000	1324401792000000
1455	defect	johnglover	dread	ckan-sprint-2011-12-05	closed	fixed	Search results when 'all_fields' don't include 'extra' fields	"When you do a search like this:
{{{
http://thedatahub.org/api/search/package?q=tauberer+census&all_fields=1
}}}
the ""extra"" fields (e.g. ""triples"", ""shortname"") get missed off the results. The docs say it should be a ""full record"" and I don't see any reason why this is missed off.

This is a problem because search all_fields is the only way for clients and front-ends to get packages in bulk. They end up (like lodcloud) doing thousands of requests to get packages individually.

The full record is:
{{{
http://thedatahub.org/api/rest/dataset/2000-us-census-rdf
{""count"": 1, ""results"": [{""res_description"": [""Download"", ""XML Sitemap"", ""SPARQL enpdoint"", ""Example (RDF/XML)""], ""name"": ""2000-us-census-rdf"", ""license"": ""Non-OKD Compliant::Creative Commons Non-Commercial (Any)"", ""author"": ""Joshua Tauberer"", ""author_email"": ""http://razor.occams.info/"", ""ckan_url"": ""http://thedatahub.org/dataset/2000-us-census-rdf"", ""notes"": ""2000 U.S. Census converted into over a billion RDF triples.\n\nPopulation statistics at various geographic levels, from the U.S. as a whole, down through states, counties, sub-counties (roughly, cities and incorporated towns)\n\nNotes: also found in the of SPARQL Endpoints.\n\nFrom home page:\n\n> * For the detailed Census statistics, you'll have to download the raw Census data files from the Census Bureau, my Perl script and the patch file below and run it yourself because the files are too big for me to offer as a download!\n> \n> * The data and scripts can be reused under Creative Commons Attribution-NonCommercial-ShareAlike.\n"", ""entity_type"": ""package"", ""site_id"": ""www.ckan.net"", ""download_url"": ""http://www.rdfabout.com/demo/census/"", ""indexed_ts"": ""2011-11-01T12:52:36.034Z"", ""url"": ""http://www.rdfabout.com/demo/census/"", ""state"": ""active"", ""title"": ""2000 U.S. Census in RDF (rdfabout.com)"", ""groups"": [""lod"", ""lodcloud""], ""res_format"": ["""", ""meta/sitemap"", ""api/sparql"", ""example/rdf+xml""], ""license_id"": ""cc-nc"", ""revision_id"": ""fcbad0de-79ea-41bd-8e01-eb832a05b732"", ""res_url"": [""http://www.rdfabout.com/demo/census/"", ""http://www.rdfabout.com/sitemap.xml"", ""http://www.rdfabout.com/sparql"", ""http://www.rdfabout.com/rdf/usgov/geo/us/ny""], ""id"": ""551ec435-f198-4d52-9b56-ec0b0be6aec9"", ""tags"": [""census"", ""data"", ""demographics"", ""deref-vocab"", ""format-dc"", ""format-geonames"", ""format-politico"", ""format-rdf"", ""geographic"", ""linkeddata"", ""lod"", ""lodcloud.nolinks"", ""no-license-metadata"", ""no-provenance-metadata"", ""no-vocab-mappings"", ""population"", ""published-by-third-party"", ""rdf"", ""statistics"", ""us""]}]}
}}}"	1320858265000000	1324474466000000
1468	defect	johnglover	thejimmyg	ckan-sprint-2011-12-05	closed	fixed	Update the search index for individual packages	"When I delete a package from the pylons shell I need to remove that package's entry from the Solr search index. That doesn't happen at the moment.

It would be useful if the search-index rebuild command could take a list of package IDs to rebuild."	1321376195000000	1322591997000000
1481	defect	johnglover	dread	ckan-sprint-2011-12-05	closed	fixed	License not shown if no resources	"The license is not displayed any more, unless there are
resources e.g.
http://test.ckan.net/dataset/dtest#section-basic-information is OGL,
but that's not shown."	1321983266000000	1323169343000000
1482	defect	johnglover	zephod	ckan-sprint-2011-12-05	closed	fixed	Minor JS bugs	"Three JS bugs have appeared:

* The slug editor is not updating the text field again. [Fixed; commit:fadda17383]

* The View Dataset page will stop executing javascript if there are no resources. [Fixed; commit:e54b089f6e]

* The Add-Dataset and Add-Group URL editor shows the wrong URL for datasets and groups. Broken in this commit:
https://github.com/okfn/ckan/commit/5a01e67
by 'map.explicit=true'. JohnG: Can you look into this? [Fixed; commit:88e88380f6]"	1321991091000000	1322062576000000
1483	defect	johnglover	johnglover	ckan-sprint-2011-12-19	closed	fixed	Tasks sometimes fail to update task_status table when trying to report a failure	"The error reports do not have a ""value"".  The original task in the task status table is not made and it tries to make a new task, but can't.

Issue: why is the original task_status entry not made?

kindly: there is a chance of course that the task is quicker than the plugin to get to the point where it needs to update the table.

kindly: so we may have to find a way to update the task status before we run the task.

kindly: I think you can supply a task_id to celery. i.e we make our own guid as the task id. So we can use that one to populate the task status.

"	1322050605000000	1323279151000000
1501	defect	johnglover	amercader	ckan-sprint-2011-12-05	closed	fixed	Pagination links broken	"Most of the different pagination links are broken and point to wrong locations. e.g:

* Dataset listings in group page:
http://test.ckan.net/group/bibliographic

* Tag letters in tag listing:
http://test.ckan.net/tag

* User listing:
http://test.ckan.net/user

* Revision listing:
http://test.ckan.net/revision

This is most probably related with:

https://github.com/okfn/ckan/commit/1772a5c

"	1322646397000000	1322684411000000
1502	defect	johnglover	johnglover	ckan-sprint-2011-12-05	closed	fixed	Group package list is ordered by revision timestamp instead of alphabetically		1322680312000000	1324480415000000
1519	enhancement	johnglover	shevski	ckan-sprint-2012-01-09	closed	wontfix	combine stats and analytics extensions into one in UI as well as deployment	"Makes more sense to only have one comprehensive stats/analytics extension, so when people are looking to add a stats extension they won't have to add two which may be confusing (is one an old version of the other? why do I have to add two? what's the difference? etc)

User-wise we need a way to display our stats with google analytics in the same place"	1323169033000000	1324317373000000
1543	defect	johnglover	amercader	ckan-sprint-2012-01-09	closed	fixed	Pagination links in the dataset listings don't keep the current filters	"E.g.
Pagination links on this page don't include groups=lodcloud
http://thedatahub.org/dataset?groups=lodcloud

Not sure if related to #1501 (probably not)"	1323442623000000	1324483367000000
1546	defect	johnglover	johnglover	ckan-sprint-2011-12-19	closed	fixed	Package metadata_created returns the wrong date	metadata_created seems to be returning the date of the earliest revision (for all packages), rather than the earliest revision for a specific package.	1323707327000000	1324034396000000
1580	enhancement	johnglover	johnglover	ckan-sprint-2012-01-09	closed	fixed	Documenting TaskStatus table and QA changes - 0.5d	" * Write up: task status table in  http://wiki.ckan.org/Domain_Model
 * Write up: http://wiki.ckan.org/Data_Quality to reflect current state of play (move all old material to ""Discussion / Talk"" page)
 * Ping ckan-discuss re new Data Quality page"	1324399664000000	1324478635000000
1582	enhancement	johnglover	johnglover	ckan-sprint-2012-01-09	closed	fixed	Deploy QA for thedatahub - 0.5d	" * deploy celery
 * deploy QA and archiver tasks
 * write up a blog post announcing QA on thedatahub"	1324458494000000	1326110801000000
1584	enhancement	johnglover	johnglover	ckan-backlog	new		QA report improvements - 2.5d	"Super: #1594

 * qa/{username}
 * qa/{groupname}
 * paginate QA results
 * search / filter QA results
 * ~~list organisation report by default, but can disable via config option~~ (done)
 * UX tidy up of report pages - hide border if no sidebar, etc"	1324459433000000	1338981975000000
1587	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	QA - Check license as part of 5 star rating	"Super: #1594

From Richard Cyganiak on the CKAN discuss list:

According to (most of) the definitions of the five-star scheme, the first star already requires an open license, so a dataset shouldn't get any stars at all if it's not under an open license."	1324480208000000	1339433296000000
1588	enhancement	johnglover	johnglover	ckan-backlog	new		QA - Give SPARQL endpoints a 4 star rating	"Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Besides considering the media type of resources, it would also make sense to check for the presence of a SPARQL endpoint. SPARQL endpoints are recorded for more than 300 datasets on the Data Hub using the pseudo-type ""api/sparql"". A few more are recorded with the format ""SPARQL"". I suggest that datasets with such resources should also be considered for the fourth star."	1324480405000000	1325475178000000
1589	enhancement	johnglover	johnglover	ckan-backlog	new		QA - Give 5 star rating to datasets with link metadata	"Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Regarding the fifth star (is the dataset linked to others?). This cannot be automatically determined just by looking at the format. It either requires inspection of the actual data, or information about links in the metadata. As you're probably aware, we've established conventions for recording information on data links in CKAN [1], as part of the work of the lodcloud group on the Data Hub. Link information is captured for hundreds of datasets. I would claim that we have the majority of four-star datasets covered there, and hence you can determine if they should get the fifth star by checking for the presence of a links:xxx field."	1324480600000000	1325475095000000
1594	enhancement	johnglover	johnglover	ckan-v1.7	closed	wontfix	[super] QA Improvements	"Now that QA has been updated for Celery and CKAN 1.5.1, there are a number of improvements that we should probably implement.

 * Integrate QA into frontend - 1d - #1583
 * QA report improvements - 2.5d - #1584
 * QA - Check license as part of 5 star rating - #1587
 * QA - Give SPARQL endpoints a 4 star rating - #1588
 * QA - Give 5 star rating to datasets with link metadata - #1589

Main pad: http://ckan.okfnpad.org/qa

Some QA discussion (relating to Celery in particular) at http://ckan.okfnpad.org/queue"	1324489620000000	1338203215000000
1622	enhancement	johnglover	johnglover	ckan-sprint-2012-01-23	closed	fixed	Deploy QA on DGU UAT test server - 0.5d	" * Update CKAN on DGU UAT to 1.5.1
 * Deploy Celery
 * Deploy QA extension"	1326116380000000	1326127702000000
1651	enhancement	johnglover	dread		closed	fixed	Explicit link mapper	"In this commit https://github.com/okfn/ckan/commit/1772a5c John Glover set map.explicit=True in ckan/config/routing.py.

The reason this was done was to avoid links collecting parameters.
e.g. if you were on page /dataset/{id}/resource/{resource_id} then by default all the links on that page generated by url_for (Routes) would include the id and resource_id parameters as well. To avoid this, you had to go through all the links and add id=None and resource_id=None to the url_for parameters.

When map.explicit was changed to True, the value of the controller, action, id and any parameters were no longer automatically carried over into the generated links for the page. So previously links within the same controller didn't need to specify the controller (for example), but now they did. So when we did this we also had to fix up links that weren't explicit:

 * https://github.com/okfn/ckan/commit/88e88380f61965f58fb105095e76a78673ce3e2c
 * #1501 Pagination links
 * https://github.com/okfn/ckan/commit/52d71102d9b07693ea240d4a66e8adb43e515414

John made the config change on 5/11/2011 which was merged to master https://github.com/okfn/ckan/commit/5a01e67 21/11/2011. The related fixes mentioned were in within the same week. This all went into release 1.5.1. The requirement"	1326709852000000	1326711005000000
1696	defect	johnglover	johnglover	ckan-sprint-2012-02-06	closed	fixed	Maintain backwards compatibility with older way of creating custom forms	"To maintain backwards compatibility, the package controller (new/edit) should check to see if the controller has a package_form variable defined, and if so render the form pointed to by this variable before calling the new self._package_form() function.

This behaviour is now deprecated however."	1327326243000000	1327326421000000
1699	task	johnglover	kindly	ckan-sprint-2012-02-06	closed	fixed	Setup Ckan on tenforce test server	Need to setup server in tenforce test server.	1327417859000000	1327425070000000
1700	enhancement	johnglover	kindly	ckan-sprint-2012-02-06	closed	fixed	Document server setup for ckan server.	Document deployment on tenforce test server.	1327417952000000	1327425247000000
1703	enhancement	johnglover	kindly	ckan-sprint-2012-03-19	closed	fixed	Make custom fields for ecportal form based on prototype metamodel	The prototype portal gave us a list of classifiers that are expected for the initial launch.  Add then to ckanext-ecportal.	1327419727000000	1331142926000000
1704	enhancement	johnglover	kindly	ckan-sprint-2012-02-06	closed	fixed	Import test datasets from prototype into ckan.	Eurostat datasets mainly.	1327420355000000	1328005759000000
1707	defect	johnglover	johnglover	ckan-sprint-2012-02-06	closed	duplicate	Over-long URLs in QA ext broken links list	"From Mark on the ckan-dev list:

In the table of broken links at
<http://thedatahub.org/qa/dataset/broken_resource_links>, the links as
printed should be abridged to a fixed length (but link to the full
URL). Otherwise the 'reason' column vanishes off the screen. Also a
table would probably be better since as things stand the 'reasons' are
not lined up"	1327488793000000	1328024253000000
1720	enhancement	johnglover	johnglover	ckan-sprint-2012-02-20	closed	fixed	Add converter functions to convert tags (with a given taxonomy) to/from tags field	Part of #1698	1327943471000000	1328625423000000
1721	enhancement	johnglover	seanh	ckan-sprint-2012-02-20	closed	fixed	Tag taxonomies table migration		1327944237000000	1329133348000000
1729	enhancement	johnglover	seanh	ckan-sprint-2012-02-20	closed	duplicate	Helper function for extensions to add vocabularies to a ckan instance	Add a helper function to make it easy for extensions to add new vocabularies to default_package_schema(). The helper function should take the name or ID of a vocabulary (which should already exist in the db) and add the necessary key: [schema] entry to default_package_schema(), with the necessary validation, authorisation, transformation.	1328007723000000	1329131023000000
1730	enhancement	johnglover	seanh	ckan-sprint-2012-02-20	closed	fixed	Form field for vocabularies	A function that takes a vocabulary name or ID as argument and returns a nice select box for selecting items from that given vocabulary. Meant to be used by form templates, to make it easy for them to integrate custom vocabularies. Could use http://harvesthq.github.com/chosen/	1328007897000000	1328714937000000
1732	enhancement	johnglover	seanh	ckan-v1.8	closed	fixed	Update CKAN's search for taxonomies	"Initially we'll just add all tags from all vocabularies into the existing tags search facet (this should happen automatically). Later we want to enable templates to use separate search facets for different vocabularies.

Also need to make sure that search terms are matched against the texts of the tag names themselves (should happen already) and maybe also against the texts of the vocabulary names?"	1328008169000000	1341224357000000
1766	enhancement	johnglover	johnglover	ckan-sprint-2012-02-20	closed	fixed	Update and deploy EC Portal theme		1328537446000000	1328541227000000
1767	enhancement	johnglover	johnglover	ckan-sprint-2012-02-20	closed	fixed	Write end-to-end functional tests for tag vocabularies	"Need to test:
 
 * API call to create new vocab
 * API call to add tags with a given vocab
 * API call to update tags
 * Dummy plugin to test that updating tags works via WUI"	1328537586000000	1329326678000000
1809	enhancement	johnglover	johnglover	ckan-sprint-2012-03-05	closed	fixed	Catch request exceptions in archiver link_checker task	Some request exceptions are currently not being caught (see the celery log on thedatahub for examples)	1329746267000000	1330528828000000
1810	enhancement	johnglover	johnglover	ckan-backlog	closed	wontfix	Store list of Eurovoc tag names and IDs	" * Tag list not be added to CKAN core, what is the best way to do this in an extension? 
 * Does the list need to be updated via API/WUI?"	1329746612000000	1335874745000000
1811	enhancement	johnglover	johnglover	ckan-backlog	closed	wontfix	Translate Eurovoc tags from ID to dataset language		1329747684000000	1335874765000000
1812	enhancement	johnglover	johnglover	ckan-backlog	closed	fixed	Restrict editing rights/permissions based on publisher	 * see how this currently works with DGU first	1329747889000000	1335874864000000
1817	enhancement	johnglover	johnglover	ckan-sprint-2012-03-05	closed	fixed	Importer for EC publisher info	" * write importer for publisher info (currently in JSON format), add to ckanext-ecportal
 * add group translations to translations table
 * run on test server"	1329762290000000	1329922818000000
2205	enhancement	johnglover	johnglover	ckan-sprint-2012-03-19	closed	fixed	Document Tag Vocab schema/form setup		1330941067000000	1330956840000000
2206	enhancement	johnglover	johnglover	ckan-sprint-2012-03-19	closed	fixed	Change site header to match latest ODP template		1330958095000000	1331046486000000
2212	enhancement	johnglover	johnglover	ckan-sprint-2012-03-19	closed	fixed	Import latest data into demo site		1331047333000000	1331720191000000
2213	enhancement	johnglover	johnglover	ckan-sprint-2012-04-02	closed	fixed	Update geographical coverage tags	" * ~~create importer for geo tags~~
 * ~~add translations to database~~
 * ~~translate tags in WUI~~
 * ~~test~~
 * ~~deploy~~
 * ~~document how to add a geo coverage field in http://docs.ckan.org/en/latest/forms.html~~"	1331047525000000	1332342835000000
2245	enhancement	johnglover	johnglover	ckan-sprint-2012-04-02	closed	fixed	Update documentation for writing CKAN extensions		1332178546000000	1332331196000000
2246	enhancement	johnglover	johnglover	ckan-sprint-2012-04-02	closed	fixed	Change published_by metadata field to reference group instead of a custom extra	 * probably needs a new converter, as needs to be usable via API as 'published_by'.	1332243036000000	1332864871000000
2248	enhancement	johnglover	johnglover	ckan-sprint-2012-04-02	closed	fixed	Update docs for creating custom forms	Should describe implementing IDatasetForm, subclassing controllers is deprecated	1332328630000000	1332331220000000
2259	enhancement	johnglover	johnglover	ckan-sprint-2012-04-02	closed	fixed	Update ecportal theme for master changes (bootstrap)	"Some recent changes caused things to break in ckanext-ecportal:

 * change from blueprint to bootstrap
 * other style / css changes (eg: tables, resource view notes field ID, etc. Forms are also broken, but that will be addressed in #2262)
 * change to going through db_to_form_schema on package_show"	1332754180000000	1332769973000000
2260	enhancement	johnglover	johnglover	ckan-sprint-2012-04-16	closed	fixed	Update ecportal theme to match latest changes to tenforce theme		1332754319000000	1333554069000000
2262	enhancement	johnglover	johnglover	ckan-sprint-2012-04-02	closed	fixed	Update ecportal form and dataset style/ux to match CKAN master		1332764002000000	1332855605000000
2269	enhancement	johnglover	johnglover	ckan-sprint-2012-04-30	closed	fixed	EC ODP review and server setup	" * code review with D.R. for ckanext-ecportal
 * merge and test all multilingual code
 * fix server issues: routing, Git access to CKAN repository

Bugs to fix:

 * ~~rename groups to publishers~~
 * ~~update home page layout (recent updates along side publishers)~~
 * ~~don't add 'en' to main menu links if english~~
 * ~~resource edit page, fix missing images~~
 * ~~change dataset 'settings' to 'edit'~~
 * ~~translate 'published by' drop-down list~~
 * ~~fix JS on 'add a dataset page'~~
 * ~~prepopulate 'published_by' when clicking 'add a dataset' from group~~
 * ~~chosen broken (dataset edit) if in different language~~

"	1333374177000000	1335779180000000
2270	enhancement	johnglover	johnglover	ckan-sprint-2012-04-16	closed	fixed	DataExplorer overview	 * get up to speed with DataExplorer code (and dependencies like Backbone)	1333374339000000	1334073875000000
2275	enhancement	johnglover	seanh	ckan-sprint-2012-04-16	closed	fixed	Implement group logos	"Groups should be able to specify a URL to an image file for a group logo, the image will be shown on the group page.

~~Actually uploading an image file (rather than giving a URL) and cropping and resizing the file will be done later, for now just give a URL.~~ (later ticket)

~~Further customisation of group pages (CSS, templates) also for later.~~ (not sure if we really want this, but in a later ticket if so)"	1333376076000000	1334594328000000
2306	enhancement	johnglover	johnglover	ckan-sprint-2012-04-30	closed	fixed	Updates to metadata model	"Finalised metadata model requires several changes to our existing model:

 * new extra: alternative title
 * new extra: identifier
 * new extra: interop level
 * new extra: version description
 * new extra: dataset status
 * dataset type: change to fixed vocab
 * resource types: prefix with documentation/distribution/etc for clarity, add new resource types
 * rename update frequency to accrual periodicity
 * add new fixed vocab field: language coverage
 * resources: missing issued date - add as new core field called 'created'

Also add functional tests for full metadata model."	1334588086000000	1335795725000000
2338	enhancement	johnglover	johnglover	ckan-sprint-2012-05-15	closed	wontfix	Help with setup of new test instance for EC ODP (running on RedHat)	I won't be needed for this during the current sprint, closing.	1335875955000000	1336491648000000
2343	enhancement	johnglover	johnglover	ckan-sprint-2012-05-15	closed	fixed	Convert all non-free metadata fields to vocabulary fields	" * convert non-free fields to vocab fields (eg: interoperability_level)
 * store values as JSON files in the ecportal repo
 * translate output on dataset view page
 * add tests"	1335876562000000	1335969506000000
2346	enhancement	johnglover	johnglover	ckan-sprint-2012-05-15	closed	fixed	EC ODP Theme updates	"Changes:

 * tidy up activity feed on home page or replace with search results (by modified date)
 * ~~move ckan menu to a left sidebar (see Sam's template)~~ (not doing this now)
 * update templates with changes from master
 * use local image in language dropdown (ecportal.js)
 * update dataset read template


To discuss:

 * should we rename 'organization' to 'publisher'? Requires updating all templates and the url mapping."	1335876838000000	1338205553000000
2459	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Faceting improvements: fix tags and decide on facet fields	" * the facet tags should only show free (non-vocab) tags
 * decide on what fields to include in search facet list"	1338211925000000	1338393403000000
2460	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Hide private datasets in 'top publishers' and 'recent updates' on home page		1338212018000000	1338301492000000
2462	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	duplicate	Add converter to rename resource_type field to type		1338212065000000	1338309596000000
2463	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Fix whitespace bug in breadcrumbs when zoomed in (Google Chrome)		1338212133000000	1338307011000000
2464	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Rename 'organization' to 'publisher'	" * Already done for basic publisher forms (index, read, edit), but needs to be done for additional forms (in particular, form paths like 'organization_apply' that are hardcoded into the organizations controller).
 * change organization mapping to /publisher"	1338212553000000	1338910189000000
2465	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Make sure dropdown lists on dataset edit form are sorted		1338212671000000	1338805853000000
2470	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Fix inconsistency in package_list snippet	Datasets listed in search results have not gone through schema validation (but on the user page they do), so package_list snippet currently has to display both 'notes' and 'description'.	1338220927000000	1338284041000000
2487	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Don't preview text/html	Quite a few of the current test dataset resources have their format incorrectly specified as text/html. Attempting to preview these results in the browser trying to automatically download the resource, so for now we will not preview this format.	1338806098000000	1338892521000000
2488	defect	johnglover	johnglover	ckan-sprint-2012-06-25	closed	worksforme	Group update via API fails validation with organization and publisher profile enabled	"Reported on IRC by 'floapps'.

 * when posting a group update with {packages: [package-1, package-2, etc.]}, getting ValidationError: {'Packages': {'name': ['Missing value']}}, although package objects did contain a name field
 * same API call worked with organization and publisher profile disabled
 * also worked if added a form_to_db_schema_options function which used default_update_group_schema for api updates instead of default_group_schema
 * tried on both 1.7 and 1.7.1b"	1338806374000000	1339412538000000
2489	defect	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Select publisher when dataset created via 'publisher > add dataset'	When clicking 'add dataset' on a publisher/group page, the new dataset form should select the appropriate group.	1338809595000000	1338886578000000
2509	defect	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	UI style broken on organization read page	Looks like some of the style has been updated/changed on the group read page but the changes haven't been applied to the organizations read page.	1339413084000000	1339419867000000
2517	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Test installation instructions for EC ODP on CentOS 6.2	 * created repo for final list of instructions: http://github.com/okfn/ecportal-server-setup	1339581689000000	1340613194000000
2518	enhancement	johnglover	johnglover	ckan-sprint-2012-06-25	closed	fixed	Create EC ODP CKAN release branch	" * branch from 1.7.1
 * apply CKAN patches that are current part of installation instructions
 * create i18n patch so custom i18n directories can be specified via a config option"	1339581818000000	1339595863000000
2551	enhancement	johnglover	ross	datahub-oct	closed	fixed	Feature metrics	"
== Requirements ==

Would be useful if we could extend the Google Analytics extension to push track events to GA so that we can determine how the system is being used, and where we can optimise the workflow.

Assuming it isn't configured to be off, track events should be added to some key action buttons so that the information on what users are doing is logged for analysis.


== Interface ==

This could possibly just be a list of JS calls to $('x').click(_gaq.push('track'...))


== User Stories ==

 * As a developer I would like to know which components of the system are used the most so I can streamline the processes.

 * As a support developer I would like to know which parts add complexity to the system for minimal gain and could possibly be removed, de-prioritised.

For which metrics, as many of the following as possible that are achievable without changing the templates. I think for now it is enough to just record counts of these pages/actions.

 * Any group link on homepage
 
 * Clicking on username (to go to profile)
 
 * In user profile, clicking on Edit Profile
 
 * Clicking Save Changes on Edit Profile page
 
 * Clicking on any dataset link on User Profile page
 
 * Any of the group links on /group
 
 * Clicking any of the right hand sidebar tags on /group/X
 
 * Visiting /group/history/X 
 
 * Compare Button on /group/history/X
 
 * Compare Button on /dataset/history/X  
 
 * Tags on right hand sidebar of /dataset/X   
 
 * Download button on any /dataset/X/resource/* page 
 
 * Data API button on any /dataset/X/resource/* page


== Tasks ==

[ ] Analysis/Clarification

[ ] Code/Config changes

[ ] Documentation


== Estimates ==

"	1340017689000000	1343042712000000
2593	enhancement	johnglover	johnglover	ckan-ecportal	closed	fixed	Setup nginx caching for EC ODP	Add caching config to ecportal-server-setup repository.	1340638428000000	1340721249000000
2594	enhancement	johnglover	johnglover	ckan-ecportal	closed	fixed	Add EU27 to EC ODP geographical coverage field	EU27 option seems to have been removed at some point, needs to be added back to geographical coverage vocabulary and tested.	1340638576000000	1340716640000000
2645	enhancement	johnglover	johnglover	ckan-ecportal	closed	fixed	Update server setup to use local files	"Server setup scripts need to have two distinct steps:

1) Download all files that aren't in RHEL repositories, create RPMs where necessary (python virtualenv).

2) Install from local files."	1341916281000000	1341996302000000
2733	enhancement	johnglover	johnglover	ckan-v1.9	new		Datastore logic functions	"== Where does the data go? ==
In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url. 

The user should have rights to create tables.

== Whats the api like? ==
We will just implement it as logic functions like the rest of CKAN and will part of core.  After that we may add some nicer api functions that use these but that is a secondary concern.

== What are the initial logic functions? ==
 * datastore_create
 * datastore_delete
 * datastore_show

== What is the JSON input format for datastore_create ==
To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers.  Except adds resource_id.

{{{
{
resource_id: resource_id # the data is going to be stored against.
fields: a list of dictionaries of fields/columns and their extra metadata.
records: a list of dictionaries of the data eg  [{""dob"": ""2005"", ""some_stuff"": ['a', b']}, ..]
}
}}}

 * The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
 * rows: can be empty so that you can just set the fields
 * fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.  
eg:  [{""id"": ""dob"", ""type"": ""timestamp"" }, {""id"": ""some_stuff"", ""type"": ""text""},  ...].
A header items values can not be changed after it has been defined nor can the ordering of them be changed.  They can be extended though.
 * Any error results in total failure!! For now pass back the actual error.
 * Should be transactional

== What json does datastore_delete take? ==

{{{
{
resource_id: resource_id # the data is going to be deleted.
filters: dictionary of matching conditions to delete
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to ""delete from table where key1 = 'a' and key2 = 'b' "".
    No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table.
}
}}}

== What json does datastore_search take? ==
{{{
{
resource_id: resource_id # the data is going to be selected.
filters : dictionary of matching conditions to select
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to ""select * from table where key1 = 'a' and key2 = 'b' ""
q: full text query
limit: limit the amount of rows to size default 100
offset: offset the amount of rows
fields:  list of fields return in that order, defaults (empty or not present) to all fields in fields order.
sort: comma separated field names with ordering e.g ""fieldname1, fieldname2 desc""
}
}}}

Some free code: https://gist.github.com/3163864

== What json does datastore_search return? ==
{{{
{
fields: same type as datastore_create accepts (i.e. with metadata)
offset: The same offset that was supplied in datastore_show
limit: The original limit
filters: The filters that were applied in data_show
total: # total matching records without size or offset
records: [same as data_create] # list of matching results
}
}}}

On error will return:
{{{
{
__error__: … sql error …
}
}}}

== What types are allowed? ==
Aim to support as many postgres/postgis types that have string representations.

http://www.postgresql.org/docs/9.1/static/datatype.html

http://www.postgresql.org/docs/9.1/static/sql-createdomain.html 

== IDs ==
Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

== Other Features ==
Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc."	1343058886000000	1343656105000000
2829	enhancement	johnglover	toby	ckan-v1.9	new		Archiver fails on 403 http response	"Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end

{{{
$ paster archiver update -c ../ckan/development.ini 


2012-08-09 11:01:37,636 INFO  [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources)
2012-08-09 11:01:37,671 INFO  [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13
2012-08-09 11:01:37,900 INFO  [ckan.lib.base]  /api/action/package_show render time 0.043 seconds
Traceback (most recent call last):
  File ""/home/toby/okfn/pyenv/bin/paster"", line 8, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 104, in run
    invoke(command, command_name, options, args[1:])
  File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 143, in invoke
    exit_code = runner.run(args)
  File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 238, in run
    result = self.command()
  File ""/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py"", line 98, in command
    response = app.post(api_url + '/package_show', data)
  File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 262, in post
    expect_errors=expect_errors)
  File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 243, in _gen_request
    return self.do_request(req, status=status)
  File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 406, in do_request
    self._check_status(status, res)
  File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 439, in _check_status
    res.body))
paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show)
{""help"": ""Return the metadata of a dataset (package) and its resources.\n\n    :param id: the id or name of the dataset\n    :type id: string\n\n    :rtype: dictionary\n\n    "", ""success"": false, ""error"": {""message"": ""Access denied"", ""__type"": ""Authorization Error""}}
}}}"	1344508484000000	1344508484000000
3011	enhancement	johnglover	johnglover	ckan 2.0	new		Recline fixes and updates for CKAN 2.0	Fixes and updates to Recline / Datapreview for CKAN 2.0.	1352288455000000	1352288455000000
3017	enhancement	johnglover	johnglover	ckan 2.0	new		New stable branch for the demo server		1352819088000000	1352819088000000
5	enhancement	johnbywater	johnbywater	v0.3	closed	fixed	Browse list of packages and select one to view	"== As a ==

Visitor

== I want to ==

Browse a list of packages resulting from a search or browse request (see other use cases)

== So that ==

I can select one of the packages to view in more detail (-> viewing an individual package ticket:6)

== Notes ==
  * When browsing a list of packages you should be able to see summary information about the package such as title (though this may be shortened in order to conveniently fit the list 
  * The list should be broken up into pages so that the number of packages per page should be kept to a reasonable number (<= 50). Response time should be kept reasonable
"	1152549884000000	1185473622000000
7	enhancement	johnbywater	johnbywater		closed	invalid	View a release associated with a package	See Update a release (ticket 14) for more information on what should be shown.	1152550022000000	1204136209000000
8	enhancement	johnbywater	johnbywater		closed	invalid	View file	"== As a ==

Visitor

== I want to ==

View the files associated with a package.

== Notes ==

  * See also ticket:16 (updating a file).
  * Files should either be listed on the same page as a release or there should be a link from the release page to a page listing all the files associated with the release. See also ticket:14 (update a release)
  "	1152550057000000	1250619147000000
10	enhancement	johnbywater	johnbywater	milestone15	closed	worksforme	Update package	"== As a ==

Authenticated user

== I want to ==

Update the attributes of a package and add/remove releases

== So that ==

The package attributes are changed. Package attributes are:
  * A title
  * A description (short or long)
  * Date added
  * Project url
  * package license(s)
  * download url
  * A list of releases with their dates and a summary line
  * a list of owners (referencing authenticated users)

I would also like to be able to add comments

== Notes ==

  * Only an authenticated user is one of owners of a package may update it
  * Any authenticated user may add comments (but visitors may not)
  * can break down into:
   * Update a package by changing simple package attributes (value: 8)
   * Update a package by adding or removing an owner (value: 5)
   * Update a package by adding or removing comments (value: 7)
"	1152550243000000	1183648150000000
11	enhancement	johnbywater	johnbywater	milestone16	closed	fixed	Delete package	"== As a ==

Authenticated user

== I want to ==

Delete a package

== So that ==

So that the package is no longer displayed for visitors of the site

== Notes ==

  * The permissions for this action are the same as for updating the package details (see ticket:10)
  * When a package is deleted it is put in an inactive state and is invisible to users and vistors of the service. However its name remains reserved and it may be undeleted or purged by the service administrator
"	1152550328000000	1157371568000000
12	enhancement	johnbywater	johnbywater		closed	invalid	Create a release associated with a package	"== As a ==

Authenticated user

== I want to ==

Create a release associated with a package

== So that ==

The package has a new release

== Notes ==

  * Permissions are as for updating a package ticket:10
  * for details of release attributes see: updating a release ticket:14
"	1152550407000000	1204135071000000
14	enhancement	johnbywater	johnbywater		closed	invalid	Update a release	"== As a ==

Authenticated user

== I want to ==

Update a release

== So that ==

The release details are amended. Attributes of a release are:
  * name (may include '.' and '-'
  * nickname
  * date
  * description

== Notes ==
  * permissions: as for updating a package (ticket:10)"	1152550712000000	1204136163000000
15	enhancement	johnbywater	johnbywater		closed	invalid	Create a file	"== As a ==

Authenticated user

== I want to ==

Create a file associated with a release

== So that ==

The release has a new file associated with it

== Notes ==
  * Permissions are as for package updating (see ticket:10)
  * file attributes are described in file updating: ticket:16)
    * there are no attributes that are required to be supplied to create a file
"	1152550770000000	1250619139000000
16	enhancement	johnbywater	rgrp		closed	invalid	Update a file	"== As a ==

Authenticated user

== I want to ==

Update a file associated with a release

== So that ==

The file attributes are changed. File attributes are:
  * (internal) id
  * description
  * source: (url usually)

== Notes ==
  * Permissions are as for package updating
"	1152550796000000	1250619162000000
17	enhancement	johnbywater	johnbywater		closed	invalid	Delete a file	See ticket:16 (update a file)	1152550822000000	1250619180000000
20	enhancement	johnbywater	johnbywater	v0.3	closed	fixed	Purge a package	"== As A ==

Administrator

== I want to ==

Purge a deleted package

== Notes ==

  * this need only be possible via the admin interface"	1152550910000000	1185473187000000
25	enhancement	johnbywater	johnbywater	v0.6	closed	fixed	Autocompletion (and suggestion) of package tags when adding tags to a package	"== Notes ==

  * Currently have a text box where users can add space separated tags
  * Tags should auto-complete in a manner similar to delicious
  * would also be good to have a list of suggested tags. This could either be:
    * [simpler] a list below the tag box chosen by existing popularity  or
    * [more complex] a list of suggestions that updates as you add tags (based again on popularity but 'filtered/reranked' to reflect choices so far).
    * For first attempt to should probably just go for the simpler option."	1152551313000000	1204131522000000
27	enhancement	johnbywater	johnbywater	v0.6	closed	fixed	Register/Update a package via a web-api	"= As A =

User

= I want To =

Create/register and update a package via a web api."	1152551379000000	1215543933000000
39	enhancement	johnbywater	rgrp	v0.7	closed	fixed	When listing packages show more information than just name.	"When a package listing is shown (as on http://ckan.net/package/list) more information than just name should be shown. For example package titles should be shown as well as names (could also list tags and other stuff but simplest is just title).

Cost: 1"	1194608920000000	1223908298000000
44	enhancement	johnbywater	rgrp	v0.7	closed	fixed	Provide RSS/Atom Feed of Repository History	"= As A =

Visitor

= I Want To =

Get an RSS/Atom Feed of the Repository History to use in my feed reader (or elsewhere).

= Details =

  * Preference for Atom.
  * should just add parameter to /revision/list/ (or /revision/) to select atom format e.g. ?format=atom.
  * should have a 'days' attribute specifying number of days back to go e.g. &days=30

= Cost =

Low"	1199788457000000	1223390660000000
54	enhancement	johnbywater	rgrp	v0.7	closed	fixed	Support dump and load of CKAN data to JSON	"= As a =

Sysadmin

= I want to =

Dump (serialize) CKAN data to a simple transport format (e.g. JSON) and be able to load it again.

= Details =

  * Already have simple db dump. However dumping to JSON has various advantages, particularly where changes to the data need to be made upon reloading (e.g. during a migration).
  * Dump should include *all* CKAN data (i.e. all the data in the CKAN db tables)"	1223907883000000	1230211256000000
59	enhancement	johnbywater	rgrp	v1.0	closed	fixed	Provide a basic guide to CKAN on the site	"  1. Use contents from ckan package notes section (rgrp)
  2. Update that section to be more guide-like (jwyg)"	1239018583000000	1273080019000000
156	enhancement	johnbywater	dread	v1.0	closed	fixed	Whitelist for html in mark-down field	Not just links that need to be displayed. Other html should be displayed too, such as tables which can't be represented in markdown. Do a search of the existing fields to see what is needed.	1255623973000000	1271962871000000
210	enhancement	johnbywater	johnbywater		closed	invalid	User registration	"Native registration scenario: Complete registration form, require unique email address (indicate this not displayed on site), full name (indicate this is displayed on site), password. Dispatch email confirmation message. Respond to email confirmation message causes user to be logged in. Users are assigned a uuid.
"	1260292578000000	1266509788000000
211	enhancement	johnbywater	johnbywater		closed	invalid	User login	"Native login scenario: Complete login form, require email address. Any session is resumed.

Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from BlastOff.

See CKAN or Microfacts for usage of repose with openid."	1260292624000000	1266509848000000
236	enhancement	johnbywater	johnbywater		closed	fixed	The system shall present to all users a form for starting data openness enquiries		1264157100000000	1264157324000000
237	enhancement	johnbywater	johnbywater		closed	fixed	The system shall accept submissions from authenticated owners of activated accounts using the data openness enquiry form by presenting the enquiry summary and prompting for confirmation		1264157377000000	1264157420000000
238	enhancement	johnbywater	johnbywater		closed	invalid	The system shall accept submissions from unauthenticated users of the data openness enquiry form by creating an anonymous pending enquiry action, and by redirecting the user to login form, passing a reference to the pending action		1264157814000000	1266509806000000
240	defect	johnbywater	rgrp	v1.0	closed	fixed	Do not allow creation of PackageResource object without a url	Currently it is possible (v0.11) to have a PackageResource object without a url. This should be impossible to do (either at creation time or via modification).	1265399747000000	1272383770000000
251	enhancement	johnbywater	johnbywater	v1.0	closed	fixed	Present in the API a list of revision IDs, and present individual revisions when referenced by ID	"Want to have:

 * /api/search/revision/

Support querying by ""since time"" and ""since revision"". Each individual revision should be available in the REST API (read-only):

 * /api/rest/revision/{revision ID}
   * should include revision attributes plus revision.packages e.g. like { ""id"": ""..."", ... ""packages"": [ list of packages names from e.g. revision.packages ] }

"	1266519743000000	1271423917000000
263	enhancement	johnbywater	rgrp	v1.1	closed	fixed	Improve and test openid login	"At the moment the user has to figure out to login with providers -- this is not always obvious (e.g. google).

We could improve this with a small bit of javascript. As an example see: http://standalone.demo.civicrm.org/standalone/

Not sure exactly what library that is using (or if bespoke) so alternatives include (NB: we already use jquery):
  * http://code.google.com/p/openid-realselector/
  * http://code.google.com/p/openid-selector/

In addition to implementing this we also need to test logging in with main providers: google, wordpress -- as we have had reports of it not working with e.g. wordpress (not sure if this testing can be automated really -- best hope would be selenium I think -- so OK if done by hand).

Cost: 3h (2h UI), (1h testing)"	1267605039000000	1279292730000000
265	enhancement	johnbywater	dread	v1.0	closed	fixed	More detail shown in Atom feed	"== Use case ==

As a user I want to stay abreast of package changes, such as a new package being created, newer data is available for a package or a new download is available for a package.

== Implementation ==

Add into the Atom feed:
  1. a package is a new one
  2. the resources have been changed
  3. the last_updated field has been changed

"	1267708364000000	1271636891000000
266	enhancement	johnbywater	dread	v1.0	closed	fixed	Package feed	"As a user I want to track changes to a particular package.

Provide an Atom feed, linked from the package read page."	1267716417000000	1271636901000000
267	defect	johnbywater	rgrp	vdm-0.7	closed	fixed	Diffs for object that is created in that revision	Currently this raises an exception because there is no package/package_revision in existence at from_revision.	1268041383000000	1272960518000000
270	enhancement	johnbywater	johnbywater		closed	fixed	Refactor ckan/lib/importer and ckanclient/loader.	"There is code to create packages in-process (ckan/lib/importer) and via HTTP (ckanclient/loaders/base). There is also code to read different kinds of package source (Google Spreadsheets spreadsheet, other spreadsheet).

Propose to refactor all this to have template method in base command class (subclasses may implement command line interface, graphical user interface, or Web interface), where the template method calls on aggregated strategy objects for: reading raw spreadsheet data from spreadsheet application; for inferring CKAN package entities from the raw spreadsheet data; and for writing package entities to CKAN service.

Spreadsheet readers to be factored to allow for variations of spreadsheet application.

Raw data inferrers to allow for variations in spreadsheet cell structure.

CKAN writer strategies to allow for either CKAN client or direct linking.

Actual cases then supported by specifying existing strategies (or defining new ones) in a script that invokes the command class."	1268671804000000	1290596640000000
271	enhancement	johnbywater	johnbywater	v1.0	closed	fixed	The system shall support distributing changes.	See DistributingChanges for analysis and functional requirements.	1268672137000000	1273596144000000
279	enhancement	johnbywater	dread	v1.0	closed	fixed	Ensure package's license is an option when it's edited	"With forms configurable to show a subset of licenses and ckan instances moving packages from server to server, there is a chance that a user will edit a package and the existing package's license is not displayed in the form. If the user submits this form without touching the license dropdown, then the value is lost. 

So for this case we should add the package's license to the list displayed."	1270205931000000	1272451384000000
284	defect	johnbywater	rgrp	v1.0	closed	fixed	Move tag/read/xxx to simple tag/xxx like package and group	"Also put in redirect from old locations (see examples with plurals in routing.py)

Estimated cost: 20m

Actual cost: 40m"	1270664552000000	1271852451000000
290	defect	johnbywater	dread	v1.0	closed	fixed	Revision API - docs	doc/api.rst needs to cover the new Revision REST interface.	1271268759000000	1271636910000000
293	defect	johnbywater	dread	v1.0	closed	fixed	REST PackageResource update causes exception	"This is caused by editing a package with resources over the REST interface.

Exception:
{{{
URL: http://de.ckan.net/api/rest/package/destatis
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:51 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.rest:145 in update
<<                  fs = ckan.forms.group_fs_combined
                   fs = fs.bind(entity, data=request_fa_dict)
                   validation = fs.validate_on_edit(entity.name, entity.id)
                   if not validation:
                       response.status_int = 409
>>  validation = fs.validate_on_edit(entity.name, entity.id)
Module ckan.forms.package:385 in validate_on_edit
<<              temp_name = orig_pkg_name
                   self.data['Package-%s-name' % record_id] = u'something_unique'
               validation = self.validate()
               if temp_name:
                   # restore it
>>  validation = self.validate()
Module formalchemy.forms:175 in validate
<<          if self.readonly:
                   raise Exception('Cannot validate a read-only FieldSet')
               return AbstractFieldSet.validate(self)
       
           def sync(self):
>>  return AbstractFieldSet.validate(self)
Module formalchemy.forms:104 in validate
<<          success = True
               for field in self.render_fields.itervalues():
                   success = field._validate() and success
               # run this _after_ the field validators, since each field validator
               # resets its error list. we want to allow the global validator to add
>>  success = field._validate() and success
Module formalchemy.fields:763 in _validate
<<          try:
                   value = self._deserialize()
               except validators.ValidationError, e:
                   self.errors.append(e)
>>  value = self._deserialize()
Module formalchemy.fields:1354 in _deserialize
<<          if self.is_composite_foreign_key:
                   return self.query(self.relation_type()).get(python_pk(self.renderer.deserialize()))
               return self.renderer.deserialize()
>>  return self.renderer.deserialize()
Module formalchemy.fields:183 in deserialize
<<          if self.field.is_collection:
                   return [self._deserialize(subdata) for subdata in self._serialized_value()]
               return self._deserialize(self._serialized_value())
       
           def _deserialize(self, data):
>>  return self._deserialize(self._serialized_value())
Module formalchemy.fields:167 in _serialized_value
<<          if self.field.is_collection:
                   return self._params.getall(self.name)
               return self._params.getone(self.name)
       
           def deserialize(self):
>>  return self._params.getone(self.name)
Module formalchemy.base:63 in getone
<<          if v is None or isinstance(v, basestring) or isinstance(v, cgi.FieldStorage):
                   return v
               return v[0]
           def getall(self, key):
               v = dict.get(self, key)
>>  return v[0]
TypeError: 'int' object is unsubscriptable
}}}

Thanks to Friedrich for spotting this."	1271756102000000	1271940083000000
296	enhancement	johnbywater	johnbywater		closed	duplicate	Commit CKAN revisions to changeset system		1272279521000000	1294407032000000
297	enhancement	johnbywater	johnbywater		closed	duplicate	Update CKAN repository from changeset system		1272279556000000	1294407051000000
298	enhancement	johnbywater	johnbywater		closed	duplicate	Pull changesets from remote CKAN instance		1272279591000000	1294407080000000
299	enhancement	johnbywater	johnbywater		closed	duplicate	Merge diverging lines of changesets		1272279698000000	1294407099000000
302	enhancement	johnbywater	dread	v1.0	closed	fixed	Advertise a package's feed visually	"Visual display of a package's feed in the Package Read (/package/xyz) and Package History (/package/history/xyz) pages.

Motivation: each package's Atom feed is mentioned in the <link rel=""alternate"" type=""application/atom+xml"" ...> tag, which Firefox displays with a feed icon, but the other 3/4 of users may not discover it.

The Repo History (/revision) page has a visual link the feed.

PS It would be nice to display the feed icon in both these places too."	1272370397000000	1272453821000000
304	defect	johnbywater	johnbywater	v1.0	closed	fixed	Previewing package with resource with missing url is broken	"Intended outcome:

See resource url validation error when previewing package with a resources that has no url.

Recurring Error:

{{{
URL: http://127.0.0.1:5000/package/new
File 'pyenv-ckan/lib/python2.5/site-packages/weberror/evalexception.py', line 431 in respond
  app_iter = self.application(environ, detect_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/repoze/who/middleware.py', line 107 in __call__
  app_iter = app(environ, wrapper.wrap_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 152 in __call__
  return self.wrap_app(environ, session_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Routes-1.10.3-py2.5.egg/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File 'pyenv-ckan/src/ckan/ckan/lib/base.py', line 50 in __call__
  return WSGIController.__call__(self, environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 216 in new
  clear_session=True)
File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 357 in _render_edit_form
  edit_form_html = fs.render()
File 'pyenv-ckan/src/ckan/ckan/forms/builder.py', line 13 in render
  return render(self.form_template)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render
  format=format, namespace=kargs, **cache_args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render
  **options)
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 533 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate
File 'pyenv-ckan/src/ckan/ckan/templates/package/form_fields.html', line 52 in <Expression u'h.literal(field.render())'>
  ${h.literal(field.render())}
File 'pyenv-ckan/lib/python2.5/site-packages/FormAlchemy-1.3.4-py2.5.egg/formalchemy/fields.py', line 1217 in render
  return self.renderer.render(**opts)
File 'pyenv-ckan/src/ckan/ckan/forms/common.py', line 311 in render
  return render('package/form_resources')
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render
  format=format, namespace=kargs, **cache_args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render
  **options)
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 510 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate
File 'pyenv-ckan/src/ckan/ckan/templates/package/form_resources.html', line 14 in <Expression u""res['id'] if res else ''"">
  <td><input name=""${c.id}-${i}-id"" type=""hidden"" value=""${res['id'] if res else ''}"" /></td>
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 335 in lookup_item
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 405 in undefined
UndefinedError: {'url': u'a', 'hash': u'a', 'description': u'a', 'format': u'a'} has no member named ""id""
}}}

How to reproduce:

 1. Go to package edit form, enter a package resource **but without a URL**.
 2. Click preview.

"	1272379550000000	1272447296000000
305	defect	johnbywater	johnbywater	v1.0	closed	fixed	Search result pagination is broken	"Expect to page through results.

Only page 1 is shown, all other pages fail to display.

Reproduce by searching for something common and browsing to the second page."	1272468229000000	1272994804000000
314	defect	johnbywater	johnbywater		closed	fixed	Bugs getting revisions from the REST API	"Bug report regarding getting revisions:

Getting revisons by ID (on the latest ID)
 
GET ""http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff""

Returns a whole load of stuff (it's also quite slow - about 10 seconds)

I'm expecting it to return just an empty array - am I doing something
wrong here - if so could you clarify correct use of the API?


GET ""http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45""
Returns the empty string - I'd expect an empty array ie []

GET ""http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45""
Returns an internal server error 500 - I think it should probably be ""bad
Request"" 400 (the date is invalid)
"	1273743755000000	1276523983000000
337	enhancement	johnbywater	rgrp	v1.1	closed	fixed	Download links for resources should open in new window	"target = _blank

Cost: 30m"	1275422151000000	1279300972000000
338	story	johnbywater	johnbywater	v1.1	closed		Reference groups by ID in addition to name, since group names can change		1275901137000000	1280446480000000
343	defect	johnbywater	dread		closed	fixed	Packages referred by ID in API	When you do a Package Search or query a Package Relationship in the API version 2, the responses have relationships which refer to packages by name, not by ID.	1276180179000000	1278066420000000
347	enhancement	johnbywater	johnbywater		closed	fixed	The system shall present the package edit form in an API		1276523692000000	1277820440000000
348	enhancement	johnbywater	johnbywater		closed	fixed	The system shall accept package edit form submissions in an API		1276523793000000	1277820496000000
349	defect	johnbywater	johnbywater	ckan-v1.2	closed	invalid	The system shall support authorised users to edit packages on data.gov.uk		1276523851000000	1282908974000000
354	defect	johnbywater	johnbywater		closed	invalid	Collect together requirements and top-level design for user/package 'groups'	"Collect together requirements and top-level design for user/package 'groups': existing tickets, Rufus spec, Sean spec, meeting notes (dread) email, based on existing user authz stuff.

http://knowledgeforge.net/ckan/trac/wiki/AccessControl

Do we add these into user-role table somehow or new table? To present this to team"	1277131335000000	1282908983000000
357	defect	johnbywater	johnbywater		closed	fixed	Fix API documentation, to correct misleading HTTP_AUTHORIZATION statement.	"Sean Burlington wrote:
> I recently came across a documentation issue
> 
> http://knowledgeforge.net/ckan/doc/ckan/api.html
> -----
> The key should be passed in the API request header:
> Header 	Example value
> HTTP_AUTHORIZATION 	fde34a3c-b716-4c39-8dc4-881ba115c6d4
> 
> -----
> 
> But the header name is actually just 'AUTHORIZATION'
> 
> For example to set it in PHP
> 
> curl_setopt($ch, CURLOPT_HTTPHEADER, array('AUTHORIZATION:
> xxxxx-xxxx-xxxx-xxxx-xxxxxxx'));
> 
"	1277459886000000	1277461466000000
372	bug	johnbywater	johnbywater	ckan-v1.2	closed		Fix system limits on CKAN for DGU	Set limits in /etc/security/limits.conf so that we can always ssh in at least.  Requested by DGU.	1279885752000000	1281522535000000
374	task	johnbywater	johnbywater	v1.1	closed	fixed	Design servicization of CKAN API		1280223118000000	1280398501000000
375	task	johnbywater	johnbywater	v1.1	closed	fixed	Integrate Forms API into Drupal	Requested by DGU.	1280223168000000	1280405812000000
376	task	johnbywater	johnbywater	v1.1	closed	fixed	Implement servicization of CKAN API	Requested by DGU.	1280223226000000	1281000755000000
378	story	johnbywater	johnbywater	v1.1	closed	fixed	My user story #1		1280325186000000	1280326706000000
379	task	johnbywater	johnbywater	v1.1	closed	fixed	My dev task #1		1280325277000000	1280326496000000
380	task	johnbywater	johnbywater	v1.1	closed	fixed	My dev task #2		1280325304000000	1280326595000000
382	story	johnbywater	johnbywater	v1.1	closed		Measure quality of service parameters	As a service administrator, I want to measure responsiveness, throughput, and availability of a CKAN service.	1280346974000000	1280854608000000
383	task	johnbywater	johnbywater	v1.1	closed	fixed	Create middleware feature to generate QoS throughput log		1280347595000000	1280854578000000
384	story	johnbywater	johnbywater	v1.1	closed		Send alert when QoS measurements break expectation		1280347841000000	1280496812000000
385	task	johnbywater	johnbywater	v1.1	closed	fixed	Send email if QoS measurement breaks QoS criteria		1280355527000000	1280496127000000
386	task	johnbywater	johnbywater	v1.1	closed	fixed	Set limits in /etc/security/limits.conf		1280356594000000	1281088994000000
387	task	johnbywater	johnbywater	v1.1	closed	fixed	Fix failing tests in search sub-system trigged by recent changes		1280357804000000	1280446047000000
388	task	johnbywater	johnbywater	v1.1	closed	fixed	"Reply to ""two projects"" question from RS"		1280395503000000	1280405288000000
389	task	johnbywater	johnbywater	v1.1	closed	fixed	Investigate timezone issue raised by SB		1280395554000000	1280422414000000
391	task	johnbywater	johnbywater	v1.1	closed	fixed	Create monitoring feature to generate QoS responsiveness log		1280446714000000	1280854592000000
392	task	johnbywater	johnbywater	v1.1	closed	fixed	Create monitoring feature to generate QoS availability log		1280446769000000	1280496140000000
393	task	johnbywater	johnbywater	v1.1	closed	fixed	Create aggregation feature to aggregate QoS log		1280446811000000	1280496962000000
394	task	johnbywater	johnbywater		closed	duplicate	Fix munin on DGU (?)		1280485351000000	1294407189000000
396	task	johnbywater	johnbywater		closed	fixed	Integrate Forms API into Drupal		1280854226000000	1282324934000000
397	task	johnbywater	johnbywater	v1.1	closed	fixed	Implement QoS monitoring for DGU		1280854935000000	1281000745000000
398	task	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Create Public API documentation for DGU (draft)		1281000828000000	1294407372000000
400	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Configure DGU Public API with TSO	Have TSO configure data.ov.uk/api/catalogue. Have TSO register catalogue.data.gov.uk.	1281000956000000	1288003690000000
413	task	johnbywater	johnbywater	ckan-v1.3	closed	invalid	Write functional tests for DGU caching		1281431633000000	1294407458000000
414	task	johnbywater	dread	ckan-v1.2	closed	fixed	Change the Apache and Varnish ports	Ask Paul for a new machine for testing. Then one for varnish-live and one for varnish-test.	1281431639000000	1288003770000000
416	task	johnbywater	dread	ckan-v1.2	closed	fixed	Meet Evan at COI at 1 o'clock Tuesday 10th August 2010 (to progress Forms API)		1281431677000000	1281529736000000
419	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Meet Paul at COI at 2 o'clock Tuesday 10th August 2010		1281431794000000	1281529739000000
425	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Document how licenses can be configured in CKAN		1281436551000000	1281605524000000
445	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Add dumps to DGU SLA	"Thinking about putting something like this:

Dumps Interface (this is NOT going in the API, it is another interface of the service):
 - resources:
    Daily Dump Folder
    Daily Dump File
 - locations:
    /dump/hmg.ckan.net-YYYY-MM-DD.[json|csv].zip 
 - data format:
    Dump Folder is a HTML document containing a table of links to Dump Files
    Dump File has either JSON or CSV format.
 - status codes (200, 404, 500)
"	1282299380000000	1283250620000000
456	story	johnbywater	dread	ckan-v1.2	closed	duplicate	Daily dump		1282299917000000	1282665858000000
468	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Default config file has commented out configuration variable		1282310659000000	1283250799000000
497	story	johnbywater	johnbywater		closed	duplicate	"Respond to CSW ""GetRecords"" request"		1282427334000000	1294407718000000
532	defect	johnbywater	johnbywater	ckan-v1.2	closed	invalid	Just a test - please ignore		1282906842000000	1282907197000000
534	defect	johnbywater	dread	ckan-v1.2	closed	fixed	buildbot config - database machine moved	"Configure buildbot to use postgres db which has been moved to eu5.okfn.org

pudo said: Could you perhaps also comment on http://knowledgeforge.net/okfn/tasks/ticket/466 with a hint on how to update this properly? I only edited /home/buildslave/okfn/full/build/buildandsmoke/buildandsmoke.ini which was the only occurence of the DB credentials I could find there."	1283165568000000	1288002762000000
545	requirement	johnbywater	johnbywater	ckan-v1.2	closed		The system shall support creating packages via the Form API		1283339310000000	1283458648000000
546	story	johnbywater	johnbywater	ckan-v1.2	closed		Get the package create form from the API		1283339414000000	1286896814000000
547	story	johnbywater	johnbywater	ckan-v1.2	closed		Submit package create form response to the API		1283339487000000	1283458645000000
548	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Functional test for getting package create form from the API		1283339598000000	1283351141000000
549	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Method on API controller to return package create form		1283339748000000	1283353945000000
550	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Routing configuration for package create form API resource		1283339785000000	1283351840000000
551	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Documentation for getting package create form from API		1283339845000000	1283354302000000
552	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Method on API controller to accept valid package create form submission		1283340009000000	1283436028000000
553	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	"Functional test for submitting valid ""package create form"" response to the API"		1283340203000000	1283433851000000
554	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	"Functional test for submitting invalid ""package create form"" response to the API"		1283340226000000	1283436393000000
555	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Documentation for submitting package create form submission to API		1283340290000000	1283354313000000
556	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Method on API controller to accept invalid package create form submission		1283340448000000	1283436605000000
566	requirement	johnbywater	johnbywater	ckan-v1.3	closed	invalid	The system shall support harvesting remote metadata		1284039619000000	1294232284000000
568	story	johnbywater	johnbywater	ckan-v1.3	closed		Pull metadata documents from given harvest source entity		1284040131000000	1288038207000000
570	story	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Validate metadata document against UKLP schematron		1284040256000000	1294407974000000
571	story	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Save metadata document and associate with harvest source entity		1284040495000000	1288038218000000
572	story	johnbywater	johnbywater	ckan-v1.3	closed		Write CKAN package from metadata document	"The attributes we need to read in are being defined here:
http://okfnpad.org/uklii

The design advice is to prepare, test, and maintain a dictionary of XPaths for each of the attributes.

Work effort is to be directed to the maintainability of the XPath statements. We need to know which are needed for which documents, and to have a way of preventing cruft."	1284040784000000	1288038237000000
610	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Load local copy of metadata document		1284218592000000	1288041706000000
611	task	johnbywater	johnbywater	ckan-v1.3	closed	invalid	Load local copy of XML schema for metadata document		1284218639000000	1294408239000000
612	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Check given XML schema validates given metadata document		1284218750000000	1294408188000000
613	task	johnbywater	johnbywater		closed		Store result of schema validation check on metadata document object		1284218828000000	1286798466000000
614	task	johnbywater	johnbywater		closed	invalid	Acquire local copy of XML schema for remote metadata documents		1284219077000000	1288038898000000
615	task	johnbywater	johnbywater		closed	invalid	Acquire local implementation of UKLP schematron	"We ""just"" need a schematron.validate(metadata) method. :-)"	1284219174000000	1288038926000000
616	task	johnbywater	johnbywater		closed	invalid	Store result of schematron validation check on metadata document object		1284219246000000	1288038907000000
617	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Check UKLP schematron validates given metadata document		1284219298000000	1294408164000000
621	task	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Metadata document links extracted from WAF response		1284220707000000	1288038326000000
622	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Metadata document identifiers extracted from CSW GetRecords responses		1284220740000000	1287510408000000
623	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	CSW GetRecords request for all identifiers (with CSW authentication)		1284220777000000	1287507837000000
645	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Domain object and migration script to associate packages and harvester objects		1284227152000000	1287774090000000
665	requirement	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	The system shall support withdrawing a harvested dataset or service from publication	"Discussion between John and Peter:
> Given we can identify a document, does the disappearance of a document
> from a registered source imply the disappearance of the metadata (such
> that we delete packages once the documents disappear from the registered
> source)?
> 
I would expect a more explicit 'delete'. The UKLP Use Case Library describes this as ""withdraw a dataset or service from publication"" (part of UCD03 Maintain resources).
"	1285588250000000	1297268097000000
675	task	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Draft specification of CSW harvesting behaviour with guide for publishing agents		1286214877000000	1288038316000000
688	task	johnbywater	johnbywater	ckan-v1.2	closed		Example GeoNetworks service for CSW development		1286786893000000	1286980827000000
689	task	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Updated package for updated metadata document		1286789134000000	1288038308000000
690	requirement	johnbywater	johnbywater	ckan-v1.2	closed	invalid	The system shall support Gemini2 service records	Wish to fold service records into package resources. Trouble is that service records don't necessarily refer to a data record, so should this be a new package?	1286822613000000	1288013718000000
692	requirement	johnbywater	johnbywater	ckan-v1.3	closed	invalid	The system shall support presenting map thumbnail for bounding box.	"To do this for the UKLP, the Ordnance Survey would need to add to their map service an operation to return image for a given bounding box, and COI would need to present the image on the search result page.

Otherwise, it would be possible to use Google."	1286823049000000	1294247841000000
696	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Decoupled generation of package group choices from validation of package group selection		1287066070000000	1287392206000000
702	requirement	johnbywater	johnbywater	ckan-v1.2	closed	duplicate	The system shall support changing package groups when editing a package		1287403778000000	1287403850000000
709	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	Metadata document extracted from CSW GetRecordById response		1287432440000000	1287684969000000
710	task	johnbywater	johnbywater	ckan-v1.2	closed	fixed	CSW GetRecordById request for given identifier		1287432675000000	1287507854000000
712	story	johnbywater	johnbywater	ckan-v1.3	closed	invalid	"Model government data package with ""v0.4"" of the attributes"	"Where said ""v0.4"" would be the result of converging four lists mentioned in requirement #711."	1287581039000000	1294408364000000
713	task	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Review four definitions of DGU package attributes		1287581158000000	1288260915000000
752	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	"Change package attribute names used by Gemini harvesting to DGU ""v.4"""		1288039205000000	1294408472000000
753	requirement	johnbywater	johnbywater	ckan-v1.3	closed	wontfix	The system shall support filtered harvest of CSW sources		1288040290000000	1294247922000000
755	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Add filter attribute to harvest source entity		1288040506000000	1294408632000000
756	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Add filter field to harvest source form		1288040545000000	1294408642000000
758	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Change API documentation to indicate harvest source entity has filter attribute		1288040643000000	1294409053000000
759	story	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Construct and send filtered CSW GetRecords request		1288040753000000	1294408652000000
760	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	"Change ""CSW Get Records"" request class to accept and used given CSW filter"		1288040993000000	1294409111000000
768	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Obtain example ArcGIS CSW server		1288178134000000	1294409308000000
769	task	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Tests for harvesting metadata from ArcGIS CSW API		1288178179000000	1294409318000000
783	story	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Delete harvested documents when remote metadata disappears from harvest source	We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer.	1289210029000000	1294409399000000
785	task	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Review document from PP regarding CSW and WAF guidance	Scope of CSW guidance document has been broadened to include WAF.	1289210338000000	1289482499000000
788	story	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	API user deletes harvest source	Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?).	1289211196000000	1294410007000000
790	story	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Submit  harvest source form with valid CSW location		1289212479000000	1294409709000000
791	story	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Submit harvest source form with invalid CSW location		1289212751000000	1294409723000000
792	task	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Meeting to review pre-alpha testing report from TM		1289212851000000	1289482513000000
793	task	johnbywater	johnbywater	ckan-v1.3	closed	invalid	Fix up changes in changeset aa9aa32e00a9		1289212907000000	1294409836000000
799	requirement	johnbywater	johnbywater		closed	duplicate	WAF harvesting shall be optimised in respect of reharvesting only records that have changed		1289483264000000	1294232675000000
800	story	johnbywater	johnbywater	ckan-v1.3	closed	duplicate	Get WAF records modified since given time		1289483343000000	1294245610000000
802	story	johnbywater	johnbywater		closed	duplicate	Save last harvested time on source		1289484226000000	1294233294000000
804	task	johnbywater	johnbywater	ckan-v1.3	closed	fixed	Rework analysis for publisher/provider in UKLP	We need an incremental plan that connects with current state of DGU and reflects what is actually required by UKLP.	1289816054000000	1294233156000000
872	enhancement	johnbywater	dread		closed	fixed	ONS loader assigns packages to same package even when dept name changes	Solutions: ResourceSeriesLoader allows synonyms for values.	1292424112000000	1292957110000000
250	enhancement	icmurray	dread	ckan-v1.9	assigned		RDF link in Atom feed	Add link to RDF representation of a package in our Atom feed.	1266507695000000	1340631430000000
1453	enhancement	icmurray	icmurray	ckan-sprint-2011-12-05	closed	fixed	Flexible tag names	"Allowing more flexible tag names:

 - allowing spaces
 - allow capital letters (search is case in-sensitive)
 - allow all punctuation except for commas and double-quotes '""'
 - allow unicode
 - commas delimit tag names in the package create/edit form

Effects:

 - package creation/edit form.
 - /tag/{tagname} uri
 - search action
 - api controller (search/package-create/edit)
 - web controller (search/package-create/edit)
 - search api documentation
 - autocomplete for tag names"	1320771850000000	1329395697000000
1523	enhancement	icmurray	icmurray		closed	invalid	[super]	"Form refactor.

New create/edit package form wizard for DGU.  See attached spec."	1323172661000000	1323172904000000
1525	enhancement	icmurray	icmurray	ckan-v1.7	closed	fixed	[super] DGU package form refactor	"Form refactor. New create/edit package form wizard for DGU.

Etherpad: http://ckan.okfnpad.org/dgu-package-form

Tasks:

 - #1526 : Simple one-page form.
 - #1527 : Tab-ify the form using JS.  No validation between steps.
 - #1541 : Setup server for the DGU form-refactor.
 - #1537 : Package create form wizard
 - #1538 : ~~Store partially-filled in package form data as a draft.~~
 - #1539 : ~~Package forms auto-save~~

How the rest of the work is carried out is up for discussion, as it depends on the approach taken.  Whether to continue with validation using AJAX queries, or whether to break the form down into multi pages.  [https://docs.google.com/spreadsheet/ccc?key=0Av2RfDDpj67zdGpSdFlKVjNrQmFXV3lHbGM3ZFlDdnc&hl=en_US#gid=1 Google doc outlining design(s)]

"	1323172839000000	1337159836000000
1526	enhancement	icmurray	icmurray	ckan-sprint-2012-03-05	closed	fixed	[super] Update and test existing DGU package form	"Re-create behaviour of old form without using form-alchemy.  And unit test.

 * [X] write high-level functional tests for the form.  Creation / edition / validation.
       Original estimate: 6 days (3 days x 2)
              Time spent: 3 days
          Time remaining: 0 days

 * [X] write the html
       Original estimate: 2 days (1 day x 2)
              Time spent: 1.3 days
          Time remaining: 0 days

 * [X] modify the existing validation schema
       Original estimate: 2 days (1 day x 2)
              Time spent: 1.3 days
          Time remaining: 1 days

 * [X] split resources into types: individual, time-series, and additional.
       Original estimate: 2 days
              Time spent: 2.5 days
          Time remaining: 0 days

 * [X] contact information is derived from Publisher, but can be altered for each dataset. [#1617]
       Original estimate: 4 days
              Time spent: 0 days
          Time remaining: 4 days
       (Waiting on publisher integration)

 * [X] foi contact information is derived from user, but can be altered for each dataset. [#1617]
       Original estimate: 1 days
              Time spent: 0 days
          Time remaining: 1 days
       (Waiting on publisher integration)

 * [X] provide stubs for the theme and sub-themes [#1618]
       Original estimate: 1 days
              Time spent: 0 days
          Time remaining: 0 days

 * [X] provide stubs for the publisher hierarchy [#1619]
       Original estimate: 2 days
              Time spent: 0.5 days
          Time remaining: 1.5 days
     Note - the groups refactor is essentially ready to be used. And this would be a good test case for it.

 * [X] don't allow package-type selection when editing a package
     Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa. (#1620)
       Original estimate: 0.5 days
              Time spent: 0 days
          Time remaining: 0 days
"	1323173313000000	1330942383000000
1527	enhancement	icmurray	icmurray	ckan-sprint-2012-01-23	closed	fixed	Break DGU package edit form into sections	" - use javascript to selectively hide/show parts of the form
 - there's no validation between steps at this stage.  It's still a
   ""big save button at the end""."	1323174829000000	1327589456000000
1537	enhancement	icmurray	icmurray	ckan-v1.7	closed	fixed	Package create form wizard	"Create the form wizard for the package-new form.

Each section of the form will be a separate page as this was decided to be simpler than the alternative of making AJAX calls for validation at each stage. (*)

 - separate pages for each section of the form
 - validation carried out at each stage against the whole schema.  Each section/page declares a list of schema keys that need to validate for that section to validate, and thus move onto the next section.
 - no draft saving to be performed in this ticket.

(*) - although the javascript alternative will probably provide better UX (each step would require a page-load in the wizard approach), it was decided that:

 - with the javascript approach it would be harder to test the workflow.
 - with the javascript approach there would be additional work displaying validation correctly.  Although not that complicated, it was felt to add another point of failure.
 - the multi-page wizard is quicker and easier to implement, and if it provided poor UX, then the javascript approach would be used instead.
 - the multi-page wizard wouldn't preclude a javascript-tabbing create-form for other cases (where the wizard workflow wasn't such a good match, eg on the hedatahub.org)
 - the multi-page wizard wouldn't preclude a javascript-tabbing edit-form."	1323358451000000	1337159772000000
1538	enhancement	icmurray	icmurray	ckan-v1.7	closed	wontfix	Store partially-filled in package form data as a draft.	"Allow a user to save a partially-complete and probably invalid package-create form in order to come back to it later.

We decided on blob storage over a state field on the package:

 - prevents a history of the drafts being stored in the revision history.  This is a particular problem if we start auto-saving, say every 20s.
 - the blob can store arbitrary extra data, such as how far through the wizard we are."	1323359221000000	1337159799000000
1539	enhancement	icmurray	icmurray	ckan-v1.7	closed	wontfix	Package forms auto-save	"Using the saved-draft feature, we enable auto-save for the package-new and package-edit forms.

An AJAX call is made every N seconds, and the auto-saved blob is updated.

note - we don't want to overwrite a previously save draft, but we will want to overwrite the last autosave."	1323359385000000	1337159806000000
1541	task	icmurray	icmurray	ckan-sprint-2012-01-23	closed	fixed	Setup server for the DGU form-refactor.	To enable us to show DGU work in progress, for feedback.	1323359484000000	1327311698000000
1591	enhancement	icmurray	dread	ckan-sprint-2012-01-23	closed	fixed	Upgrade CKAN 1.5 instances	"CKAN 1.5 installations need to be upgrade to 1.5.1.

Working methodically through the CKANs on http://datacatalogs.org/group/ckan :

At 1.5:
 * ckan-brazil http://br.ckan.net/ s057 - DONE
 * ckan_ireland http://ie.ckan.net/ s057 - DONE 
 * cz-ckan-net http://cz.ckan.net/ s057 - DONE
 * datakilder-no http://no.ckan.net/ s057 - DONE
 * data_gov_uk http://catalogue.data.gov.uk/ 
 * datacatalogs-org http://datacatalogs.org/ s054 - DONE

Other versions:
 * berlin-open-data http://daten.berlin.de/ CKAN hidden
 * ckan-italia http://it.ckan.net/ 1.3.3b
 * ckan-poland http://pl.ckan.net/ Down
 * data-norge-no http://data.norge.no/ CKAN hidden
 * datadotgc-ca http://datadotgc.ca/ http://ca.ckan.net/ 1.4.2a 
 * datagm http://www.datagm.org.uk/ 1.3.3
 * datagov-ru http://datagov.ru/ CKAN hidden
 * datospublicos-org http://datospublicos.org/ 1.4.2
 * iati-registry http://iatiregistry.org/ 1.5.1c
 * katalog-offene-daten-oesterreich http://offener.datenkatalog.at/ 1.3.3b
 * nosdonnees_fr http://www.nosdonnees.fr/ 1.3.1a
 * opengov-fi http://opengov.fi/ CKAN hidden
 * opengov_es http://opengov.es/ 1.2
 * publicdata-eu http://publicdata.eu/ 1.4.2a
 * the-data-hub http://thedatahub.org/ 1.5.1
 * helsinki_finland http://hri.fi/ CKAN hidden

Now deleted from datacatalogs.org:
 * ckan-belgium http://be.ckan.net/ Migrated
 * si-ckan-net http://si.ckan.net/ migrated
 * gr-ckan-net http://gr.ckan.net/ Migrated
 * hu-ckan-net http://hu.ckan.net/ Migrated
 * ckan-lithuania http://lt.ckan.net/ Migrated
"	1324487658000000	1328789470000000
1593	enhancement	icmurray	amercader	ckan-sprint-2012-04-16	closed	fixed	[super] Create feeds from search results	"Recent changes (see #191 and #1498) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy.

Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest.

Apart from the general feed with changes on the whole instance, other different ""pre-set"" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags:
{{{
http://<ckan_server>/feed/group/{id}.{format:rss|atom}
http://<ckan_server>/feed/tag/{id}.{format:rss|atom}
}}}

These could be shown on the group/tag page, both on the <head> section, so browsers can autodiscover them, and on the UI, with a message like ""Subscribe to this group/tag"".

Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on

{{{
http://<ckan_server>/feed/custom.{format:rss|atom}?res_format=api%2Fsparql
http://<ckan_server>/feed/custom.{format:rss|atom}?q=tags:animal*
}}}

On this case a message ""Subscribe to these search results"" would appear.

Note that a lot of work on this front needs to be done for the IATI registry project (see #1590)


Other tickets related:
 * #1592: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now.


"	1324489115000000	1334566668000000
1607	enhancement	icmurray	dread	ckan-v1.7	closed	fixed	[super] Data.gov.uk Maintenance Refactor and UKLP Development Support	"A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU.

Design doc: https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB  The design doc is the authoritative source of tasks.  This ticket acts as a synopsis.

Current sprint (2012-02-20):

 * 7a. ~~Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1~~
 * 6. ~~Publisher Registration Improvements MUST PHASE 1~~
 * 3. ~~Themes/Taxonomy as well as tags WONT PHASE 2~~
 * 32. ~~Browse by Tags PHASE 1 MUST~~
 * 12. ~~Edit/Delete for Harvested Datasets MUST URGENT PHASE 1~~
 * 13. ~~Provider labelling MUST PHASE 1 [UKLP #14 MUST]~~
 * 17. ~~Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]~~
 * 22. ~~Mixed Licenses MUST PHASE 2~~

Plus, brought forward from last sprint:

 * ~~Publisher hierarchy support~~
 * ~~Improved dataset creation tools (form) - #1525~~
 * ~~Edit/delete for harvested datasets~~



Overview of tasks: see the design doc."	1325503348000000	1337159969000000
1617	enhancement	icmurray	icmurray	ckan-v1.7	closed	fixed	Update and test existing DGU package form : publisher / foi information	"[Child of #1526]

Contact/FOI information is derived from Publisher/User, but can be altered for each dataset.

       Original estimate: 4 days
              Time spent: 0 days
          Time remaining: 4 days"	1326103384000000	1332151332000000
1618	enhancement	icmurray	icmurray	ckan-v1.7	closed	fixed	Update and test existing DGU package form : provide stubs for the theme and sub-themes	"[Child of #1526]

Provide the interface for the themes and subthemes hierarchy."	1326103557000000	1332151341000000
1619	enhancement	icmurray	icmurray	ckan-v1.7	closed	fixed	Update and test existing DGU package form : provide stubs for the publisher hierarchy	"[Child of #1526]

Create the interface used to retrieve publisher and authz information from the publisher hierarchy."	1326103672000000	1332151351000000
1620	enhancement	icmurray	icmurray	ckan-sprint-2012-02-06	closed	fixed	Update and test existing DGU package form : don't allow package-type selection when editing a package	"[Child of #1526]

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa."	1326103837000000	1327589576000000
1645	enhancement	icmurray	icmurray	ckan-sprint-2012-01-23	closed	fixed	Update and test existing DGU package form : Apply a simple theme	"Theme the DGU form.

Doesn't need to be an exact replica of DGU, but just enough to show it's possible."	1326394622000000	1327311679000000
1711	enhancement	icmurray	icmurray	ckan-sprint-2012-03-19	closed	fixed	Resource validation page	" * On the resources tab, there's a ""Check Resources"" button which, when clicked makes an ajax request
   with the list of URLs entered by the user.
   
 * The server checka each link for errors and header information about the linked resource.  (Using
   ckanext/archiver/tasks.py:link_checker()).
   
 * The server returns a list of dicts (json), containing information about the linked resource, and
   the client uses that to:

     - populate the format field of each resource
     
       The (guessed) 'file_extension' populates the '__format' field.  If it's 'htm' or 'html',
       then we assume it's a listing page, and so don't populate the format field with 'htm' or
       'html'.
     
     - provide feedback if a URL appears to be invalid
     
       If the URL doesn't appear to be a URL at all, or returns a HTTP error, or times-out, then
       URL field is highlighted in red to indicate it's a bad URL.  A tooltip shows the
       error message to the user.
     
     - [Optional] provide feedback if a URL appears to point to ""Additional Information"" - ie the
       Resource should be entered under ""Additional Resources"", rather than ""Timeseries"" or
       ""Individual"" datasets.

Analysis and further description on etherpad: http://ckan.okfnpad.org/dgu-package-form? [Section I]"	1327589759000000	1332151557000000
1712	enhancement	icmurray	icmurray	ckan-sprint-2012-02-06	closed	fixed	DGU /data page	"CKAN provides the /data page page of data.gov.uk

Breakdown of tasks:

 * [x] Analysis / refinement of spec.
 * [x] Log-in / register as publisher
       (Waiting on publisher form)
 * [x] Population of ""browse by publisher""
       list groups ordered by most datasets
       (Waiting on publisher integration)
 * [ ] ~~Browse by nation~~
 * [ ] ~~Featured datasets~~
       Now a possible integration point with drupal
 * [X] Tag cloud
 * [X] Developers section"	1327590527000000	1329733846000000
1713	enhancement	icmurray	icmurray	ckan-sprint-2012-02-06	closed	fixed	DGU dataset search page	"Search results page of DGU.

Breakdown:

 [X] ensure faceting by group (publisher) works
 [X] allow faceting by UKLP dataset type
 [X] allow faceting by INSPIRE"	1327590700000000	1327932401000000
1714	enhancement	icmurray	icmurray	ckan-sprint-2012-02-06	closed	fixed	DGU dataset view page	"Dataset read page for DGU

Breakdown:

 - [X] Fix missing title
 - [X] Links to the social media stuff
      - ""Share your app""
      - ""Share your idea""
      - ""Request new data""
 - [ ] ~~Presentation of additional information should be broken into sections~~
       Content is there.  Don't do anything until theming/design work is undertaken.
 - [ ] ~~""More like this"": links to similar datasets.~~
       Dropping
 - [N] ~~Stars~~ (I think these are being removed)
 - [ ] ~~Comments~~
       Possible integration point with Drupal.  Needs it own ticket.
 - [X] Give feedback to department
 - [X] Tidy up the QA stars"	1327591090000000	1329734018000000
1737	enhancement	icmurray	rgrp	ckan-v1.8	closed	fixed	Expose solr-based search API	"Super ticket: #1745

 * Convert /api/rest/dataset to be search query (i.e. take arguments in ?....)
 * Directly expose solr though may want to override limit. See https://github.com/okfn/openspending/blob/master/openspending/ui/controllers/api.py#L48

Required for some improvements to UX (such as autocomplete and better search)."	1328014626000000	1340112732000000
1751	enhancement	icmurray	icmurray	ckan-sprint-2012-02-20	closed	fixed	DGU dataset form: collection of updates from feedback	"A collection of improvements to the dataset creation/edition form (feedback form DGU).

 - [x] validation error formatting
 - [x] geograohic coverage: add local authority
 - [x] resource format field
 - [x] resource format auto-complete
 - [x] renamed data tab -> files tab
 - [x] integrate Adris'a INSPIRE changeset, and migration scripts as part of build
 - [x] remove ""discontinued"" from the update frequency
 - [x] disable the disabling of save
 - [x] Remove ability to edit url on the edit-form (and remove ""url not available"")"	1328526612000000	1329733458000000
1752	enhancement	icmurray	icmurray	ckan-sprint-2012-02-20	closed	fixed	DGU: collection of updates from feedback	" - [x] hide black stars on the data-view page
 - [x] Edit on view form (if you have the right permission)
 - [x] RES_FORMAT ~> Resource Format
 - [x] Map Based Search -> UK Location Map Based Search
 - [x] Note at the top right expalining that these are seach filters (if they are) -> this requires ""add a dataset"" and ""other access"" to be moved to a different place (just some whitespace is fine)
 - [x] Display publisher and provider (UKLP records only).
 - [x] analysis of filter by OGL / non-OGL
 - [x] provide /dump on the build server"	1328526750000000	1329733515000000
1768	enhancement	icmurray	icmurray	ckan-v1.7	closed	fixed	DGU: 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1		1328539740000000	1337159873000000
1769	enhancement	icmurray	icmurray	ckan-sprint-2012-02-06	closed	fixed	6. Publisher Registration Improvements MUST PHASE 1		1328539789000000	1329737597000000
1770	enhancement	icmurray	icmurray	ckan-sprint-2012-02-06	closed	wontfix	3. Themes/Taxonomy as well as tags WONT PHASE 2		1328539834000000	1329733935000000
1771	enhancement	icmurray	icmurray	ckan-sprint-2012-02-20	closed	fixed	32. Browse by Tags PHASE 1 MUST		1328539862000000	1329733648000000
1772	enhancement	icmurray	icmurray	ckan-sprint-2012-02-06	closed	fixed	13. Provider labelling MUST PHASE 1 [UKLP #14 MUST]		1328539902000000	1329733962000000
1773	enhancement	icmurray	icmurray	ckan-sprint-2012-02-20	closed	fixed	17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]		1328540130000000	1329733534000000
1774	enhancement	icmurray	icmurray	ckan-sprint-2012-02-20	closed	fixed	22. Mixed Licenses MUST PHASE 2		1328540153000000	1329733550000000
2277	refactor	icmurray	icmurray	ckan-backlog	new		Use the new atom feeds in IATI	Atom feeds have been implemented in core in #1593.  The IATI-specific implementation can now be removed when IATI move to CKAN >= 1.6.1	1333386628000000	1333386628000000
2285	enhancement	icmurray	icmurray	ckan-sprint-2012-04-30	closed	fixed	Data Previewer / Viewer v3 : Embedding! - /resource/../embed	"Allow a user to embed the data viewer in their webpage.

* [x] Embedding widget [1d]

* [x] The embed page [1-2d]

* [x] Documentation and Announce [0.5d]

Depends upon:

* [x] Recline improvements [?d]

All the above, plus user stories are fleshed out in [1]

[1] http://ckan.okfnpad.org/feature-2283-data-viewer-v3?"	1334230810000000	1335440451000000
2295	enhancement	icmurray	icmurray	ckan-sprint-2012-04-16	closed	fixed	Migrate CKAN packaging and apt.ckan.org from dgu-buildbot	" * Create Jenkins task for packaging CKAN
 * ... and publishing to s3://apt.ckan.org"	1334566976000000	1334566986000000
2296	enhancement	icmurray	icmurray	ckan-sprint-2012-04-16	closed	fixed	Initial analysis of moderated edits	What needs doing, and how long will it take?	1334567077000000	1334567150000000
2297	enhancement	icmurray	icmurray	ckan-sprint-2012-04-16	closed	fixed	Initial analysis of follow support		1334567223000000	1334569632000000
2298	enhancement	icmurray	icmurray	ckan-v1.9	new		Add sort-by controls to the search results page	"Sort-by functionality was exposed through the package controller in [1].  But no controls were added to the search-page.

 * What should the sortBy controls/widget look like?
 * Which fields should be exposed?

The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master.

[1] https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c"	1334569162000000	1340635947000000
2307	enhancement	icmurray	icmurray	ckan-sprint-2012-04-30	closed	fixed	Merge 1.6.1 into master	There are a number of features developed against 1.6.1 that need to be merged into master.	1334591402000000	1335440476000000
2308	enhancement	icmurray	icmurray	ckan-sprint-2012-04-30	closed	fixed	Merge facet html snippets in the multi-lingual branch	Seanh and icmurray both touched on the html snippets in facets.html.  So there's a merge issue.  icmurray to merge changes in the multi-lingual branch prior to merging into master.	1334591529000000	1335440505000000
2310	enhancement	icmurray	icmurray	ckan-backlog	new		Refactor the search-query construction in feeds.py	"The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication."	1334592091000000	1337159386000000
2313	enhancement	icmurray	icmurray	ckan-v1.8	closed	fixed	Deprecate old facet data structures and related functions.	"In branch feature-1821-multilingual-extension, a new faceting datastructure was introduced.  This makes the old one and related functions obsolete.  These have been marked as deprecated for the 1.7 release, and should be removed for the 1.8 release.

 * helpers.py:facet_items()
 * facets.html:facet_sidebar()
 * uses of c.facets (rather than c.new_facets)
"	1334677916000000	1343124732000000
2314	defect	icmurray	seanh	ckan-sprint-2012-05-15	closed	fixed	parse_rfc_2822 tests failing in some timezones	"These two tests:

ckan.tests.lib.test_helpers.TestHelpers.test_parse_rfc_2822_gmt_case

ckan.tests.lib.test_helpers.TestHelpers.test_parse_rfc_2822_with_offset

have been failing in certain timezones. In the UK they pass, if you're in Germany they'll fail.

The function being tested uses datetime.datetime.fromtimestamp() which uses the local time, but the tests compare the results against hardcoded strings, so the tests cannot work in different timezones."	1334768832000000	1337073270000000
2330	enhancement	icmurray	icmurray	ckan-sprint-2012-05-15	closed	fixed	Make api read-actions GETable	Whitelist any GETable api actions, and optionally construct the query from url params rather than body.	1335460585000000	1337073314000000
2347	enhancement	icmurray	ross	ckan-v1.8	closed	fixed	Related stuff dashboard and changes	"
Some additions to the related stuff functionality to make it more useful post the 1.7 release.

[x] Edit item (after checking auth)

[x] My related things on my user page

    [x] Show user created related things on the users' page with links direct to the related page of that dataset. {0.5d}
  

[x] Related stuff usage counts { 1d }

    Record the number of times the link for a related item is clicked.  This is likely to mean that the link fires to call CKAN and then redirects to the actual URL.

[x] Un-attached related items {0.25d}

    Allow related items to be created without being attached to datasets.  This may mean that the dataset field is optional in the API.

[ ] Dashboard for related 

    A dashboard for related items, site-wide (not package specific) and orderable by date, views, {rating}, popularity of the dataset.  Would be useful (possibly) for site-admin to mark one or more as 'featured'

    [x] Dashboard + Controller (Related) {1d}

    [x] Ordering at logic layer (date, views, rating, popularity of package, featured) {1d}
         [x] date
         [x] views
         [x] featured

    [x] 'Featured' related item with mechanism to fetch it. {0.5d}

        [ ] UI within dashboard for sysadmins to check/uncheck feature status {0.5d}

[x] Changing auth for related.

    Currently auth is based on deletion being possible by dataset owner and the related owner but it may be that a lot of the auth for related items should be configured to have more restrictive auth.

    [x] Implement auth changes for publisher auth profile. { 0.5d }


[x] General issues

  [x] Serverside validation of URLs

  [x] Activity updates

  [x] Make optional


 [x] Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204



[ ] UI Changes

  [ ] General tidy up, fix form etc (icon for menu bar)

  [ ]  Highlight featured items py:if=""related.featured""

  [ ] Show either view count or popularity ${ related.view_count }

  [ ] A user's related items should be on their home page

  [ ] Related item plus sign is very poorly aligned within box

  [ ] Add related items should be in drop down like for resources (?) to make it easy to add items

  [ ] Hover should show a title that gives some information about what related are (not at all clear)
       ""Apps, Ideas, Visualization and other material related to this dataset""

[x] Related is a pretty terrible name in the UI - much better to have it called Apps, Ideas etc (perhaps title tag could even give more details e.g. ""Apps, Ideas, Visualization and other material related to this dataset"") -- btw i thought this was in #2332 but realize it wasn't (apologies for that!)

 - Went with Apps & Ideas (for now)

[x] Documentation, documentation, documentation - AFAICT I can't see anything in v1.7 or master docs. I imagine this would be a short section like http://docs.ckan.org/en/ckan-1.7/commenting.html (and probably coming right after that in the ToC) which says

  * What it does
  * How to enable (and perhaps includes a nice-screenshot!)
  * How to customize (e.g. can one customize the list of options of things one can create (e.g. can I set it to just be app and idea or ...)
  * Also liase with Mark W re a blog post about this from the user perspective (a screenshot walkthrough ...)

[ ] Clearly mark the old extension as deprecated from v1.7 forward: https://github.com/okfn/ckanext-apps (I just met someone last week who was working on integrating this and had no idea it was replaced by something better in v1.7)"	1335879539000000	1343226149000000
2350	enhancement	icmurray	icmurray	ckan-v1.9	new		Hooks  in package controller for validating form data	"When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data.  Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to PackageController subclasses to do form validation/processing.

Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override.

NB - form processing/validation *is* different to validating data.  For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&Cs.

 - [ ] Add hooks
 - [ ] Use in ckanext-example"	1335881761000000	1340635966000000
2351	enhancement	icmurray	icmurray	ckan-ecportal	closed	fixed	Install EC Portal on CentOS	"Install and document (step-by-step) CKAN (and dependencies, and extensions) onto VM on s031.okserver.org which has CentOS 6.2 installed.  Paying particular attention to the fact that we won't have root access on the staging nor production servers, and so a) anything that requires root access must be documented carefully; and b) we want to be able to perform maintenance without being root.

== Maintenance ==

We want to have ''non-root'' access to:

 - CKAN logs; (relevant) apache logs; qa process' logs; rdf cron logs
 - restarting CKAN
 - restarting qa processes
 - changing/disabling rdf cronjob
 - our database (through sql)
 - restarting solr

== CKAN and Dependencies ==

 - CKAN installed from source
 - Required plugins: synchronous_search ecportal ecportal_form organizations multilingual_dataset multilingual_group multilingual_tag qa

== Extra configuration ==

 - uploads
 - analytics (toby)
 - qa
 - rdf cron
 - caching
 - datastore

== Things to remember ==

 - Use ckanext/multilingual/solr/schema.xml and stopword files
 - Use nginx as proxy.

"	1335883455000000	1340615641000000
2383	defect	icmurray	icmurray	ckan-sprint-2012-05-29	closed	fixed	Date on add/edit resource breaks if offset is specified		1337080275000000	1338197024000000
2392	enhancement	icmurray	icmurray	ckan-sprint-2012-05-29	closed	fixed	Add language field to dataset	"An extra field on datasets called ""metadata_language"".  The language that the meta-data is in.  Drop-down list needs to be translatable.     Default value is ""English""."	1337159501000000	1337270483000000
2393	enhancement	icmurray	icmurray	ckan-sprint-2012-05-29	closed	fixed	Map licenses with countryCode to license name without countryCode	Map licenses: given http://creativecommons.org/licenses/by/2.5/{countrycode}/ as a license value through the API, convert it to http://www.opendefinition.org/licenses/cc-by .  Should also work if given http://www.opendefinition.org/licenses/cc-by  .  NOTE: the API may send through a list of licenses; in this situation, take the head of the list.	1337159564000000	1337270463000000
2405	defect	icmurray	rgrp	ckan-v1.9	assigned		Improvements to data viewer embed	" * Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...)
   * Current icon and text looks really bad
 * Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
 * Data viewer embed url includes all the resource info (not needed -- just url and backend info ...)
 * Embed extension urls do not respect original domain but instead use site_url (I'm guessing). E.g. datahub.io domain ends up with links to embed on thedatahub.org
"	1337583565000000	1340635981000000
2437	task	icmurray	icmurray	ckan-v1.8	closed	fixed	Add coding standards to CKAN docs	"* [ ] Write up python coding standards as discussed at dev-meetup into CKAN docs.  (Mostly written-up as part of dev-meetup notes.)

* [ ] Collate frontend coding standards (http://aron.github.com/ckan-style/styleguide/) into same doc(s).
"	1338196895000000	1343124799000000
2438	enhancement	icmurray	icmurray	ckan-v1.8	closed	fixed	resource_search action not accessible via GET request	This action is not currently usable via a GET request as it relies upon a nested dict of fields.	1338197305000000	1341406005000000
2439	defect	icmurray	icmurray	ckan-v1.8	closed	fixed	tag_search and tag_autocomplete not fully available via GET request	`tag_search` and `tag_autocomplete`: The `fields` argument is not available when accessing this action with a GET request.	1338197549000000	1341295142000000
2485	defect	icmurray	nils.toedtmann	ckan-v1.9	new		Encourage leak containment by limiting the number of requests a CKAN process serves	"CKAN has [ticket:1345 memory leaks]. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...
 * research ways to limit requests-per-process with the different web servers (see below for a start);
 * explain these safeguards in the CKAN documentation and encourage users to apply them;
 * consider the helper script '''[https://github.com/okfn/ckan/blob/master/ckan_deb/usr/bin/ckan-create-instance ckan-create-instance]''' to generate Apache configs with '''WSGIDaemonProcess ... maximum-requests=XY''' being active instead of [https://github.com/okfn/ckan/blob/master/ckan_deb/usr/lib/ckan/common.sh#L262 commented out]. Or at least let it warn the user to use '''!MaxRequestsPerChild''';
 * Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

----

How to limit requests-per-process
 * Apache:
  * Use '''WSGIDaemonProcess''' with '''maximum-requests=50''' or whatever limit is appropriate. We did this sucessfully on [http://trac.okfn.org/ticket/904 datahub.io] and the [http://trac.okfn.org/ticket/1245 ckan farm]
  * (This need to be verified) Without WSGIDaemonProcess, '''!MaxRequestsPerChild 50''' should achieve the same.
 * nginx/supervisord: to be researched.

"	1338551229000000	1340636028000000
2550	enhancement	icmurray	ross	datahub-july	assigned		User types	"== Requirements ==

In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub).

== Required interface ==

Once changes have been made to the user schema, for a given user we want to be able to:

 * determine if they have a paid or a free account, and
 * get a string name of the type of paid account.

Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin.

== User Stories ==


User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases.  For now it is adequate that we can manually control these things through paster commands.

Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all.

 * As a sysadmin I would like to be able to use a paster command to
   manually set a user's payment level, or remove it entirely.

 * As a sysadmin I would like to be able to run a paster command to
   view a list of users who have a payment plan, grouped by the plan
   that they have.

 * As a sysadmin I would like to be able to use the API to change the
   payment status of a specific user through user_create and user_update.
   This shouldn't be available to anybody else.

 * As a user, and only if I have one, I'd like to see my current payment
   level on my user profile page.
   

== Tasks ==

[x] Tests

[x] Plugin based migration

[x] Code

    [x] Model

    [x] API

[x] Documentation


== Estimates ==


"	1340017590000000	1346669497000000
2603	refactor	icmurray	icmurray	ckan-v1.9	new		Remove deprecated 'fields' parameter from resource_search	The fields parameter of resource_search was deprecated when fixing #2438.  It can be removed in release 1.9, and the action tidied up as a result.	1340730601000000	1340730601000000
2621	refactor	icmurray	icmurray	ckan-v1.9	new		Remove the deprecated 'fields' parameter from tag_search and tag_autocomplete	"This was deprecated in 1.8 as it wasn't accessible via GET requests due to being a dict.  See #2439

In a future release of CKAN (probably 1.9) it can be removed.

Internal uses of it were removed in #2439, but there are tests that still use it."	1340900569000000	1340900569000000
2651	enhancement	icmurray	ross	ckan-v1.8	closed	fixed	Check support for TSV which doesn't appear to work well.	"TSV support doesn't seem to work very well, may be the mimetype ( text/tab-separated-values )

See http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96
"	1341923318000000	1343209784000000
2672	enhancement	icmurray	icmurray	ckan-v1.8	closed	fixed	Session.is_modified should use passive=True	"According to sqlalchemy docs [1], when calling Session.is_modified(), the `passive` argument needs to be passed in as True.

[1] http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html?highlight=is_modified#sqlalchemy.orm.session.Session.is_modified"	1342185523000000	1343124666000000
2679	enhancement	icmurray	icmurray	ckan-v1.9	new		Change default behaviour of TemplateController.view to 404.	"The current behaviour of TemplateController.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file.

Although this may be a feature that some instances want.  In general, it leads to:

- 500s when attempting to access a normal template (eg - http://datahub.io/importer/preview)
- A way of inadvertantly serving things you may not want to serve.  (Small risk, as it needs to be renderable as a genshi template).

Solution:

- Change the controller to 404
- Ensure there's a way for existing ckan instances to override that behaviour should they need it."	1342436133000000	1342436133000000
2709	enhancement	icmurray	markw		new		Atom feeds are undocumented	There doesn't seem to be any documentation yet for Atom feeds.	1342624310000000	1342626212000000
2784	defect	icmurray	icmurray		new		model dictize sensitive data	The model dictize layer doesn't consistently remove sensitive data from the dictized models.  It should use the current context to decide whether to include sensitive data or not.	1343814685000000	1343814685000000
2859	defect	icmurray	icmurray	ckan-v1.8	new		Fix the build	"The requires files have been removed.  These were required by buildkit.

We've lost the information regarding which packages were already in ubuntu repos, which conflicted and which were missing.  But perhaps we can just package everything up."	1345109217000000	1345109217000000
2878	enhancement	icmurray	ross	ckan 2.0	closed	wontfix	Roles and Permissions for Organisations	"As part of merging Organisations into core, it is necessary that we clarify the capacity field with which the users/datasets are added as members to the group 'subclass'.

Rather than the capacity being an opaque string that implies auth but doesn't clearly specify it, we will use role names where roles are defined in the database - with a clearly defined set of standard roles.  The Role table is expected to have simply a string name/representation and acts as a container for permissions.

Each permission is a string of the form object.action (such as package.add, group.delete) of which several are expected to be associated with a role.  This means the permission table will contain a string and a reference to the role.

This work will require UI changes to the screens allowing users to be added to a group/organisation so that the list of available roles is available to add those users.

[x] Model for Role and Permission

[ ] Logic layer changes for managing roles/permissions etc.

[ ] Determine default roles, perhaps just admin/editor/viewer

[ ] Fix the auth layer to use the permissions/roles - may be better implemented as another ticket.

"	1345466266000000	1350561906000000
2880	enhancement	icmurray	ross	ckan 2.0	closed	wontfix	Datasets (Package) should have owners	"Each dataset should have an explicit owner that is an organisation.  This is the organisation that owns the dataset* and will be used for specifying who can move a dataset out of the default organisation.

This should probably be a foreign key link from dataset to organisation (or rather group).

* Currently this won't necessarily be the organisation that the dataset is a member of, but it is likely that this will be the case initially (at least until someone moves it to another organisation)."	1345466535000000	1350561968000000
2959	defect	icmurray	icmurray	ckan 2.0	new		Changing a Group's name through the action api disassociates it from its datasets in the index	"Repro:

 - Create a new Group, named ""test-group"".
 - Add a dataset to it.
 - Verify the dataset belongs to the group by visiting the Group's read-page
 - Update the Group through the action api (group_update), using the uid in the ""id"" field, and a new name in the ""name"" field.
 - Visit the group's read-page.  The list of datasets will be empty.

This was an issue when editing a Group through the web interface, which was fixed in [1].  However it only fixes the issue in the group controller.

[1] https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335"	1349363935000000	1349363935000000
62	enhancement	dread	rgrp	v0.10	closed	fixed	Change tags to contain any character (other than space)	Requires us to url encode the tag names when displaying them ...	1240585095000000	1250181376000000
74	enhancement	dread	rgrp	v0.10	closed	fixed	Add Is It Open links to package pages	"On each package page in the openness field if the package is not open provide a link to is it open create enquiry page.

Details:
  * may want to add to the url a query parameter indicating which package this relates to e.g. ?ckan-package=...
  * http://isitopen.ckan.net/enquiry/create/?ckan-package=...
  * do we provide link even if package is open (e.g. so that people can make enquiries about e.g. adding open data/content buttons ...)

Cost: 0.5h"	1247828218000000	1250182938000000
75	enhancement	dread	rgrp		closed	duplicate	"Record and display package ""usage"" information"	"  * Number of package page visits on ckan (can we get this straight from google analytics)
  * Number of times url or download url is used - now ticket:937 (Record download stats for resoures)

How do we do this?

  * Google analytics will miss a lot of this usage (and how do we get that data out anyway)
  * Could use javascript but again misses usage.
  * One option is to redirect link but that is kind of nasty (but may be only option ...)
"	1247828785000000	1296341223000000
76	enhancement	dread	rgrp	v0.10	closed	fixed	Convert to use formalchemy for all forms	"Current form stuff is not very good (uses formencode). Switch to formalchemy would improve this, especially on validation.

As an extra we could utilize the formalchemy pylons admin interface (pretty much for free).

Cost: 12h

== Details ==

  1. Replace htmlfill and formencode extract in controllers/templates with formalchemy
    1. Crude and simple
    2. Suppress unwanted fields (revision, state, all revisions) and sort out ordering to be similar to before.
    3. Sort out tag field with a special renderer
  2. Validation - testing definitely required.
  3. Fix up description and pretty css etc"	1247829041000000	1251301765000000
77	enhancement	dread	rgrp	v0.11	closed	fixed	Rate packages	"= As a User or Visitor =

I wish to rate a package in some way. On the package view, below Openness, there is a 'Rating' heading with the star rating which is selectable.

Rating out of 5. One user or IP, one vote.

package_rating table:

package      | user                  | rating 
annakarenina | joe.bloggs.openid.com | 4
annakarenina | 154.2.5.47            | 1"	1247829193000000	1255176732000000
79	enhancement	dread	rgrp	v0.10	closed	fixed	Improvements to package search	"Several things:

  * Default package search (used e.g. on front page) should deal well with multiple items: ""abc xyz"" means search for both abc and xyz (as separate searches)
  * Should automatically display a list of matching tags in a line at top with number of associated packages
  * have an option to only search for fully open material (or openly licensed material ...)


Since we should be moving to a more sophisticated search solution anyway (to allow prioritisation etc) maybe we should cautious as to what we do now."	1247829864000000	1265890806000000
80	enhancement	dread	rgrp	v0.10	closed	fixed	Refactor or remove modes code	"Get rid of modes in the RESTful API. Do json stuff directly in controllers.

 * package will have to_dict and from_dict methods (called by rest controller) which provide and consume JSON friendly dictionaries representing the object. from_dict - class_method. stuff in forms for tags as_string, maybe factor out. reuse validation stuff."	1247844263000000	1265890912000000
91	enhancement	dread	rgrp	v0.10	closed	fixed	Add author and maintainer attributes to package	"Add the following attributes to package:
  * author, author_email
  * maintainer, maintainer_email

Gives us full compatibility to: http://docs.python.org/distutils/setupscript.html#additional-meta-data

Column ordering - should come after name, title, url, download_url."	1249049780000000	1250864156000000
94	enhancement	dread	rgrp	v0.10	closed	fixed	Use sqlalchemy-migrate to handle db/model upgrades	"Details of how to go about this here <http://www.rufuspollock.org/2009/07/27/sqlalchemy-migrate-with-pylons/>

Cost: 2h"	1249134185000000	1250604808000000
105	enhancement	dread	rgrp	v0.10	closed	fixed	Package groups (view)	"As a user I want to be able to create groups of packages.

This functionality is different from tags:

  * The ability to add a package to a given group is restricted whereas anyone can add a given tag to a package
  * Specifically groups have owners and only the owners can add a package to that group

Group properties
  * id (primarykey, uuid style)
  * name (restricted content - same rules as package)
  * title (no restrictions)
  * owners -- many:many with user object
  * description (markdown)
  * packages -- many:many with package object

Don't version groups for now.

Groups address in the WUI will be:
  * /group     - browse list of groups (reuse list action)
  * /group/list - same thing for now
  * /group/<groupname> - display: group properties with links to packages. No links to user pages (yet).

Group editing and searching will be another ticket.

Cost: 3d
"	1251392282000000	1252488496000000
106	enhancement	dread	rgrp		closed	duplicate	Regularly convert CKAN data to RDF and put on Talis CC	"Sister to ticket:90 (Link to RDF version of CKAN data on Talis Connected Commons).

Talis have already kindly done an initial conversion. We should repeat this process regularly and re-upload the data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Attached is the ruby script used by Talis for conversion

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)"	1251454474000000	1256140649000000
108	enhancement	dread	rgrp	v0.10	closed	fixed	Package search in the REST API	"Add package search facility in the rest api at /api/search

Queries can be provided as for the normal package search either by posting to that url or by performing a get with a query string.

E.g. .../api/search/package?q=xyz

Query parameters:
  * q is keyword string (searches name, title, tags by default)
    * split by words and ANDed
  * additional parameters for specific fields
  * qjson is alternative to q for Searching by specific fields (in addition to keyword string). Payload to json is a JSON-encoded dict which is a dictionary with a q field for free text (keywords) and additional key/value pairs for specific fields
     * if q and qjson specified ignore q and just process qjson value

Additional parameters in addition to query (""q"" or ""qjson"") are:
  * limit
  * offset
  * fullinfo=0/1 - return full record for each result (default=0)
  * order_by=field_name
  * search_notes=0/1 (default 0) do we search notes field in a keyword search

Return value is json encoded dictionary with keys:
  * 'results': list of results
  * 'count': total number of results

=== Extras (for the future) ===

  * Prioritisation is not part of this ticket (requires fulltext support in DB or in external app such as Xapian) 

"	1251915845000000	1252340511000000
110	enhancement	dread	dread	v0.10	closed	fixed	Integrate groups into packages	" 1. Package needs a groups property (backref Group.packages).
 2. Put groups in the search api.
 3. Display them on the package, but not editable. (Only editable from the group page.)

Follow up to ticket:105 (Groups).

Cost: 1d"	1252315956000000	1252488660000000
113	enhancement	dread	dread	v0.11	closed	fixed	Simple script to test data migration on a production db	"Must not alter the db. 

Related to ticket:112."	1252316523000000	1265294054000000
114	enhancement	dread	dread	v0.10	closed	fixed	Access Control - model	"Create in the model basic operation of Access Control.

roles table
{{{
name | context | action
-----------------------
admin| package | edit
admin| package | edit-permissions
admin| package | read
editor| package | update
editor| package | read
reader| package | read
This data is set-up on db init and will have no interface.
}}}
user-roles table:
{{{
username | context_type | objectid | role
rgrp     | system  | n/a | admin
visitor  | package | * | reader
bob      | package | geonames | admin
visitor  | package | geonames | editor
visitor  | package | geonames | reader
john     | group | ukgov | admin
dread    | group | ukgov | editor
visitor  | group | ukgov | reader
This data will be added when someone is given permissions for the system, a package or a group.
}}}
Pseudo code:

class Package
    def is_allowed(name, action):
        is_allowed(name, action, context=self)

class Group
    def is_allowed(name, action):
        is_allowed(name, action, context=self)

def is_allowed(name, action, context=None):
''' name: string - a username or IP for 'visitor'
    action: string - 'read', 'edit', 'delete', 'edit-permissions'
    context: object - a Group or a Package or None (which means system)
'''
    # look up user from name.
    # look up in user-roles table what roles this user has for this context.
    # for each roles, look up in roles table what actions are allowed.
    # return True if action is allowed, else False.
"	1252494527000000	1253034529000000
115	enhancement	dread	dread	v0.10	closed	fixed	Access Control - wui constrained by model	"Based on a section of AccessControl design:
Reading and writing to packages in the WUI and REST API are now dependent on the authz tables."	1253034394000000	1253091426000000
116	enhancement	dread	dread	v0.10	closed	fixed	Access Control - edited in wui	"Based on a section of AccessControl design:
WUI gives controls to user and administrator to change permissions on a package."	1253034802000000	1253709460000000
117	enhancement	dread	dread	v0.10	closed	fixed	Access Control - group core functionality	"Based on a section of AccessControl design:  
Group reads and edits are controlled by access control. WUI and REST interfaces covered."	1253271333000000	1254735855000000
118	enhancement	dread	dread	v0.10	closed	fixed	Use paginate in webhelpers	"Take out import of paginate in setup.py.
Use paginate in webhelpers instead.
Make changes to take account of any i/f changes."	1253273657000000	1253784902000000
119	enhancement	dread	dread	v0.10	closed	fixed	Ensure non-active packages don't show up	"Ensure pending packages don't show up in search or browse

cost: 4h"	1253529414000000	1253791147000000
120	enhancement	dread	dread	v0.10	closed	fixed	Security audit	"Look for all places where model is accessed and check authorization is checked.

Document holes (and, as necessary, suggestions for fixes) as new tickets. Likely areas that need looking at:

 * search i/f
 * package WUI commit

Write holes are obviously much more significant to us than read holes."	1253529427000000	1254406544000000
121	enhancement	dread	dread	v0.10	closed	fixed	Add 'Group' to main menu	And associated page to browse group.	1253694827000000	1253716782000000
122	enhancement	dread	dread	v0.10	closed	fixed	Add Group authz page		1253694842000000	1253716757000000
123	enhancement	dread	dread	v0.10	closed	fixed	Ability to edit Group in WUI	"Add Group editing page.

If no permissions to change group can't edit group. Also cannot view edit page.

Editable attributes: name, title, description

No preview needed

"	1253708041000000	1254321447000000
125	enhancement	dread	rgrp	v1.0	closed	duplicate	Edit Generic Package Attributes in WUI	Split out from ticket:43	1253709712000000	1258377621000000
126	enhancement	dread	dread	v0.10	closed	fixed	Change package state in the WUI (delete and undelete)	"As a Package Admin I want to change the state of the package. In particular I wish to delete and undelete it.

(NB: this is quite separate from ""purging"" objects which is the term we shall use for irrevocable removal of an object from the domain model).

  * Only Package Admins (and sysadmins) should be able to change state

== Implementation Suggestions ==

  * 'delete' action should be renamed to 'change-state' (NB: this requires a db migration ...)
  * Have new package formalchemy form (created via inheritance?) to incorporate state attribute. Suggest this is rendered as a dropdown (and may be simple object rendering of state, i.e. do NOT need to change it to a single name such 'active').
  * This form should then be used when the user satisfies is_authorized(..., model.Action.CHANGE_STATE) instead of the usual fieldset
"	1253789571000000	1254740244000000
127	enhancement	dread	dread	v0.10	closed	fixed	Minor form and UI improvements	" * group form: description field for  needs to be a text area
 * group form: make title and name field wider
 * package view: need to move tags and license above notes
"	1253798659000000	1253868048000000
128	enhancement	dread	rgrp	v0.11	closed	fixed	Add ckan_url attribute to REST JSON representation of a Package	"Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN.

Cost: 30m"	1253866713000000	1275694573000000
130	enhancement	dread	rgrp	v0.10	closed	fixed	Create a New Group via the WUI	"== As a ==

User

== I want to ==

Create a new group via the WUI

== Details ==

  * locate at /group/new/
  * You must be logged in to create a group
  * Group creator is automatically given role of group admin
  * Edit screen is same as edit screen ..."	1254130212000000	1254735501000000
131	enhancement	dread	dread	v0.10	closed	fixed	Groups REST interface	Controlling Groups through a REST interface.	1254307959000000	1254308115000000
141	task	dread	rgrp	v0.11	closed	fixed	Search api docs	"Write up search api documentation and put it in a template that shows up at api/search/. 

Cost: 2h"	1254903008000000	1255007583000000
147	enhancement	dread	dread	v0.11	closed	duplicate	Parser and loader for esw.org data		1255440695000000	1255515162000000
148	enhancement	dread	dread	v0.11	closed	fixed	Parser and loader for esw.org data	" * parse like data4nr code

Follow up tickets: ticket:149 and ticket:150"	1255440696000000	1255515222000000
149	enhancement	dread	dread	v0.11	closed	fixed	Loader for esw data recognises existing packages	"Follows on from ticket:148.
Sister to ticket:150."	1255440751000000	1257414916000000
150	enhancement	dread	dread	v0.11	closed	fixed	Loader for esw data uses REST API	"(instead of sqlalchemy interface, to facilitate general usage of the loader)

* use ckanclient

Follows on from ticket:148 and ticket:149"	1255440828000000	1256751974000000
151	enhancement	dread	rgrp	v0.11	closed	duplicate	User object should have a created attribute	"User object should have a ""created"" attribute initialized to current datetime.

Require a db migration but o/w very simple.

Cost: 1.5h"	1255589694000000	1257414545000000
152	enhancement	dread	dread	v0.11	closed	fixed	Package has editable 'Extra' fields	"Package edit page has fields for extra key-value pairs. Don't use formalchemy. The form displays the existing ones and new ones as follows:

{{{
Key           Value            delete
[ country   ] [ uk, spain   ]  [x]
}}}

When presenting the form, existing pairs and 3 blank pairs are provided.

When saving the data, keys that are not changed use the existing !PackageExtra object (perhaps with new value). Deleted keys put the !PackageExtra state to deleted. For new keys, only create a new !PackageExtra if there isn't a state=deleted one to resurrect.

Validators ensure no repeated keys."	1255615316000000	1256056193000000
153	enhancement	dread	dread	v0.11	closed	duplicate	Group's packages listed alphabetically	"This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there.

Suggested by Jonathan Gray"	1255621515000000	1258971895000000
155	enhancement	dread	dread	v1.0	closed	duplicate	Adding multiple packages to a group	"Ability to add multiple packages to a group in one go (e.g. with
'add' link which makes drop down menu appear - so can add one after
another - then submit simultaneously)

Use a bit of javascript to add more dropdowns.

Suggested by Jonathan Gray
"	1255621779000000	1271760041000000
157	enhancement	dread	rgrp	v0.11	closed	fixed	Log message not persisted through preview	"Log message on package edit page is no longer persisted through a preview (no doubt due to recent form upgrades). Let's fix this.

cost: 0.5h"	1255694396000000	1256062680000000
158	enhancement	dread	dread	v0.11	closed	fixed	Add 'extras' to REST API	" * Ensure you can add/change/delete extra fields from packages using the REST API. 
 * Update API docs to cover this.

On Package get, ensure separation of extra attributes from other attributes.

What is the meaning of posting an incomplete dictionary (updating a package)?
Option 1 - if field not present, leave field alone
 * Delete field if field value is None

Option 2 - user must supply all fields
 * Delete field if field absent

Prefer option 1. Document this.

Follows on from ticket:152"	1255703259000000	1255949818000000
159	defect	dread	rgrp	v0.11	closed	fixed	Searching for tags:... resulting in lots of tags being found	"Search of form: tags: ... behaves differently depending on whether there is a leading space:

  * tags: postcode - tags found correctly but no packages found
  * tags:postcode - tags incorrectly found but correct packages found

Let's fix this.

Cost: 0.5h"	1256030097000000	1256060264000000
160	enhancement	dread	rgrp	v0.11	closed	fixed	Implement internal links in CKAN notes markup	"Suggest form is: key:value (with optional space after ':'.

Keys would be for an object, i.e. 'package', 'tag', 'group'.

Becomes a link.

This would be implemented by implementing a 'Parser' or 'Processor' object which is run before markdown is done. Regex replace.

Cost: 4h"	1256030751000000	1265312986000000
161	defect	dread	rgrp	v0.11	closed	fixed	Tag listing is incorrect	"E.g. http://ckan.net/tag/read/postcodes this has no-postcode listed but when you look at that package 'postcodes' isn't a tag ...

I suspect this may be to do with search using deleted tags as I think no-postcode had postcodes as a tag at some point earlier ..."	1256030925000000	1258573607000000
162	enhancement	dread	rgrp	v0.11	closed	fixed	Tags should be automatially lowere-cased if not already	"Rather than raising a validation error automatically lower-case tags on submission.

Cost: 0.5h"	1256048269000000	1256142462000000
166	enhancement	dread	dread	v0.11	closed	fixed	Add extras editing to ckanclient	Ensure logging is working on server.	1256203581000000	1265891199000000
167	defect	dread	dread	v0.11	closed	fixed	REST interface you can't update multiple tags	Only first tag is updated.	1256209993000000	1265891178000000
169	enhancement	dread	dread		closed	duplicate	Package derivations	"A 'Derived' relationship can be applied from one package to another.

e.g.
sussex-demography is derived from census-2001

'Derived' relationship is:
 * directional
 * many:many
 * stateful

'derived' table columns:
 * id (primary key)
 * source_package (foreign key)
 * result_package (foreign key)
 * description (markdown text)

Further tickets:

 * WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y
 * WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate
 * REST if - expose reading and writing this property"	1256304927000000	1266928708000000
176	enhancement	dread	dread		closed	duplicate	Package dependencies	"(Related to ticket:169 - Package derivations)

A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.)

e.g. london-traffic-visualisation depends on road-map

'Dependency' relationship is:
 * directional
 * many:many
 * stateful

'dependency' table columns:
 * id (primary key)
 * dependent (foreign key)
 * dependency (foreign key)

Further tickets:

 * WUI - package view - have list of dependencies (do not need to list packages which depend on this one)
 * WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package')
 * REST api - expose reading and writing 'depends on' property.

=== Issues ===

 * How do we deal with dependency at a particular version?"	1257162812000000	1266928721000000
179	defect	dread	rgrp	v1.0	closed	fixed	Restore 404 and 500 messages in WUI	"Cost: 0.5h (?)

Conjecture this went missing in cset:a35db862a841"	1257412668000000	1265305549000000
184	enhancement	dread	rgrp		closed	fixed	Stats page	"Create /stats/ page displaying main statistics, e.g:

  * Most highly rated packages
  * Most edited packages
  * Largest groups
  * Top tags (by packages)
  * Package addition rate
  * Users with most packages

Related to ticket:181 - Stats side-bar

== Details ==

  * Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display)
  * For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see
   * http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py
   * http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html"	1257534756000000	1266837414000000
189	enhancement	dread	dread	v0.11	closed	fixed	Multiple download urls	"See also: PackageResources which defines a 'distribution' as a package's payload (perhaps with metadata). In our case, at least within CKAN, the representation of the distribution will just be the reference to get it (plus some minor metadata). We are expanding this functionality to be any kind of 'resource' associated with a package since it is likely in some cases people will want to link to a file containing a part, but not the whole, of a package's data.

== As a ==

User (package editor and package user)

== I want to ==

(package editor) 
Associate multiple distributions with a package.

(package user)
Easily have access to multiple distributions of a package should they exist.

== Implementation ==

There is an object called 'Resource' with the following attributes: 

 * url (text) (required in UI, but not in db table)
 * format (text)
 * description (text)
 * package (association) - a resource is always associated with 1 package

A package's list of resources will have an order which is preserved. The first in the list will be known as the 'default resource' (the presumption will always be that this is a 'distribution').

=== How to replace download url ===

download_url removed and replaced as a proxy attribute. Read returns the url of the default distribution. Write changes the url of the default resource. Ensures backward compatibility.

REST API has new attribute in package dictionary 'resouces' - an ordered list.

In WUI, replace download url in package read with a table  of the distributions. 

In the WUI package edit, be able to add/remove/edit/reorder distributions.

== Example ==

Example part of package:

=== Distributions: ===
|| !http://some.com/pollution.pdf || PDF || Original report ||
|| !http://some.com/pollution.csv || csv || Original data ||
|| !http://some.com/pollution || sparql || Linked version of data ||

Existing data to be migrated to new structure.

Cost: 4 days
"	1258386120000000	1265284961000000
196	defect	dread	dread	v1.0	closed	fixed	RDF URI to resolve on CKAN	"Provide RDF version of a package at the RDF URI:
http://ckan.net/package/rdf/32000-naples-florida-businesses-kml

See email on OKFN help:

For the information to be Linked Data, the following URI (from the
api.talis.com SPARQL call) should be resolvable to RDF, or an RDFa
enhanced HTML page."	1258727988000000	1265389771000000
197	enhancement	dread	dread	v0.11	closed	fixed	Simple dumper	"== As a ==

admin / user 

== I want to ==

Admin: export all HEAD data to JSON / CSV / RDF format dump.

User: wants to get hold of this dump.

== Details ==

 * Just do JSON & CSV for now.
 * Put this in CLI.
 * Just do list of packages - as_dict info. CSV needs a bit of flattening.

Cost: 2h"	1258972495000000	1258989700000000
203	enhancement	dread	dread	v0.11	closed	fixed	show package counts for groups in WUI	"In two places in the WUI

 * list of groups
 * group page in title e.g. ukgov (5)

Cost: 1h"	1260187690000000	1265891080000000
204	enhancement	dread	dread	v0.11	closed	fixed	Custom package input form	"== As a ==

admin

== I want to ==

input government data in a constrained way 

== so that ==

you get consistency in the db which is easily searched, discovered and automatically linked.

== implementation ==

Form has fields suitable for government data.
(related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

'''On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)'''

=== Simple fields ===

 * all basic ckan fields
 * default license to OKD::Crown Copyright
 * co_id, precision, taxonomy url, agency

=== Date fields ===

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

 * date_released

 * date_updated

 * temporal_coverage_from, temporal_coverage_to - two date fields combined

=== Strict selection ===

Drop down combo

 * department

=== Selection or free text ===

Editable combo

 * update_frequency - selections: never, weekly, monthly, annually, discontinued

 * geographic_granularity - selections: national, regional, local authority, ward, point

 * temporal_granularity - selections: yes, months, weeks, days, hours, point

=== Specialised fields ===

 * geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

 * national_statisic - UI is check box, store as 'Yes' or 'No'

 * resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.

 * categories - multi values from strict selection of ONS categories. Stored as comma-separated.

=== Field data processed ===

 * tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.


=== Future - optional ===

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

"	1260188965000000	1285082169000000
207	enhancement	dread	nickstenning	v1.0	closed	fixed	Reimplement tag selection in jQuery	"Remove the current dependency on Yahoo YUI toolkit for doing tag autocompletion on package edit forms.

The (KForge-derived) behaviour files for CKAN can easily be switched to the new jQuery-based ones: see ticket:212. Once this is done, it would make sense to implement the tag-selection code with jQuery.

Relevant pages include:

 * http://bassistance.de/jquery-plugins/jquery-plugin-autocomplete/
 * e.g. http://ckan.net/package/edit/acawiki"	1260200340000000	1271250740000000
208	enhancement	dread	rgrp	v0.11	closed	fixed	Obfuscate author email and maintainer email field	"Obfuscate author email and maintainer email field.

Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@jones.com', encode='javascript')

Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate?
"	1260203938000000	1265891070000000
213	enhancement	dread	dread	v0.11	closed	fixed	Automate CKAN deployment	fabric script to automate ckan deployment	1260811707000000	1265891061000000
221	enhancement	dread	dread	v1.0	closed	fixed	UI Review - Package edit	"Package edit page [/package/edit/mypackagename]
  * Ability to add the package to groups on this page.
  * Fields could be grouped.
  * Inconsistent capitalisation (Url -> URL).
  * Tag editor YUI script appears to break occasionally.
  * Perhaps move to two column forms to save vertical space?
  * Notes field to monospace?
  * Extras fields need some jQuery love -- shouldn't have a fixed number available.
  * Typography is a bit of a mess.
	
"	1260880175000000	1271756757000000
229	enhancement	dread	dread	v0.11	closed	fixed	Resource hashes	"New field for resources - hash of the resource file.

 * CKAN itself will not calculate the hash value - user just pastes it in.
 * Display text field in resource table.
 * Requires migration script.

Questions for the field's value:

1. Which hash to use? Restrict to python hashlib and other major hash libraries.

2. Should we use merkle trees?

Thanks to Julien D'Assanges for the suggestion.
"	1262686287000000	1265891612000000
230	task	dread	dread	v1.0	closed	fixed	Refactor 'package preview' to use 'package read'	" * Requires Package Read to not use the side-bar
 * Need to sync params to a package object that is taken out of the session so that it doesn't go into the database."	1262788084000000	1265293953000000
239	task	dread	dread	v1.0	closed	fixed	Test migrate scripts	"Integrate new test upgrade into testmigrate stuff. 

New ""migrate.ini"" config file programmed with the db to be used for migrate tests and a param saying where the db dump is."	1264439926000000	1265286521000000
247	enhancement	dread	rgrp	v1.0	closed	fixed	Alternative formats links on package page	"""Alternative formats"". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f). 

  * How do we know if RDF available? Ans: see details in ticket:90.

Cost: 1h"	1265890326000000	1271248813000000
248	enhancement	dread	rgrp	v1.0	closed	fixed	i18n of forms	Need to do i18n of forms (not in templates but in code). Should consult with tonfa about this.	1265892313000000	1270745304000000
252	enhancement	dread	johnbywater		closed	invalid	Change revision object so that it has parent(s) attribute		1266519767000000	1296477560000000
254	enhancement	dread	dread	v1.0	closed	fixed	Package relationships - 1. model	"New domain object: PackageRelationship (revisioned)

Attributes:
  * subject (Package reference)
  * object (Package reference)
  * type (string)
  * comment

Relationship type values:
depends_on (dependency_of)
derives_from (has_derivation)
child_of (parent_of)

Relationship type is stored as a string, as given in the first column. The reverse relationship (bracketed) is given just for display purposes only.

Subject / Object - as in a sentence: ""Dracos is derived from OSM"" is <subject> <predicate> <object>. We'll use 'type' instead of 'predicate' as that is a more familiar word.
"	1266928449000000	1273596180000000
255	enhancement	dread	dread	v1.0	closed	fixed	Package relationships - 2. Read in WUI	"WUI:
  * View: show both sides of the relationship (but think carefully -- e.g. a given package may have *many* dependents ...)
"	1266928542000000	1273596174000000
257	enhancement	dread	dread	v1.0	closed	fixed	Package relationships - 4. Read in API	"API:
  * Appear in package listing Example: 'relationships': [{'is_dependency_of':'osm', comments:'Since version 0.2'}, {'is_parent_of':'bobs_maps'}]
"	1266928630000000	1273596170000000
260	enhancement	dread	rgrp	v1.0	closed	fixed	Display group package list in standard (richer) way	"Currently use old ""bullet-point"" style. Should change to now standard style used on e.g. package search results page.

Cost: 0.5h"	1267100660000000	1267122137000000
261	defect	dread	rgrp	v1.0	closed	fixed	License does not work through REST API	"Presence of license_id field in JSON input show below breaks REST API (with weird error about int not being subscriptable as if license_id field is being subscripted).

Using normal license field does not cause breakage but has no effect. 

(Also think we should remove license_id from the API since there is no guarantee this cross system compatible since locally generated on install).

{{{
    ourjson = '''{
   ""maintainer"" : ""Applications Division"",
   ""version"" : ""1999-01-01"",
   ""license_id"" : 9,
   ""name"" : ""geogratiswaterfraction1km_v100"",
   ""author_email"" : ""..."",
   ""author"" : ""Dissemination Branch"",
   ""tags"" : [
      ""GeoGratis"",
      ""Canada"",
      ""GCMD"",
      ""Hydrosphere"",
      ""Ground Water"",
      ""Surface Water"",
      ""Hydropattern"",
      ""Earth Science"",
      ""Land Surface"",
      ""Land Cover"",
      ""Landscape""
   ],
   ""extras"" : {
      ""License"" : ""http://geogratis.ca/geogratis/en/licence.jsp"",
      ""Ministry"" : ""Natural Resources Canada"",
      ""Level of Government"" : ""Federal""
   },
   ""maintainer_email"" : ""richard.fernandes@nrcan.gc.ca"",
   ""notes"" : ""This product is a raster coverage"",
   ""url"" : ""http://geogratis.cgdi.gc.ca/geogratis/en/collection/metadata.do?id=67"",
   ""title"" : ""Geogratis - 1-km Water Fraction From National Topographic Data Base Maps, Canada""
}
'''
    def test_15_random(self):
        offset = '/api/rest/package'
        import urllib
        tmpjson = simplejson.dumps(simplejson.loads(self.ourjson))
        postparams = urllib.urlencode({tmpjson: 1})
        res = self.app.post(offset, params=postparams, status=[200],
                extra_environ=self.extra_environ)
        model.Session.remove()
}}}"	1267134778000000	1267207776000000
264	defect	dread	rgrp	v1.0	closed	fixed	Investigate why there are revisions with no associated packages	"Revision.packages should not include all changes so why are we getting revisions with no associated changes? See e.g. http://ckan.net/revision/read/814e39e1-2540-4449-90d1-ac9a92ab40fb?

Investigate and fix.

NB: if this is bugfix this should go against stable or metastable branch if at all possible and then be merged into default.

Cost: 1.5h (?)"	1267605428000000	1272390013000000
275	defect	dread	rgrp		closed	fixed	Add forms renderer for JsonType fixing bugs in admin interface	"This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type)

Cost: 0.5h

{{{
from ckan.model.types import JsonType

forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer
}}}"	1269122740000000	1281002082000000
276	defect	dread	rgrp	v1.0	closed	fixed	Resources in Package form seen multiple times upon preview	Create a new package with a name 'blah' and resource format 'blah'. Hit preview. There is an error because of the lack of resource url, but in the resource input boxes, there are now four resources with format 'blah'.	1269255399000000	1272996237000000
278	defect	dread	dread	v1.0	closed	fixed	Spaces in extra field keys	If the key to an extra field has a space in it then various form code doesn't work. Either need to disallow spaces or cope with them.	1269520845000000	1271173752000000
280	defect	dread	dread	v1.0	closed	fixed	Disappearing extra fields	"When I edit a package on no.ckan.net with lots of extra fields, lots of them completely disappear.

Metastable branch."	1270209684000000	1271173769000000
281	enhancement	dread	dread	v1.0	closed	fixed	Refactor forms to be plugin-able		1270542248000000	1270723675000000
286	enhancement	dread	dread	v1.0	closed	fixed	Refactor forms templates	"Use Nick's examples as a basis to revamp the formalchemy usage, principally in the package edit form.

* http://static.whiteink.com/css/examples/forms.html
* http://whiteink.com/temp/ckan-forms/"	1270723513000000	1270723629000000
287	enhancement	dread	dread	v1.0	closed	fixed	Customise Canadian package edit form	"Spec - see attached, which is a copy of the Pirate Pad from Lauren: 
http://piratepad.net/2C2iwiLDhd"	1270801086000000	1270801210000000
288	defect	dread	dread	v1.0	closed	fixed	Foreign and escaped characters in extras fields	These don't work.	1271171899000000	1271173777000000
292	defect	dread	dread	v1.0	closed	fixed	PackageResources are created new on every package edit	When you edit a package, all of its resources are deleted and a fresh set created. This causes all history of them to be lost.	1271755964000000	1272286005000000
295	enhancement	dread	dread	v1.0	closed	fixed	Add multiple packages when editing a group	"In Group edit page, you can only add one package at a time. It would be better to use some scripting to allow adding more than one at once, to make it easier and quicker when you have a long list to add.

Ticket is split off from ticket:221

See also ticket:155 (duplicate)."	1271756713000000	1272384758000000
303	defect	dread	dread	v1.0	closed	fixed	Package history & diff & feed to show changes to tags, extras and resources	"Currently a package's history page (/package/history/xyz) and related feed only shows changes to the core package i.e. ones which create a PackageResource. This doesn't include changes to tags, extras and resources. These need to be added.

There is the complication that the 'diff' function doesn't currently work beyond PackageResource. Unless this is quick and simple to fix, I suggest we let the diff remain slightly broken for now.

Rufus suggests alongside the existing methods:
{{{
pkg.all_revisions()
pkg.diff()
}}}
we have ones which also include the related objects (tags, extras, resources):
{{{
pkg.all_revisions_full()
pkg.diff_full()
}}}
You could have a function which returned these related objects. Ideas:
{{{
revobjlist = [ self, resources, extras ]
list of tuples: (revobj, changed_revobjs)
list of tuples: (revobj, {objtype: [changed_revobjs]})
tuple: (rev1, {Package: [ object_id, object_id], PackageTag: [object_id, object_id2] ...}
}}}"	1272370659000000	1272989728000000
310	defect	dread	rgrp	v1.1	closed	fixed	Commit message box looks wrong in edit page since edit style overhaul	"Suggest move this below the label and make full width of screen and only 3/4 rows high (more like a wiki site).

  * Also change label to: Edit summary (Briefly describe the changes you have made)
  * Remove: you can markdown formatting here.
  * Move author: if you have not signed in smaller and closer (like markdown instructions are nwo).
  * Change commit -> save
  * Remove ""please save"" just have the bullet points"	1273348714000000	1279300525000000
311	defect	dread	rgrp	v1.1	closed	fixed	Reordering of package resources can lead to integri	"Created a new package resource line and then moved it up above existing one and hit save. Result: 500 error. In logs have:

{{{
[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] Error - <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint ""package_resource_revision_pkey"", referer: http://ckan.net/package/edit/cofog
[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30]  'INSERT INTO package_resource_revision (id, package_id, url, format, description, hash, position, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(url)s, %(format)s, %(description)s, %(hash)s, %(position)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'hash': '', 'description': 'The Treasury record of COFOG functions. ', 'format': 'XLS', 'url': 'http://www.hm-treasury.gov.uk/d/cofog_definitions_coins250609.xls', 'package_id': '8482334d-fe2e-4285-9114-5243130f80c0', 'state': 'active', 'continuity_id': '8bf302db-8a80-47d3-b5dc-bc07512a3928', 'position': 3, 'revision_id': 'e4e2cb2d-4bd5-414e-b646-e484f174d9ab', 'id': '8bf302db-8a80-47d3-b5dc-bc07512a3928'}, referer: http://ckan.net/package/edit/cofog
}}}
"	1273348968000000	1274282065000000
315	defect	dread	rgrp	v1.1	closed	fixed	Improvements and fixes to csv dump	"1. Issues with quote in fields: http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html

2. Issues with package resource serialization into csv table.

<quote>
In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent.
</quote>"	1274100887000000	1275846764000000
319	defect	dread	dread		closed	fixed	Local Authority license misnamed in migration script 18	"Migration script 18 converts license ""OKD Compliant::Local Authority Copyright with data.gov.uk rights"" into ""uklocalauthority-withrights"" when it should be ""localauth-withrights"", according to the license service SoS 2. This causes clients difficulties looking it up.
"	1274349714000000	1274366882000000
320	enhancement	dread	rgrp	v1.1	closed	fixed	site_title configuration variable which is used in template	"As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

  * ckan.site_title config variable
  * set this on g in app_globals.py e.g.
    * from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
  * use in head title and in main site title/logo section (use it as alt on logo image)
  * Also all other pages (e.g. index, about) which talk about CKAN
    * Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:
  * Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
  * Probably yes, but '''not''' part of this ticket."	1274378967000000	1279130535000000
322	enhancement	dread	dread	v1.1	closed	fixed	Client interface for Notification Service	"=== Use cases ===
 * Register for package changes
 * Register for all revisions
 * Notified of a package change
 * Notified of a revision
 * Deregistration
 * Configuration of port in pylons config

=== Design ===
 * Default port: 5672 (standard for AMQP)
 * Exchange name: 'ckan'
 * Exchange type: topic exchange (most flexible)
 * Routing keys: (see below)

=== Routing detail ===
Routing key format: ""OBJ_TYPE""
(NB tags should be identified by their name, not ID)

Example routing keys
 * 'package' - Package edited/created
 * 'resource' - Resource edited/created
 * 'revision' - Any change
 * 'db.clean'
 * 'db.rebuild'

Example queue bindings that clients may use: 
 * * - no filtering - client receives all notifications
 * package - only changes to packages
 * revision - all revisions
 * db - all database operations

=== Versioning ===
Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps?

=== Documentation ===
   * How to use
   * simple example of an external client?"	1274720042000000	1277722821000000
323	enhancement	dread	dread	v1.1	closed	fixed	Notification message	"=== Which events to notify on ===

Listed by domain object, these are the notification message 'change types' that will be sent:
 * Package
 * !PackageResource

Also it is clear that it could be useful to know when db-wide maintenance is carried out:
 * db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration)

=== Ignored domain objects ===
These parts of the domain model will not carry notifications as no use case has been identified for them:
 * Revision
 * Group 
 * Tag
 * Rating
 * User - list of users is sensitive info
 * Relationships - complicated
 * Authz - complicated and sensitive info
 * License - change of a license's metadata is a question for the 'license service'

=== Message format ===

A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity.

For the 'db' notifications there shall be no payload."	1274723333000000	1278578841000000
324	enhancement	dread	dread	v1.1	closed	fixed	Search indexing using notifications	"Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead.

The indexing shall run in a separate shell/process, managed by supervisord."	1274723483000000	1278599927000000
325	enhancement	dread	dread	v1.1	closed	fixed	Event push notification	"=== As a === 

CKAN client program

=== I want to ===

be notified when changes to the CKAN metadata occur.

=== Examples of use ===

 1. An external search engine needing to (re)index a package. (interest: Package)
 2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333.
 3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: !PackageResource)
 4. Do some processing on resource (e.g. extract sample data for display) (interest: !PackageResource)

=== Context ===

The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling.

=== Design ===
Split-off into two tickets:
 * Notification message - ticket:323
   * Which events to notify on
   * Message format
 * Interface for Notifier Service - ticket:322

=== Testing ===
To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine.

=== Related ===
 * Run CKAN search indexing using this system - ticket:324
 * Run SOLR indexing using this system - ticket:353
 * Web hooks for notifications - ticket:327 "	1274723512000000	1278599979000000
326	task	dread	dread	v1.1	closed	fixed	Centralise importation of json library	Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers.	1274784223000000	1274789296000000
328	enhancement	dread	rgrp	v1.1	closed	fixed	Mention code libraries in API documentation	"Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

  * Python: ckanclient
  * Perl: luke closs wrote something
  * PHP: drupal library?
  * ...?"	1274988106000000	1275318745000000
330	defect	dread	dread		closed	fixed	getdata/ons timezone not recognised on non-British servers	"The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error:

DateConvertError: Could not read date as ISO format ""%a, %d %b %Y %H:%M:%S %Z"". Date provided: ""Mon, 04 Jan 2010 09:30:00 BST""

It turns out that it only recognises local names of timezones.

The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems."	1275300271000000	1275303122000000
335	enhancement	dread	dread	v1.1	closed	fixed	Post-package-edit redirect to configurable URL	"== As a ==

third-party interface to a CKAN instance

== I want to ==

link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return.

== Design ==

 1. The 'return URL' is passed as a parameter to CKAN.
 2. CKAN substitutes the package name into the return URL.
 
== Example ==

Front-end links to:
!http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/<NAME> (but with the parameter URL-encoded)

When finished editing and the user commits, CKAN redirects the user to: !http://datadotgc.ca/dataset/pollution_data
"	1275408834000000	1276179605000000
336	defect	dread	donovanhide	v1.1	closed	fixed	Resource Search API	"== As a ==

CKAN client such as !ScraperWiki

== I want to ==

search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL.

== Proposed implementation ==

Add resource search API at:

/api/search/resource

AND resource added to model API at:

api/rest/resource
 (see ticket:358)

Functional differences from the !ScraperWiki suggested patch:

 * URL is not normalised
 
 * URLs are not grouped

 * All fields of the resource object are returned, not just the URL

 * Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend 'http://ckan.net/package/' to the package ID)

This is to make our API more general, simple and consistent. It means that the !ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok?

=== Example search ===

POST 
{{{
{""url"": ""scraperwiki.com/"", ""all_fields"": 1}
}}}
to: /api/2/search/resource

returns JSON:
{{{
 [{""id"": ""a3dd8f64-9078-4f04-845c-e3f047125028"",
   ""package_id"": ""b8a325c8-af2a-43f3-8245-9db7d73dfbfe"",
   ""URL"": ""http://scraperwiki.com/lincolnshire-councillors"", 
   ""format"": ""CSV"", 
   ""Description"": ""Scrape of www.lincs.gov/councillors.pdf by ScraperWiki."",
   ""hash"": """", 
   ""position"": 2
 }]
}}}

Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done ticket:341 then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe


=== Search Parameters ===
{{{
Key:  q
Description: Search all resource fields for the value

Key: url / description / format / 
Description: Search particular field for the value

Key: all_fields
Value: 0 or 1 (0 is default)
Description: If 1 (true), the full record of the package resource
(and it's package reference) are returned, rather than just the
PackageResource ID.
}}}

May also choose to introduce 'offset' and 'limit' to page through a large number of results.

JSONP achieved through API-wide parameter - see ticket:342

Search is case insensitive.

== Original request ==

Hi,
have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of #388.

Could also add a search version. Not sure what the best url would be for that though.

Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere?

Donovan"	1275411765000000	1279373842000000
340	enhancement	dread	dread	v1.1	closed	fixed	Web UI theme easier to configure	"An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in.

http://wiki.okfn.org/ckan/doc/theme"	1276009729000000	1328807317000000
342	enhancement	dread	dread		closed	fixed	JSONP parameter in API	"== As a ==

CKAN client using JQuery

== I want to ==

call the CKAN API and instead of receiving back JSON I get JSONP. i.e. ""%s(%s)"" % (callback, json_content)

== Suggested implementation ==

All API calls allow the JSONP 'callback' parameter to be specified in the request and this wraps the JSON response. See suggested patch to rest.py by Donovan Hide:

http://knowledgeforge.net/ckan/trac/attachment/ticket/336/resource.patch

== Test ==

import re
import unittest

def test_jsonp_callback():
  response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com&callback=jsoncallback')
  match = re.match('jsoncallback\(.*\);',response)
  self.assertTrue(match)

  response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com')
  match = re.match('jsoncallback\(.*\);',response)
  self.assertFalse(match)

I think the point needs to be made that JSONP only works for GET
requests and not POST/PUT/DELETE, so there needs to be a check
for that in the _finish_ok method.

(thanks to Donovan Hide for test)"	1276166426000000	1276278485000000
345	requirement	dread	johnbywater		closed	invalid	metastable, stable & ultrastable branches shall be automatically built and tested		1276523083000000	1291831615000000
346	defect	dread	johnbywater	ckan-v1.3	closed	wontfix	Revision search API (response data format and documentation issue)	"Whilst going through the API docs for the revision search API, it was noticed that the ""Gdu"" SoS doc doesn't match up. It returns revision IDs (perhaps this is useful to note in the spec?) so the format is probably not 'limitedstring'. Also, they appear to be ordered youngest first, not oldest as stated.

And in the revision model, it refers to 'simplestring' which it
doesn't define - I guess the names should be 'limitedstring'?

Could this be checked out?"	1276523517000000	1296477510000000
352	enhancement	dread	dread		closed	wontfix	Package notification worker - sends XML-RPC	"== As an ==

external front-end

== I want to ==

be notified (by XML-RPC) about package creations and updates.

== Implementation ==

 1. A message queue worker waits for package update notifications
 2. On reception, it constructs XML detailing the changes and PUSHes it to a configured URI.

Rather than turning the package fields into XML fields, the JSON dump of the list of package dictionaries will become a single XML parameter.

Config - in the CKAN config will be:

 * URI to callback to
 * API version to use (version 2 gives packages referred by ID not name)"	1276597996000000	1286375870000000
362	defect	dread	dread		closed	invalid	Ratings should not be created with a GET	"== Background ==

In the Web UI, when you rate a package it simply links to something like:

http://ckan.net/package/rate/mke-liquor-licenses?rating=3

This creates a GET request. 

This is bad because:

 * Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this)
 * There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept GET> Apache instruction in. But the database may still get written for these ratings.
 * Best practise for web requests is for GET to be a read-only request."	1278925451000000	1311176564000000
365	enhancement	dread	dread		closed	fixed	ResourceNotifications	If you change a resource then you not only get a PackageNotification, but also a ResourceNofication.	1279037411000000	1279300621000000
367	enhancement	dread	dread		closed	fixed	Notfication monitor	Runs on the commandline and prints out notifications.	1279303310000000	1279303693000000
373	requirement	dread	johnbywater		closed	fixed	Sort out gov daily script	"Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally.

Requested by DGU."	1279885887000000	1286376176000000
390	task	dread	johnbywater	ckan-v1.2	closed	fixed	Remove a package for SB		1280418625000000	1282214629000000
399	task	dread	johnbywater	ckan-v1.2	closed	fixed	Import BIS data for SD		1281000890000000	1282294177000000
420	task	dread	dread	ckan-v1.2	closed	fixed	Release ckan 1.1		1281432518000000	1281522091000000
421	task	dread	dread	ckan-v1.2	closed	fixed	Upgrade test-hmg.ckan.net to latest ckan 1.1		1281432541000000	1282909772000000
423	task	dread	johnbywater		closed	fixed	Add examples to documentation about uploading data		1281435260000000	1282812768000000
424	task	dread	dread	ckan-v1.2	closed	fixed	Tell JF the format spec & examples		1281436534000000	1282919429000000
426	task	dread	dread	ckan-v1.2	closed	fixed	Ask list for suggestions improving API docs		1281436574000000	1282925305000000
432	defect	dread	dread	ckan-v1.2	closed	fixed	Creating package over REST gives 500 error	"This occurs when CKAN is run with mod_wsgi (not under paster).

This is because of a unicode header being creating in rest.py."	1282065101000000	1282552819000000
438	requirement	dread	dread	ckan-v1.3	closed	invalid	Data importers use minimal processing	e.g. ONS importer	1282224383000000	1294661079000000
440	task	dread	dread		closed	duplicate	Write and pass comprehensive performance tests	"Run latest ckan on eu0.
Automate some queries and searches.
Check load and database connections / processes."	1282226932000000	1294417436000000
441	requirement	dread	dread	ckan-v1.3	closed	duplicate	CKAN read-only state	"When performing maintenance on CKAN it may be necessary to make CKAN obviously read-only, telling the users and restricting access to 'edit' pages.

Examples of use:
 * Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done.
 * A CKAN is used just for distributing metadata and so is always read-only. Updates may still arrive through direct db manipulation, e.g.:
   * another (but writable) CKAN instance is connected to the same db
   * restoring database dumps from another CKAN db
 * Should a security be breached, all editing could be stopped

"	1282227314000000	1292586309000000
442	requirement	dread	dread	ckan-v1.3	closed	worksforme	Importing data	"Review what chunks of code we have and how to move forward with requirements.

Requirements:

 * Bulk parsing is done away from the CKAN server

Possible future requirements:

 * /importer for importing spreadsheets in WUI
 * validation-light - 'scoring' a package

Current importers:

 * /importer controller uses: PackageImporter, form, PackageSaver
 * loader uses: package_importer -> API
 * API uses: form, PackageSaver


Modules

 * package_importer:
   * xls/csv --SpreadsheetData--> table
   * table --DataRecords--> records
   * records --PackageImporter--> pkg_dicts

 * pkg_dict --form--> fs & validation
 * fs --PackageSaver--> package"	1282228977000000	1294661028000000
443	task	dread	dread	ckan-v1.2	closed	fixed	Get DGU daily dump working (local)	"* Producing daily dump file
* Publishing at ckan.net/dumps/ or better"	1282299046000000	1282735977000000
451	task	dread	dread	ckan-v1.3	closed	invalid	Show/Allow only departments associated with user	Use drupal api	1282299641000000	1292587233000000
454	story	dread	dread	ckan-v1.2	closed	fixed	Import ONS data via API		1282299851000000	1286376044000000
457	story	dread	dread	ckan-v1.2	closed	fixed	Dump database on a daily basis		1282299921000000	1282909051000000
461	task	dread	johnbywater	ckan-v1.2	closed	fixed	Change ONS data importing to work via API	"  * Move script out to ckandgu repo
  * Change script to convert xml into package dicts
  * Test (against test.ckan.net, hmg.test.ckan.net)
  * Deploy"	1282303411000000	1283250478000000
462	story	dread	johnbywater	ckan-v1.2	closed	fixed	Visitor downloads a daily database dump		1282305305000000	1285757238000000
466	requirement	dread	johnbywater	ckan-v1.3	closed	wontfix	Name of HTTP header for API key shall be configurable		1282308995000000	1294835610000000
470	task	dread	johnbywater	ckan-v1.3	closed	fixed	"Adjust API key's HTTP header name on DGU ""live"" and ""test"""		1282310854000000	1292587187000000
474	task	dread	dread		closed	wontfix	Questions related to fi.ckan.net	"Several questions to answer (and see file in email). To respond to the Finnish team and copy all Q&A to the ckan-discuss list.

1 ) I have played with two different datasets to get the metadata from those imported to CKAN. The other dataset is in PCAXIS format and other one is in Excel format. Please find the scripts + sample files attached in a zip file that contains instructions too to run the scripts. I looked at import scripts written in object oriented manner and getdata samples in CKAN but it was easier for me too write these from scratch (copied the unicrap function from CKAN samples). If I run the PCAXIS import script I get Unicode decode error (see errors file). When I use the unicrap function then the error does not appear in logs. I assume this is some Python environment setting issue. Of course I would like to run the 
script without unicrap function to get scandic letter imported correctly to ckan(and displayed in WUI). In Excel case there is no scandic letter issue which is kind of strange but this probably relates how the Excel library handles encoding.

To be honest the whole unicode/encoding/decoding issue has been quite difficult to deal with and debug. I hope I would get proper environment configuration to handle scandic letters.


2 ) It seem in CKAN by default the localisation file contains 462 strings. The transifex link Rufus sent contains 400 strings. Is transifex the one we should use as localisation file ? Where are the rest 62 strings then fetched from or are they needed at all. I have found that as I have been using the .po file coming in CKAN source (which has API translations too) this also causes similar unicode problems when an error in API occurs (the API error messages containing scandic letters provide the similar UnicodeDecoding errors as mentioned above). So, it seems the unicode issues appear in various cases. Maybe a proper environment settings will be able to prevent also these problems... If transifex is the one to use I can prepare it and maybe then you could upload it to fi.ckan.net as there will be also other people checking the translations. To run the test and see the API translation errors especially in pcaxis case I would recommend to install attached FI translations to your environment and include lang = fi configuration. It looks like the API errors does not come when running default lang = en. 


3 ) Is it possible to change the license selection/options available in license field drop-down ? Where are these located and can these be modified locally ?

4) It looks like the values stored in package extra fields are not searchable. Is this intentional ? It looks more or less that all other fields can be searched using free-text search in CKAN. 

5) How package dependencies are defined and managed ? For instance we want to import some packages in a manner that the other package is a child package of another. Does API support this and/or is this possible to do in WUI ?

Best Regards,
Henri"	1282312255000000	1294916760000000
479	task	dread	johnbywater	ckan-v1.2	closed	fixed	Suggest DGU presents database dumps through front end		1282324365000000	1288004211000000
502	requirement	dread	rgrp	ckan-v1.3	closed	wontfix	Data4nr uploading script shall remove deleted packages	"Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed.
"	1282659304000000	1292586466000000
503	task	dread	rgrp		closed	fixed	Provide exemplar package relationships for some packages	"  1. Find some relationships.
  2. Add them via API
  3. Notify DGU of completion and point to exemplars"	1282659631000000	1314031851000000
504	requirement	dread	rgrp	ckan-v1.3	closed	invalid	The DGU fieldset shall conform to v3 of DGU metadata specification	No need for requirement - closing in favour of #505	1282660246000000	1292586686000000
505	enhancement	dread	rgrp		closed	fixed	DGU fieldset supports v3 of DGU metadata specification		1282660416000000	1298368280000000
506	task	dread	rgrp		closed	fixed	Plan transition to DGU metadata-fields v3		1282660490000000	1287080989000000
508	task	dread	dread	ckan-v1.2	closed	fixed	Create branch ultrastable	" * Create a branch named ultrastable that represents latest version of 1.0 ''as found on dgu''
  * Going forward ultrastable will act as stable - 1 (and either ultrastable or stable will be on the rock-solid production systems such as dgu)
 * Do '''not''' deprecate the stable or metastable branch for the present
 * Ask for recommendations on mailing list about best practice for branching (e.g. versioned based versus a simple HEAD versus stable) and how this affects standard operations such as applying bugfixes.
  * For useful discussion of how to do backporting and applying bugfix patches see: http://www.python.org/dev/peps/pep-0374/#backport
  * NB: in general do '''not''' always have to apply in stable/ultrastable and merge upwards for bugfixes -- can use transplant extension as well.
"	1282663663000000	1282908795000000
509	story	dread	dread		closed	invalid	ONS data imported daily		1282665870000000	1291734435000000
510	enhancement	dread	dread		closed	fixed	Setup cron for ons import		1282665939000000	1294138332000000
536	task	dread	dread	ckan-v1.2	closed	fixed	Extend base importer merging in resources	"Multiple ONS records can go into one package e.g. Stats for Jan, Stats for Feb, Stats for Mar etc.

We already identify an existing package by 'external_id'. Now need to find package by title and an extra field (department in ONS case) and allow it to merge in resource and tags. Alert if rest of meta data is different."	1283173313000000	1286376029000000
677	task	dread	dread	ckan-v1.3	closed	fixed	License change script	"* Uses API to change licenses of all packages in a CKAN instance to a particular one.

* Run it on DGU."	1286384756000000	1292587315000000
678	task	dread	dread	ckan-v1.2	closed	fixed	Form for DGU metadata v3	" * Create form
 * Deploy on test
 * Deploy on live
"	1286385712000000	1286979133000000
680	task	dread	dread		closed	fixed	Prepare for DGU tech meeting		1286386074000000	1287141809000000
681	task	dread	dread	ckan-v1.3	closed	fixed	Department field bug	https://trac.dataco.coi.gov.uk/projects/datagov/ticket/613	1286386685000000	1290507180000000
682	enhancement	dread	dread	ckan-v1.4-sprint-1	closed	fixed	Add ckanclient tests to buildbot	Some work done on this to make tests run independently.	1286386769000000	1297358266000000
683	story	dread	dread		closed	fixed	metastable, stable & ultrastable smoketests run by buildbot		1286388134000000	1287142289000000
684	task	dread	dread		closed	fixed	metastable tested on buildbot		1286388181000000	1286452710000000
685	task	dread	dread		closed	fixed	stable tested on buildbot		1286388206000000	1287997008000000
686	task	dread	dread		closed	wontfix	ultrastable tested on buildbot		1286388225000000	1287997047000000
687	task	dread	dread	ckan-v1.2	closed	fixed	Release ckanclient 0.4	"Done.
Added release process here: wiki:ReleaseProcess"	1286452098000000	1286452650000000
731	bug	dread	dread	ckan-v1.3	closed	worksforme	Geo coverage field losses in Form API	Sometimes editing a package via api on dgu results in some countries being lost in geo coverage.	1287738539000000	1288038226000000
732	task	dread	dread	ckan-v1.3	closed	fixed	Reproduce and investigate (geo coverage issue)		1287738761000000	1288038335000000
733	task	dread	dread	ckan-v1.3	closed	invalid	Fix problem (geo coverage issue)		1287738764000000	1288038341000000
734	task	dread	dread	ckan-v1.3	closed	fixed	Investigate department issue (ONS import)	"Creation of duplicate packages named with trailing underscore

ww 22/9/10:

>> Need to ...
>> merge some packages (the ""find packages by attributes""
>> is not 100% reliable because of inconsistent data
>> already in there).

The ""find by tags"" algorithm uses (title, department).
Unfortunately some of the data already there has no
department. This causes it to make a new package (with
trailing underscore).

dread 24/9/10:

I've done a test for the missing department issue and I can't
reproduce the problem. This may be due to my general tidy up and
getting tests passing using the latest ckanclient. Anyway, can you
check over my test to see if this is what you mean?:
dgu/ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadMissingDept

ww 20/10/10:

you should uncover it if you run against (a copy of) the dgu database."	1287747273000000	1292587603000000
735	task	dread	dread		closed	fixed	Investigate 'deduping' (ONS import)	"ww 22/9/10:

(discussion of ticket:509) This leads us to deduping which
should be the next order of business oin this thread...

ww 20/10/10:

if a dataset is missing a department and the ons script makes a new version of it with a department we now have two reccords for the same dataset. these should be manually merged - need a merge tool where you can tell it that two datasets are the same."	1287748786000000	1292957248000000
742	task	dread	dread	ckan-v1.3	closed	fixed	Release ckan 1.2 to metastable	Need to make sure #700 is fixed.	1288002397000000	1289824163000000
777	task	dread	dread	ckan-v1.3	closed	fixed	Put 503-returning middleware into a module	Also configure hmgckan, hmgapi, test-hmg and dev-hmg machines.	1288608324000000	1292586843000000
798	task	dread	dread	ckan-v1.3	closed	fixed	cospread importer using API	Move existing getdata/cospread.py into dgu repo and use the importer/loader mechanism.	1289475362000000	1294916973000000
805	task	dread	dread	ckan-v1.4-sprint-2	closed	fixed	Migration scripts on buildbot		1289816755000000	1298379084000000
823	defect	dread	dread		closed	fixed	Etags cache doesn't update when only non-core package fields changed.	"1. Create a package.
2. Edit package and add a resource.
3. View package in browser - no sign of the resource - ERROR.
4. View package in API - resource is there.

Reason: hash doesn't change because package.revision only takes account of core fields."	1290505078000000	1290506116000000
824	enhancement	dread	dread		closed	fixed	'Getdata' moved out of CKAN	"Data importers should use API instead of object model directly:
 * saves load on server
 * much better error checking and authentication
 * eating our own dogfood (tests the API well)"	1290595507000000	1290595559000000
825	enhancement	dread	dread		closed	fixed	Document fabric tool		1290609160000000	1290624449000000
840	enhancement	dread	dread	ckan-v1.4-sprint-6	closed	fixed	On/off switch for ETags cache	" * Use config variable to switch ETags caching on or off. Consider joining in with 'cache_enabled'.
 * Default setting for (all) caching should be off.
 * Needs documenting in configuration.rst"	1291308727000000	1302694133000000
858	defect	dread	dread		closed	fixed	Tests for diff	Test coverage for ckan.lib.diff is 24%. Need to improve	1291724171000000	1291734297000000
867	enhancement	dread	dread	ckan-v1.3-sprint-1	closed	fixed	ckanclient raises exceptions	"To be more pythonic, raise exceptions when ckanclient gets status which isn't 200.
"	1291915162000000	1299866685000000
873	defect	dread	dread		closed	fixed	ONS loader assigns packages to same package even when dept name added manually	"e.g. national_child_measurement_programme had no department but ""Information Centre for Health and Social Care"" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_."	1292437253000000	1292939274000000
897	defect	dread	dread		closed	duplicate	"""New package"" highlighted in Navigator when searching"	"On ckan.net 1.3.1b, click on ""Search"" in the Navigator. Now both ""Search"" and ""New Package"" are highlighted."	1294648815000000	1294914333000000
919	enhancement	dread	dread	ckan-v1.4-sprint-6	closed	fixed	Package preview contains API & datapkg instructions	"1. Edit a package
2. Hit 'preview'.

The preview contains the section ""CKAN API / datapkg"" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the ""CKAN API / datapkg"" section too?"	1295346730000000	1303202627000000
931	enhancement	dread	dread	ckan-v1.4-sprint-1	closed	fixed	Search results generator hides paging functionality	ckanclient's search results list only packages up to the 'limit'. It would be good to return a generator instead of a list. When the limit is reached on the generator then another 'page' is loaded automatically.	1296147360000000	1298379187000000
932	enhancement	dread	dread	ckan-v1.3	closed	fixed	Move to SQLAlchemy-Migrate 0.6	"Basic steps from kindly:

 * changing the setup.py dependencies for sqlalchemy-migrate and sqlalchemy to their latest versions.
 * running the migrate script on the repository (or changing the repository scripts manually).
 * checking in both these changes at once."	1296156660000000	1297076710000000
950	enhancement	dread	dread	ckan-v1.3	closed	fixed	Move importer controller to ckanext-importer	"ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN.

In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext."	1296666038000000	1297079743000000
955	defect	dread	dread	ckan-v1.3	closed	fixed	Tidy up tests	"* Removal of init_db from every setup etc.
* Get them working"	1296819300000000	1297342534000000
982	enhancement	dread	rgrp	ckan-v1.4-sprint-2	closed	fixed	Remove various pip-requirements versions	"We can just use the branch name to pull down specific versions of pip requirements so we don't need all the different versions in head.

Remove them and update any client systems.

Cost: 20m"	1297516060000000	1298887980000000
983	defect	dread	dread	ckan-v1.3	closed	fixed	'db upgrade' creates system priviledges	(should leave priviledges alone)	1297518265000000	1297773407000000
1002	enhancement	dread	rgrp	ckan-sprint-2011-12-05	closed	fixed	Remove changeset model code	This code is unused and tests are disabled i belive. I suggest stripping this out (can always be re-added in an extension later). If we remove we should probably delete the relevant tables (and so a migration is needed).	1298489773000000	1322515696000000
1005	defect	dread	dread		closed	duplicate	Editing in ckan missing publishers	Package edit form at hmg.ckan.net has no publishers any more.	1298561348000000	1300100085000000
1018	task	dread	dread	ckan-v1.4-sprint-3	closed	fixed	Remove gov form	(It has been moved into ckanext-dgu)	1299072516000000	1299073340000000
1025	enhancement	dread	dread	ckan-v1.4-sprint-3	closed	fixed	Default authz can be set in config	"Currently the default authz for a package is hard-coded to: 
{{{
 <PackageRole user=""visitor"" role=""editor"" context=""Package"">,
 <PackageRole user=""logged_in"" role=""editor"" context=""Package"">,
}}}
This should be configurable in the config, so that you can have a more locked down instance etc. "	1299596110000000	1299751045000000
1026	enhancement	dread	dread	ckan-v1.4-sprint-3	closed	fixed	cli for creating users	It's handy to be able to create users using the cli (e.g. dgu migration)	1299604652000000	1299605128000000
1028	defect	dread	dread	ckan-v1.4-sprint-3	closed	fixed	Open redirect in locale setting		1299759883000000	1299760360000000
1033	defect	dread	dread		closed	fixed	Register user with blank password causes 500	Go to http://ckan.net/user/register and fill in all the fields apart from password. On submit you get 500 error.	1299796274000000	1308310446000000
1038	enhancement	dread	dread	ckan-v1.4	closed	fixed	Authz tool - operate on all packages at once	Add 'package:all' to authz tool to allow mass changes of authz.	1300212788000000	1300212841000000
1039	enhancement	dread	dread	ckan-v1.4	closed	fixed	Default user roles read from config	(instead of being hard coded)	1300212822000000	1300212856000000
1042	task	dread	dread		closed	fixed	'Ckanext' split-up	"It's not good to have ckanext doing lots of different things with different dependencies. Split it off into:
 * ckanext-importlib
etc.

And then deprecate the ckanext repo itself."	1300293907000000	1300969865000000
1045	defect	dread	dread	ckan-v1.4-sprint-4	closed	fixed	Group identified by ID in API	"{{{
http://ckan.net/api/2/rest/group
}}}
returns group IDs but I can only reach a group by name:
{{{
http://ckan.net/api/2/rest/group/economics
}}}
when I also want to get a group by ID:
{{{
http://ckan.net/api/2/rest/group/04fb43d2-8ddf-4485-9bf5-66d47d3672f3
}}}"	1300360642000000	1300793261000000
1048	enhancement	dread	dread	ckan-v1.4-sprint-4	closed	fixed	Complete making groups versioned	" * Deleting a group changes state to 'deleted' rather than purging it
 * Adding authz tests for deleted groups"	1300387655000000	1300702752000000
1052	defect	dread	dread	ckan-v1.4-sprint-4	closed	fixed	Authz holes	"No authz on:
 * Group creation/edit/listing
 * Package relationship create/edit/delete"	1300709144000000	1300895410000000
1053	defect	dread	dread	ckan-v1.5	closed	fixed	Deletion in Model API	"Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted.

There is no way to delete objects at the moment - changes to 'state' are ignored in the API.

Do we need an alternative way to purge objects in the API?"	1300790039000000	1310126546000000
1055	defect	dread	dread	ckan-v1.4-sprint-4	closed	fixed	@search_related tests not running	"Tests marked decorated ""@search_related"" should only be run against postgresql, but in fact they don't get run at all. "	1300985228000000	1300992395000000
1056	defect	dread	pudo	ckan-v1.4-sprint-6	closed	fixed	User links for OpenID users are broken	"Use case:
 * Login using OpenID
 * Click on 'My account' - results in 404

Solutions:

 * User user.id instead of their name
 * Escape the URL properly. "	1301060249000000	1302882616000000
1058	defect	dread	dread	ckan-v1.4-sprint-4	closed	fixed	Give 400 error (not 500) for invalid locale or package_form	"Examples which prompt annoying exception emails:
{{{
http://ckan.net/locale?locale=ja
Module ckan.i18n:21 in set_session_locale
           assert locale in _KNOWN_LOCALES
}}}
A bot has caused these:
{{{
http://ca.ckan.net/package/new?package_form=gov
Module ckan.forms.registry:32 in get_fieldset
               raise ValueError('Could not find package_form name %r in those found: \n%r' % (package_form, [en.name for en in entrypoints]))
ValueError: Could not find package_form name u'gov)' in those found: ['gov', 'standard', 'ca']
}}}"	1301302303000000	1301303315000000
1059	defect	dread	dread	ckan-v1.4-sprint-5	closed	fixed	Loader coping better with poor search indexing	Loader currently checks for same name, but also should check for name_, name__ etc.	1301310596000000	1301312516000000
1061	enhancement	dread	dread	ckan-v1.4-sprint-5	closed	invalid	Orphaned home/license page	No links to home/license and it contains out of date references to knowledgeforge. Remove it.	1301392968000000	1301922350000000
1066	enhancement	dread	dread	ckan-v1.4-sprint-5	closed	fixed	Default reader role too permissive	"The definition of the 'reader' role includes creating packages, which is too permissive for some CKAN instances (e.g. DGU). 'Reader' suggests only reading, so I think this role should avoid creating and editing.

All projects so far want all roles to be able to create users, so this stays as a Reader action for now, as a convenience.

Implementation:

 * Action.PACKAGE_CREATE removed from reader's default_role_actions
 * Visitor has a new default role, called 'anon_editor' which can edit packages, but not groups / auth groups - you have to log in for that.
 * Migration script not needed?
 * Code comments written, to make clear the suggested policy"	1301645250000000	1301932136000000
1067	enhancement	dread	dread	ckan-v1.4-sprint-5	closed	fixed	CLI for loading/dumping complete databases	Use 'db dump' and 'db load' for 'pg_dump' and 'psql -f' of a database. Use pylons config to find out database options.	1301645463000000	1302186503000000
1068	defect	dread	dread	ckan-v1.4-sprint-5	closed	fixed	metadata_modified problem	"This test has been failing since the clocks changed:
{{{
======================================================================
FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified
----------------------------------------------------------------------
Traceback (most recent call last):
  File ""/home/dread/hgroot/pyenv-ckan2/lib/python2.6/site-packages/nose-0.11.3-py2.6.egg/nose/case.py"", line 186, in runTest
    self.test(*self.arg)
  File ""/home/dread/hgroot/ckan2/ckan/tests/models/test_package.py"", line 283, in test_02_metadata_created_and_modified
    assert out == exp, (out, exp)
AssertionError: (datetime.datetime(2011, 4, 1, 10, 45, 50, 875509), datetime.datetime(2011, 4, 1, 9, 45, 50, 875509))

----------------------------------------------------------------------
}}}"	1301652085000000	1302109505000000
1071	defect	dread	dread	ckan-v1.4-sprint-5	closed	fixed	Package history API moved to /api/rest/package/revisions	"api/rest/package_history is not RESTful or follow API naming conventions. Therefore move it to /api/rest/package/revisions

Also, API docs incomplete."	1301937882000000	1301943180000000
1072	enhancement	dread	dread	ckan-v1.4-sprint-5	closed	fixed	Add filters to authztool	It takes several minutes to print the 'rights' on DGU, which is annoying when you only want to grep for a few lines. Much quicker than grepping is to filter in the query.	1302106311000000	1302106474000000
1073	enhancement	dread	dread	ckan-v1.4-sprint-5	closed	fixed	Search index checker	"Tool that checks which packages have not been indexed.

Required for DGU: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/940"	1302185444000000	1302185825000000
1085	defect	dread	johnlawrenceaspden		closed	fixed	local development copy of ckan depends on existence of ckan.net	"ckan.net appears to have either gone down or be running ultra slowly.

this means that ckan copies running locally on my machine run very slowly indeed.

is this behaviour desirable?

This command finds lots of http://~~~ckan.net references in python, html and javascript files:

find ~/pyenv/src \( -name ""*.py"" -or -name ""*.html"" -or -name ""*.js"" \) -print0 | xargs -0 -e grep --color -nH -e ""http://.*ckan.net""

output for reference:
-------------------------------------------------------------------

/home/okfn/pyenv/src/ckan/ckan/__init__.py:5:Network (CKAN) site: http://www.ckan.net.
/home/okfn/pyenv/src/ckan/ckan/lib/create_test_data.py:346:<http://ckan.net/>
/home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:3:DOMAIN = 'http://ckan.net'
/home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:4:CKAN_NAMESPACE = 'http://ckan.net/#'
/home/okfn/pyenv/src/ckan/ckan/lib/talis.py:60:                      'ckan':'http://ckan.net/ns#',
/home/okfn/pyenv/src/ckan/ckan/public/scripts/bookmarklet.js:2:    f='http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&title='+encodeURIComponent(document.title);
/home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:16:    addtockan.src='http://ckan.net/scripts/bookmarklet.js';
/home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:27:    <p><strong>Proper bookmarklet (compressed -- need to escape &amp;):</strong> <a href=""javascript:(function(){f='http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;title='+encodeURIComponent(document.title);if((n=document.getElementsByName('description')[0])&amp;&amp;(d=n.content)){f+='&amp;notes='+encodeURIComponent(d);}a=function(){if(!window.open(f)){location.href=f;}};if(/Firefox/.test(navigator.userAgent)){setTimeout(a,0)}else{a()}})()"">Add to CKAN</a>
/home/okfn/pyenv/src/ckan/ckan/templates/home/license.html:31:      For convenience, all material - including all package, tag and revision information - is available in bulk, in the form of a full dump of the CKAN database. This (gzipped) dump file is updated daily and can be downloaded from <a href=""http://www.ckan.net/dump/"">http://www.ckan.net/dump/</a>.
/home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:71:            'notes': u'Some test notes\n\n### A 3rd level heading\n\n**Some bolded text.**\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n',
/home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:137:             'notes': u'Some test notes\n\n### A 3rd level heading\n\n**Some bolded text.**\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n',
/home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:447:                                  'notes': u'Some test notes\n\n### A 3rd level heading\n\n**Some bolded text.**\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n',
/home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:458:                                  'notes': u'Some test notes\n\n### A 3rd level heading\n\n**Some bolded text.**\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n',
/home/okfn/pyenv/src/ckan/ckan/tests/functional/api/base.py:178:        assert '""ckan_url"": ""http://test.ckan.net/package/annakarenina""' in msg, msg
/home/okfn/pyenv/src/ckanclient/ckanclient/__init__.py:116:    api e.g. http://ckan.net/api rather than http://ckan.net/api/rest)
/home/okfn/pyenv/src/ckanclient/ckanclient/__init__.py:261:    :param base_location: default *http://www.ckan.net/api*
/home/okfn/pyenv/src/ckanclient/ckanclient/__init__.py:267:    base_location = 'http://www.ckan.net/api'

--------------------------------------------------------------------"	1302620434000000	1302625314000000
1089	enhancement	dread	dread	ckan-v1.4-sprint-6	closed	fixed	"Check for ""--ckan"" when running nosetests"	(because if you forget, you get difficult to understand errors, and more than one person has tripped up on this)	1302631189000000	1302631733000000
1090	defect	dread	dread	ckan-v1.4-sprint-6	closed	fixed	Visitor can't create packages on new CKAN install	"Default visitor roles in default config is reader, not anon_editor. 

Problem caused by changes in #1066 (released in 1.3.3)

New installs will be affected, although simple to just increase permissions when the installer realises a visitor can't create packages.

The solution to the config getting out of sync with the code like this is to not have the default_roles in the config - refer to the code in the configuration instructions."	1302635219000000	1302635699000000
1093	defect	dread	dread	ckan-v1.4-sprint-6	closed	fixed	500 errors on GET to api/rest/licenses	CKAN gets its license list from a license service, which can be a local file, but is often the http://licenses.opendefinition.org/2.0/ckan_original server. This server is currently flakey, but I think we only request the list on start up. The problem is we query it much more often than required. It is queried for every request to api/rest/licenses, and we are returning lots of 500 errors when the license server is timing out.	1302862261000000	1302865470000000
1097	enhancement	dread	dread	ckan-v1.4-sprint-6	closed	fixed	Sidebar hideable	The web interface has a sidebar (#primary) which should be hidden in some pages. This is for QA extension and useful for package new and edit pages. Must be compatible with DGU theme.	1303293416000000	1303293476000000
1098	task	dread	dread	ckan-v1.4-sprint-6	closed	fixed	--ckan-migration tests not initialised correctly	Only tests with failing --ckan-migration fail, due to authz not being initialised.	1303377336000000	1303406017000000
1104	defect	dread	johnlawrenceaspden	ckan-v1.4-sprint-7	closed	fixed	create-test-data doesn't index the packages it creates	"With the default test data created by

paster db clean
paster db init
paster create-test-data

going to the front page shows two recently changed packages
A Wonderful Story
A Novel by Tolstoy

But none of those words ""Wonderful"", etc produce search hits.
In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.
"	1303494635000000	1303920791000000
1112	enhancement	dread	dread	ckan-v1.4-sprint-7	closed	fixed	Allow searching for all packages	"{{{
GET api/search/package?q=
}}}
returns all packages. This is so you can filter them e.g. by openness, which is not currently possible."	1304007852000000	1304358643000000
1114	enhancement	dread	dread	ckan-v1.4-sprint-7	closed	fixed	CLI for viewing search index of a package	To see what lexemes are generated for debug purposes.	1304078353000000	1304085484000000
1116	defect	dread	dread	ckan-v1.4-sprint-7	closed	fixed	api search loses boolean q options	filter_by_openness and filter_by_downloadable options don't work when specified as URI parameters. (They do work in qjson parameters)	1304086823000000	1304358664000000
1123	requirement	dread	nils.toedtmann		closed	fixed	"Please re-package CKAN packages as ""noarch"""	"... or, if the CKAN packages do contain architecture-specific binary code, build packages for i386 too. 

Currently, http://apt-alpha.ckan.org/debian only offers packages for amd64, but e.g. ""m1.small"" EC2 instances are i386.

We would need this in order to migrate the community instances to a packaged based CKAN.

Rufus, pls prioritise."	1304530050000000	1311863806000000
1125	enhancement	dread	nils.toedtmann		closed	fixed	"Debian package ""ckan"" should not depend on ""postgresql"""	"The debian package ""ckan"" with the two scripts ""ckan-create-instance"" and ""ckan-instance-maintenance"" depends against ""postgresql"". But ""ckan-create-instance"" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs.

Please add a flag ""--without-local-db"" to ""ckan-create-instance"" and remove the postgres dependancy from the debain package."	1304538095000000	1310134813000000
1126	enhancement	dread	dread	ckan-v1.4-sprint-7	closed	fixed	Exceptions arising from error page	"I'm not completely clear what the use case is for loading the error page in this way, but somehow original_request is None and that creates an unnecessary exception with the logic refactor.

{{{
http://ckan.net/error/document?__cache=39020485
...
Module ckan.controllers.error:29 in document
<<          original_response = request.environ.get('pylons.original_response')
               # Bypass error template for API operations.
               if original_request.path.startswith('/api'):
                   return original_response.body
               # Otherwise, decorate original response with error template.
>>  if original_request.path.startswith('/api'):
AttributeError: 'NoneType' object has no attribute 'path'
}}}"	1304586683000000	1304587078000000
1128	task	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Upload Scotland gov data	"Upload to ckan.net:
https://sites.google.com/site/scotlandsdata/"	1304712293000000	1310568028000000
1131	defect	dread	dread	ckan-v1.4-sprint-7	closed	fixed	Search param validation exception not caught	"Example request:
{{{
http://nl.ckan.net/api/2/search/package?q=delft&order_by=&offset=&limit=&tags=
}}}
Gives 500 error:
{{{
<type 'exceptions.ValueError'>: invalid literal for int() with base 10: ''
}}}"	1304942023000000	1305537897000000
1143	enhancement	dread	dread	ckan-v1.5-sprint-2	closed	fixed	Improve stats page	" * Ensure we don't include deleted packages in the stats
 * Some visual improvements:
   * Number of packages:
      * fix x axis to start at first revision
      * fix y axis to start at zero
   * Fix problem with legend to 'Revisions to packages' graph.
   * Hide sidebar in the template, so it isn't right-cropped in the DGU theme
 * Add some testing of the stats lib - results of basic stats"	1305734836000000	1307024681000000
1169	enhancement	dread	dread	ckan-v1.5-sprint-2	closed	fixed	Link to csv/json dumps on ckan.net		1306749768000000	1306754833000000
1170	defect	dread	dread	ckan-v1.5-sprint-2	closed	fixed	Production deployment docs out of date	"Need to cover pip-requirements.txt on different branches in the doc/deployment.rst. 

Also could do with tidying up upgrade.txt into this document too."	1306864171000000	1306864422000000
1172	defect	dread	thejimmyg	ckan-sprint-2011-10-28	closed	fixed	Remove all try: except: blocks that don't re-raise the original exception	The current codebase has one or two try: except; blocks that don't catch specific exceptions. Under no circumstances should any broad try: except: blocks be allowed unless the exceptions they catch are immediately re-raised. Uncaught exceptions are wasting us quite a lot of time when trying to track down problems.	1306926049000000	1314118883000000
1175	defect	dread	fccoelho@…		closed	invalid	Stats extension not working	"Hi,
I get a 500, Internal server error when I enable ckanext-stats.
Flavio"	1307350823000000	1325355170000000
1180	defect	dread	dread	ckan-v1.5-sprint-3	closed	fixed	Links in markdown can be badly formed	"User can insert bad anchor tags into the User-About and Package-Notes fields and when you view them (web interface) it causes a 500 error.

Need to improve filtering for anchors in markdown.

{{{
<a href=""http://xxxsex.com>nasty/website
}}}
Also check this related exception:
{{{
Module ckan.controllers.user:59 in read
<<          c.is_myself = user.name == c.user
               c.api_key = user.apikey
               c.about_formatted = self._format_about(user.about)
               revisions_q = model.Session.query(model.Revision
                       ).filter_by(author=user.name)
>>  c.about_formatted = self._format_about(user.about)
Module ckan.controllers.user:167 in _format_about
<<      def _format_about(self, about):
               about_formatted = ckan.misc.MarkdownFormat().to_html(about)
               return genshi.HTML(about_formatted) 
       
           def _get_form_password(self):
>>  return genshi.HTML(about_formatted)
WebApp Error: <class 'genshi.input.ParseError'>: junk characters in start tag: u'\u201dhttp://www.settingu': line 1, column 3
}}}"	1307469037000000	1307544223000000
1181	defect	dread	dread		closed	fixed	Link spam vulnerability in Notes and User-About fields	"When viewing a user and a package, the about/notes fields contain Markdown, which may have links. These should have rel=""nofollow"" to discourage link spam."	1307523160000000	1307532816000000
1187	enhancement	dread	dread	ckan-v1.5-sprint-3	closed	fixed	Stop spam users with links in the 'about' field	When you edit a user, you should be stopped from putting links in the 'about' field, because this is a common tactic by spammers.	1308142999000000	1308143605000000
1191	defect	dread	dread	ckan-v1.5-sprint-3	closed	fixed	Unicode in user password gives 500 error on user registration	Create/register a user with a unicode character in the password. It creates the user, but the redirect to the user page doesn't work and results in 500 error.	1308312895000000	1308650930000000
1196	defect	dread	dread	ckan-v1.5-sprint-3	closed	fixed	Markdown logging ignores config settings	Markdown produces Debug logging, even when the pylons logging config is set to level of Info or Warning. This clogs up the logfile.	1308743476000000	1308743857000000
1207	enhancement	dread	dread		closed	fixed	ckanclient.package_entity_get should raise more specific exception	When package does not exist in ckan catalogue, ckanclient.package_entity_get should raise more specific exception, such as CkanNotFoundError instead of generic CkanApiError.	1309515582000000	1311325343000000
1210	defect	dread	dread	ckan-v1.5-sprint-4	closed	fixed	POST application/json error handling with newer WebOb	"WebOb from v1.0.7 has some interesting new behaviour with reading request data for different Content-Types:

 * It seems that looking at request.body mangles request.POST. See fix in ckan cset:e83bad9caa1b (problem with WebOb v1.0.7 but may go back to v1.0.4?)

 * A particular request caused an AssertionError during looking at request.body, so we need to enclose this in a try/except. I couldn't recreate it in a test unfortunately (problem with WebOb v1.0.7 but may go back to v1.0.2?)

Example:
{{{
Module ckan.controllers.api:206 in create
<<          log.debug('create: %s' % (context))
               try:
                   request_data = self._get_request_data()
               except ValueError, inst:
                   response.status_int = 400
>>  request_data = self._get_request_data()
Module ckan.lib.base:149 in _get_request_data
<<          cls.log.debug('Retrieving request params: %r' % request.params)
               cls.log.debug('Retrieving request POST: %r' % request.POST)
               cls.log.debug('Retrieving request POST body: %r' % request.body)
               if request.POST:
                   try:
>>  cls.log.debug('Retrieving request POST body: %r' % request.body)
Module paste.registry:137 in __getattr__
<<      
           def __getattr__(self, attr):
               return getattr(self._current_obj(), attr)
           
           def __setattr__(self, attr, value):
>>  return getattr(self._current_obj(), attr)
Module webob.request:470 in _body__get
<<          Return the content of the request body.
               """"""
               self.make_body_seekable() # we need this to have content_length
               r = self.body_file.read(self.content_length)
               self.body_file.seek(0)
>>  self.make_body_seekable() # we need this to have content_length
Module webob.request:697 in make_body_seekable
<<              self.body_file_raw.seek(0)
               else:
                   self.copy_body()
>>  self.copy_body()
Module webob.request:714 in copy_body
<<                  self.body = self.body_file_raw.read(length)
               elif self.is_body_readable:
                   self.body = self.body_file_raw.read()
                   self._copy_body_tempfile()
               else:
>>  self.body = self.body_file_raw.read()
Module webob.request:1190 in read
<<      def read(self, size=-1):
               body = self._get_body()
               if size < 0:
                   v = body[self.position:]
>>  body = self._get_body()
Module webob.request:1207 in _get_body
<<                  self._body = _encode_multipart(self.vars, self.content_type)
                   else:
                       assert 0, ('Bad content type: %r' % self.content_type)
               return self._body
>>  assert 0, ('Bad content type: %r' % self.content_type)
AssertionError: Bad content type: '; charset=utf-8'
}}}"	1309949451000000	1309974781000000
1214	defect	dread	dread		closed	worksforme	API improvements (following javascript use)	"Notes on the CKAN API from Aron:
 * Server returns text/html for errors even when ""Accept: application/json"" header is set.
 * DELETE package request requires a Content-Length? Should be 405 Method Not Allowed?
 * Tag returned as a JSON object when updating but as a string when requesting. Same with ""extras"" content.
 * How to remove key from ""extras"" object? Passing None as described in the docs is invalid JSON and using null doesn't work.
 * Tags return package names rather than ids when querying GET /tag/{id}
 * Doesn't return rendered_text property on package update.
 * Relationships use ""object"" key rather than id or package_id."	1309975222000000	1314029628000000
1217	defect	dread	dread	ckan-sprint-2011-11-07	closed	invalid	Diff exception	"Exception raised doing this particular diff.

http://ckan.net/revision/diff/sudocfr?diff_entity=package&oldid=a18b2520-09e0-44be-80dd-4fea94ffe7d1&diff=3934ff16-a3a7-4481-b390-72944903b945

{{{
...
Module ckan.controllers.revision:142 in diff
<<          if c.diff_entity == 'package':
                   c.pkg = model.Package.by_name(id)
                   diff = c.pkg.diff(c.revision_to, c.revision_from)
               elif c.diff_entity == 'group':
                   c.group = model.Group.by_name(id)
>>  diff = c.pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:449 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
}}}

Seems to be to do with this particular extra key:
{{{
[<PackageExtraRevision id=ba1f3b3e-f61f-4924-bae0-ec3dd2163ae4 package_id=febe4da9-5874-4411-a7da-b497fdebf739 key=namespace value=http://www.sudoc.fr/ state=active revision_id=5ec8e730-c73d-4f1c-989b-2e015eb3d315 continuity_id=ba1f3b3e-f61f-4924-bae0-ec3dd2163ae4 expired_id=None revision_timestamp=2011-07-08 13:48:11.027260 expired_timestamp=9999-12-31 00:00:00 current=True>]
}}}

Is it because it was created between the from_revision and to_revision?"	1310375498000000	1320663277000000
1218	enhancement	dread	minspamboks@…	ckan-sprint-2011-10-28	closed	fixed	Colour the History tab icon	"Change the color of the ""History"" tab icon to yellowish, like the rest of the icons in the other tabs (""View"" and ""Edit"", ""Authorization"").

== Reasoning ==

When you view a data package, for instance
http://ckan.net/package/thesaurus-w, you will see ""View"", ""Edit"",
""History"" tabs on the top. ""History"" tab has a black-and-white icon
which makes it look like an inactive/disabled tab (since the text is
also grayed out when the tab is not selected). This is not a major
issue, but it is a little bit confusing for the users. This icon
exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net).

The simple solution would be to change the color of the ""History"" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. ""View"" and ""Edit"")."	1310375768000000	1310389390000000
1221	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Password reset tidy up	" * No tests.
 * 500 error given when user is not found."	1310475661000000	1310556607000000
1222	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Mailer has no tests		1310479136000000	1310556618000000
1225	enhancement	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Tool for mass tag changes	Using ckanapi	1310642161000000	1310642407000000
1228	enhancement	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Display OpenID login errors	When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again.	1311086562000000	1311325161000000
1229	enhancement	dread	thejimmyg	ckan-sprint-2011-10-24	closed	fixed	Refactor all database code out of the controllers and into the logic layer	"Progress can be tracked in this document:

https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdFQzRnVoUHV6ckoyblI5TUdEcnJLVlE&hl=en_US#gid=0"	1311093924000000	1319645778000000
1234	enhancement	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Setting locale error conditions handled badly	Needs tests for error conditions for bad locale and redirect parameters.	1311179871000000	1311183328000000
1236	enhancement	dread	dread	ckan-sprint-2011-11-07	closed	fixed	View a package at a given revision	"As a user I want to view a package at a given revision:

 * When I visit /package/osm@yyyy-mm-dd or /package/osm@rrrrrrr I should be shown package at date yyyy-mm-dd or revision rrrrrrr.
 * viewing this page, it should say it is the old revision, just like wikipedia
 * package history page should provide links to these pages
"	1311269333000000	1330019788000000
1248	defect	dread	dread	ckan-sprint-2011-11-07	closed	fixed	License exception	"Exception received on test.ckan.net at 10.12 on 29/7/11
{{{
http://test.ckan.net/package?__cache=2759373&page=4
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:153 in search
<<                          }
       
                   query = get.package_search(context,data_dict)
       
                   c.page = h.Page(
>>  query = get.package_search(context,data_dict)
Module ckan.logic.action.get:514 in package_search
<<      for package in query.results:
               result_dict = table_dictize(package, context)
               result_dict = _extend_package_dict(result_dict,context)
       
               results.append(result_dict)
>>  result_dict = _extend_package_dict(result_dict,context)
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, ""License not found: %s"" % key
       
           def get(self, key, default=None):
>>  raise KeyError, ""License not found: %s"" % key
KeyError: u'License not found: None'
}}}

and a similar one: 
{{{
 http://test.ckan.net/tag/barbecue
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.tag:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__(""logging"").getLogger(""proxy_cache"")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
       
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.tag:64 in read
<<          data_dict = {'id':id}
               try:
                   c.tag = get.tag_show(context,data_dict)
               except NotFound:
                   abort(404, _('Tag not found'))
>>  c.tag = get.tag_show(context,data_dict)
Module ckan.logic.action.get:341 in tag_show
<<      extended_packages = []
           for package in tag_dict['packages']:
               extended_packages.append(_extend_package_dict(package,context))
       
           tag_dict['packages'] = extended_packages
>>  extended_packages.append(_extend_package_dict(package,context))
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, ""License not found: %s"" % key
       
           def get(self, key, default=None):
>>  raise KeyError, ""License not found: %s"" % key
KeyError: u'License not found: CreativeCommonsAttributionCCBY25'
}}}"	1311954136000000	1319797763000000
1254	task	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Feature sheet	Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers.	1312211925000000	1313401629000000
1263	enhancement	dread	dread		closed	fixed	Bad name for logic action in API causes Exception	"A bad action name:
e.g.
curl http://test.ckan.net/api/action/user_showaa -d '{}'
causes 500 response.

Occurs only on 1.4.3 beta."	1312479020000000	1314029279000000
1265	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	user/edit error conditions cause exceptions	"We get exceptions on these occasions:
 * user/edit if not logged in
 * user/edit/bob if bob is not a known user

These are a problem only on 1.4.3b.

Introduced in #1229 - merge of branch feature-1229-db-out-of-controllers"	1312565788000000	1312566052000000
1266	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Timestamps without microseconds causing exception	"I'm not sure why some timestamps are missing microseconds in the ckan.net data, but plenty seem to. This causes this exception when viewing the package rss feed:
{{{
URL: http://ckan.net/package/history/sfk_publishing?format=atom&days=7
In Module ckan.lib.helpers:227 in date_str_to_datetime
WebApp Error: <type 'exceptions.ValueError'>: time data '2008-04-13T20:40:20' does not match format '%Y-%m-%dT%H:%M:%S.%f'
}}}

Only seen with ckan 1.4.3b"	1312794831000000	1312797477000000
1267	defect	dread	dread		closed	fixed	Random exception: cannot set the body to a unicode value	"Occasionally see this problem when producing an error page. Here's an example
{{{
WebApp Error: <type 'exceptions.TypeError'>: You cannot set the body to a unicode value without a charset
URL: http://at.ckan.net/authorizationgroup/new
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:117 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:284 in __call__
<<              if log_debug:
                       log.debug(""Calling Response object to return WSGI data"")
                   return response(environ, self.start_response)
               
               if log_debug:
>>  return response(environ, self.start_response)
Module webob.exc:248 in __call__
<<              return []
               if not self.body and not self.empty_body:
                   return self.generate_response(environ, start_response)
               return Response.__call__(self, environ, start_response)
>>  return self.generate_response(environ, start_response)
Module webob.exc:239 in generate_response
<<              status=self.status,
                   headerlist=headerlist,
                   content_type=content_type
               )
               return resp(environ, start_response)
>>  content_type=content_type
Module webob.response:94 in __init__
<<                  if charset is None:
                           raise TypeError(
                               ""You cannot set the body to a unicode value without a charset"")
                       body = body.encode(charset)
                   self._body = body
>>  ""You cannot set the body to a unicode value without a charset"")
TypeError: You cannot set the body to a unicode value without a charset
}}}"	1312889084000000	1312897724000000
1270	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Default site www.ckan.net is a redirect and loses POSTs	"The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters:
{{{
In [1]: import ckanclient

In [2]: c = ckanclient.CkanClient()

In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api')

In [4]: c.package_search('', search_options={'groups':'openspending'})
Out[4]: 
{u'count': 2102,
 u'results': <generator object _result_generator at 0x10168b1e0>}

In [5]: c2.package_search('', search_options={'groups':'openspending'})
Out[5]: {u'count': 29, u'results': <generator object _result_generator at 0x10168b410>}
}}}

Thanks to borior for finding this and raising it."	1312911847000000	1312912188000000
1283	enhancement	dread	dread	ckan-backlog	closed	fixed	Deleted packages shouldn't be searchable or browsable	"A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else. 

(But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.)

Same in the Web interface and API.

This ticket ties in with #948 highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can #1076."	1314112481000000	1330083933000000
1284	defect	dread	rgrp		closed	fixed	Lots of revisions being created on datahub.org	Revisions aren't assigned to packages. Where are they coming from? This seems to have been occurring since the upgrade to 1.4.3b a couple of weeks ago?	1314113015000000	1315215626000000
1290	enhancement	dread	dread	ckan-backlog	closed	fixed	Better error when blank database	"When installing CKAN, when doing ""paster serve development.ini"", lots of users encounter the error for every request: 
{{{
ProgrammingError: (ProgrammingError) relation ""user"" does not exist
}}}
This is because the database tables have not been created - they have forgotten or missed the ""paster db init"" step.

Can we provide a better error to say that the database is not initialised yet?

== Implementation options ==

1. At the start of every request we reflect the database tables and check they are there. This is rather expensive!

2. Uncached requests to the home page start with a cheap database query. If there is an exception then return this error about database setup. I really like Drupal's page for this that has in large letters that the site is currently off-line. Below a line, in small letters, there are developer suggestions on what is wrong and where to look to fix it."	1314264255000000	1314270894000000
1292	enhancement	dread	dread	ckan-sprint-2011-11-07	closed	fixed	Internationalization improvements	" * Review process of releases and updating strings (pot) and translations (po) files
   * c.f. http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html
 * Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?)
 * Review long list of strings for improvement
   * Consolidate any?
   * Get rid of i18n for strings in api
   * Try and remove javascript
   * Add in extensions
 * #1374 Fix switching to en from other default language
 * #1417 Fix browser language detection"	1314303314000000	1320145732000000
1310	defect	dread	dread	ckan-sprint-2011-09-12	closed	fixed	API GET of a revision that has been approved causes exception	"e.g. this particular revision on ckan.net: http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c
(most others have not been approved)
Causes 500 error."	1315234899000000	1315820419000000
1312	enhancement	dread	florian.marienfeld@…	ckan-sprint-2011-10-28	closed	invalid	Particular characters in JSON cause exception creating package on API	"From Florian:
{{{
I am having trouble with the characters ""=&;"".
if they occur e.g. in the title of a package, I get a 400-Bad-Request
when I try to register a package by POSTing to ckan/rest/api/package

However, when I PUT the same package as an edit to
ckan/rest/api/package/existing_package, it works and the title shows
correctly on the front end and API.
}}}

== Reproduced ==
{{{
(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test3"", ""title"": ""Test &""}' -H ""Authorization:tester"" 
HTTP/1.0 400 Bad Request
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 14:09:31 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Length: 87

""Bad request - JSON Error: Unterminated string starting at: line 1 column 27 (char 27)""
}}}"	1315318863000000	1315327743000000
1315	enhancement	dread	dread	ckan-sprint-2011-11-07	closed	fixed	Document /api/action	We need some documentation for the /api/action. Even though it's beta and subject to change it would be good to have a guide.	1315396722000000	1320173768000000
1318	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	'Register user' doesn't log you in	"When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to ""log in"" and ""register"". Also the ""create package"" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected."	1315415783000000	1315569362000000
1321	defect	dread	dread	ckan-sprint-2011-10-24	closed	fixed	Flash message deferred	"When you reset a password, the flash message saying ""check your email"" sometimes doesn't display on the next page. You then click on the link in your email and it displays on that page (i.e. too late).

Seen this on thedatahub.org. To do with the proxy caching?"	1315474657000000	1319814460000000
1323	enhancement	dread	dread	ckan-sprint-2011-10-28	closed	fixed	GET /authorizationgroup causes Exception when logged in	on thedatahub.org	1315475685000000	1315594333000000
1325	defect	dread	dread	ckan-sprint-2011-09-12	closed	fixed	python 2.5 incompatibilities	"We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in:
 * 'with' statement requires {{{from __future__ import with_statement}}}
 * import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5."	1315580452000000	1315587113000000
1329	defect	dread	amercader	ckan-sprint-2011-10-28	closed	fixed	i18n poor in some templates - missing i18n tags	"Just a tip for developers regarding translations:

* When writing templates, mark <p>, <div>, etc elements with the
i18:msg attribute, otherwise the messages don't get extracted
properly.

Good: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40
{{{
<p i18n:msg="""">More about CKAN, and the background to its creation can
   be found in <a
     href=""http://m.okfn.org/files/talks/opentech_20080705/"">these
     slides</a>.</p>
}}}
gives this:
https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

{{{
<p class=""hints"">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href=""http://opendatacommons.org/licenses/odbl/1.0/"">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>
}}}
gives this:
https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the
grammar or syntax of the translated language."	1315824288000000	1317404566000000
1331	defect	dread	dread		closed	worksforme	Setting a tag twice causes exception	"To reproduce:
 1. Create a package with two tags the same: ""bulk bulk""
 2. Click 'save'
 3. 500 ERROR - 'Server Error'"	1315905959000000	1325355631000000
1340	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Editing non-existent group causes Exception	"e.g. http://ckan.net/group/edit/history
{{{
Module ckan.controllers.group:135 in edit
<<          try:
                   old_data = get_action('group_show')(context, data_dict)
                   c.grouptitle = old_data.get('title')
                   c.groupname = old_data.get('name')
>>  old_data = get_action('group_show')(context, data_dict)
Module ckan.logic.action.get:347 in group_show
<<      if group is None:
               raise NotFound
       
           check_access('group_show',context, data_dict)
>>  raise NotFound
}}}"	1315991374000000	1315991534000000
1346	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	'get_authorization_group_dict not found' exception	"This stems from a feature to allow you to prefill fields in the authorizationgroup creation form. 

e.g. http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it."	1316081974000000	1316094632000000
1347	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Exception caused by New Group form POST with bad formed parameter	"Exception caused by bots POSTing 'new group' form, but with bad (old-style) parameters.
{{{
<type 'exceptions.ValueError'>: invalid literal for int() with base 10: 'cache'
...
Module ckan.controllers.group:102 in new
<<          if context['save'] and not data:
                   return self._save_new(context)
               
               data = data or {}
>>  return self._save_new(context)
Module ckan.controllers.group:152 in _save_new
<<          try:
                   data_dict = clean_dict(unflatten(
                       tuplize_dict(parse_params(request.params))))
                   context['message'] = data_dict.get('log_message', '')
                   group = create.group_create(data_dict, context)
>>  tuplize_dict(parse_params(request.params))))
Module ckan.logic:57 in tuplize_dict
<<          for num, key in enumerate(key_list):
                   if num % 2 == 1:
                       key_list[num] = int(key)
               tuplized_dict[tuple(key_list)] = value
           return tuplized_dict
>>  key_list[num] = int(key)
ValueError: invalid literal for int() with base 10: 'cache'
}}}"	1316094466000000	1316094608000000
1365	defect	dread	dread	ckan-sprint-2011-11-07	closed	fixed	Support old search formats	"Lots of things rely on the old (i.e. pre-SOLR) search param formats, including ckanclient / DGU. Need a translation layer to convert these params (in api v1&2) to SOLR syntax.

This functionality was broken by the changes in #1277 which went into 1.5a"	1317233338000000	1319815052000000
1367	defect	dread	dread		closed	fixed	PyPI ckan broken	"manifest.cfg was not included in the pypi package, causing InvalidRepositoryError when you do 'paster db init'. 

Appears to affect most CKAN PyPI releases up to and including 1.4.3."	1317379931000000	1317394273000000
1374	defect	dread	dread	ckan-sprint-2011-10-24	closed	fixed	Can't switch to English if default is non-English	e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English.	1317893975000000	1319648746000000
1385	enhancement	dread	dread		closed	fixed	Resolve postgres permissions issues	"Currently there is a problem because the docs guide us to set the sqlalchemy url to use 'localhost' i.e. loopback, whereas paster commands don't specify '-h localhost' so use unix sockets (you need to do 'sudo -u postgres'). These should be the same.

Also do we need to tell people to add a line to their postgres authentication config /etc/postgresql/8.4/main/pg_hba.conf
to help things? Florian suggests:
{{{
local   std         std                          md5
}}}
/etc/postgresql/8.4/main/pg_hba.conf
"	1318418537000000	1326218703000000
1394	defect	dread	dread	ckan-sprint-2012-01-09	closed	fixed	Resource validation error messages misleading	"(Editing a dataset) If the second resource contains any validation error then it says ""Resources: Package resource(s) incomplete"" and ""Resource 1:""."	1318515262000000	1325604784000000
1405	defect	dread	jonathan.gray@…	ckan-sprint-2011-11-07	closed	fixed	You can add a package twice to a group	"To reproduce (from Jonathan Gray):
 * I go to add an item to the group
 * As I type, I get autocomplete options (cool!)
 * When I click on one, it adds this, and copies the same autocomplete option in the box
 * When I click submit, it seems to add the item twice
 * On the main group page I think it just shows up once, but when I click edit it shows two identical boxes, both ticked - and the count includes two copies of the same item

Basically the problem is allowing the same package associated multiple times with a group.

This is an issue on CKAN 1.5b and probably earlier."	1318856354000000	1320152291000000
1416	defect	dread	dread	ckan-sprint-2011-11-07	closed	fixed	Registering with blank email can cause exception	"This might well be a bot, but not POSTing the email field should not cause an exception - it should be 400 and then the sysadmin doesn't get these nags.
{{{
http://www.publicdata.eu/user/register
...
Module ckan.controllers.user:79 in register
<<              c.login = request.params.getone('login')
                   c.fullname = request.params.getone('fullname')
                   c.email = request.params.getone('email')
                   if not model.User.check_name_available(c.login):
                       h.flash_error(_(""That username is not available.""))
>>  c.email = request.params.getone('email')
Module webob.multidict:335 in getone
<<          values were found.
               """"""
               return self._decode_value(self.multi.getone(self._encode_key(key)))
       
           def mixed(self):
>>  return self._decode_value(self.multi.getone(self._encode_key(key)))
Module webob.multidict:99 in getone
<<          v = self.getall(key)
               if not v:
                   raise KeyError('Key not found: %r' % key)
               if len(v) > 1:
                   raise KeyError('Multiple values match %r: %r' % (key, v))
>>  raise KeyError('Key not found: %r' % key)
KeyError: ""Key not found: 'email'""
}}}
{{{
HTTP_USER_AGENT	'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MRA 4.3 (build 01218); .NET CLR 1.1.4322)'
webob._parsed_post_vars	(MultiDict([('login', 'ahusernames9'), ('fullname', 'ahusernames9'), ('password1', '******'), ('password2', '******'), ('s', 'Sign up')]), <FakeCGIBody at 0xb9d7c5ec viewing MultiDict([('lo...p')])>)"	1319533989000000	1320153507000000
1417	defect	dread	dread	ckan-sprint-2011-10-24	closed	fixed	Browser language detection doesn't work	"In Firefox:
 * Edit | Preferences | Content | Languages - put 'de' at top language
 * Clear cookies
 * Load thedatahub.org
 - it should be in German, but it's in the default language - English."	1319539010000000	1319651617000000
1418	defect	dread	dread	ckan-sprint-2011-10-28	closed	duplicate	If default language is not 'en' then you can't switch language to 'en'	"e.g. cz.ckan.net has lang=cz but when you click on Home page link ""English"" it flashes message ""Language has been set to: English"" but the rest of the page is still Czech."	1319539146000000	1319539691000000
1421	defect	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Tag cloud recipe broken	"home page doesn't have c.facets setup.

Only affects 1.5 branch - ok 1.4.3.1. Was broken in cset:05f60f2d26e8."	1319638575000000	1319639311000000
1422	enhancement	dread	dread		closed	fixed	Remove proxy & etag caching	"Rufus decided this was more trouble than it was worth.

Removes ckan/lib/cache.py"	1319639663000000	1319645722000000
1426	defect	dread	dread	ckan-sprint-2011-10-28	closed	invalid	Validation errors cause 409 not 400 response	"Validation error is incorrect arguments, so should be a ""400 Bad Request"" response. ""409 Conflict"" is for when there is a conflict creating two resources with the same identifier. Yet we get 409 for all validation errors."	1319708619000000	1319709236000000
1427	enhancement	dread	dread	ckan-sprint-2011-10-28	closed	fixed	Tags validation needs to limit length to 100 characters	The tag name is limited in length in the db table to 100 characters. If you try and create a longer one you get an exception. This should cause a ValidationError and tell the user this sensibly.	1319708698000000	1319709925000000
1428	defect	dread	dread	ckan-sprint-2011-11-07	closed	fixed	config options not used or documented	Need to ensure config options in the deployment.ini_tmpl match up with doc/configuration.rst. Remove any unused ones too.	1319803594000000	1319804773000000
1431	defect	dread	dread	ckan-v1.5	closed	fixed	Captcha field - foreign chars cause exception	"During registering a user, the user inputs foreign chars into the captcha field.
{{{
URL: http://thedatahub.org/user/register
...
Module ckan.lib.captcha:22 in check_recaptcha
<<                                     remoteip=client_ip_address,
                                          challenge=recaptcha_challenge_field,
                                          response=recaptcha_response_field))
           f = urllib2.urlopen(recaptcha_server_name, params)
           data = f.read()
>>  response=recaptcha_response_field))
Module urllib:1267 in urlencode
<<          for k, v in query:
                   k = quote_plus(str(k))
                   v = quote_plus(str(v))
                   l.append(k + '=' + v)
           else:
>>  v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 0: ordinal not in range(128)
}}}"	1320078849000000	1320084104000000
1437	defect	dread	dread	ckan-sprint-2011-11-07	closed	fixed	JSONP parameter in Action API	"Action API needs JSONP support - be able to return responses encapsulated in a function of a supplied name. This is important for remote sites running javascript to interact with a CKAN site.

Specifying the callback parameter is the way we've achieved JSONP with the RESTful and Search APIs. It should work like this:
{{{
curl http://test.ckan.net/api/action/package_show?callback=jsoncallback -d '{""id"": ""fd788e57-dce4-481c-832d-497235bf9f78""}'
}}}

Or maybe the callback should be specified in the payload in the context or data_dict?

(My understanding is that CORS is similar - when more browsers support it, can we drop JSONP?)
"	1320161088000000	1320173795000000
1440	defect	dread	dread	ckan-sprint-2011-11-07	closed	fixed	'paster db load' doesn't update search index	"{{{paster db load ckan.pgdump}}} will load in the database but not update the search index, no we're using SOLR.

= implementation =

We can't tack on 'rebuild' to this command, because it needs a 'db upgrade' first. I'm tempted to make 'db load' do the upgrade and rebuild all in one command."	1320232415000000	1320235084000000
1441	defect	dread	dread		closed	fixed	Use local ckan logo image	Using local image rather than the same image at assets.okfn.org reduces dependency on another server that may be down, slow or remote. James agrees.	1320234597000000	1320235062000000
1442	defect	dread	dread	ckan-sprint-2011-11-07	closed	fixed	You can't GET then POST a dataset as new	"If you try to GET a dataset, purge it then POST it again, it fails checking authz for the group:
{{{
Module ckan.controllers.api:278 in create
<<                  (register, subregister))
               try:
                   response_data = action(context, data_dict)
                   location = None
                   if ""id"" in data_dict:
>>  response_data = action(context, data_dict)
Module ckan.logic.action.create:240 in package_create_rest
<<      api = context.get('api_version') or '1'
       
           check_access('package_create_rest', context, data_dict)
       
           dictized_package = package_api_to_dict(data_dict, context)
>>  check_access('package_create_rest', context, data_dict)
Module ckan.logic:129 in check_access
<<          #    log.debug('Valid API key needed to make changes')
               #    raise NotAuthorized
               logic_authorization = new_authz.is_authorized(action, context, data_dict)
               if not logic_authorization['success']:
                   msg = logic_authorization.get('msg','')
>>  logic_authorization = new_authz.is_authorized(action, context, data_dict)
Module ckan.new_authz:16 in is_authorized
<<      auth_function = _get_auth_function(action)
           if auth_function:
               return auth_function(context, data_dict)
           else:
               raise ValueError(_('Authorization function not found: %s' % action))
>>  return auth_function(context, data_dict)
Module ckan.logic.auth.create:115 in package_create_rest
<<          return {'success': False, 'msg': _('Valid API key needed to create a package')}
       
           return package_create(context, data_dict)
       
       def group_create_rest(context, data_dict):
>>  return package_create(context, data_dict)
Module ckan.logic.auth.create:15 in package_create
<<      else:
               
               check2 = check_group_auth(context,data_dict)
               if not check2:
                   return {'success': False, 'msg': _('User %s not authorized to edit these groups') % str(user)}
>>  check2 = check_group_auth(context,data_dict)
Module ckan.logic.auth.create:90 in check_group_auth
<<      groups = set()
           for group_dict in group_dicts:
               id = group_dict.get('id')
               if not id:
                   continue
>>  id = group_dict.get('id')
AttributeError: 'unicode' object has no attribute 'get'
}}}
(using RESTful Model API)

The reason is the POST is expecting the group to be expressed as a dict, not the list of names that GET returns."	1320263318000000	1320275623000000
1443	defect	dread	dread	ckan-sprint-2011-11-07	closed	fixed	language detection	"If language detected doesn't match any of the offered CKAN languages, you get the default language, but see an error flash up for every page, saying:
{{{
Could not change language to 'None': IOError: [Errno 2] No translation file found for domain: 'ckan'
}}}

Affects 1.5b only."	1320425588000000	1320432511000000
1444	defect	dread	dread	ckan-future	closed	wontfix	Language detection defaults to en_us?	"Using curl you always get English, even if ckan.locale_default=de. Find out why. (1.5b)

Maybe we should disable locale detection for this release if lots of people's browsers don't have it set correctly (e.g. chinese with us internet explorer)"	1320432634000000	1340190852000000
1452	enhancement	dread	dread		closed	wontfix	Offer detected browser language, rather than auto-switch	"There are issues with setting the site's language according to the browser detection:

* Many browsers have the language setting incorrectly set. e.g. people who download US versions of Mozilla, pirated US copies in the far east.

* Although thedatahub.org has a worldwide audience, many CKAN instances are designed for a particular country's citizens. So Open Berlin should probably be in German as default for all.

I suggest the site should have a default language saved in the config. The browser language *should* be detected, and that prompts a flash message offering to change to that language. And if you change language that is saved in the cookie (as we currently do). 

Then do some testing to see if this suits people."	1320682652000000	1340190587000000
1454	enhancement	dread	dread	ckan-sprint-2011-11-21	closed	fixed	postgres search code/data removal	"We made SOLR the main and only supported way to do CKAN search three
months ago (#1275), but I notice that we still have remnants of the
postgres full-text search hanging around:
 * model code (ckan/model/search_index.py)
 * table in the database (package_search)
The presence of these still create sqlalchemy warnings and add to our
db, (& db backups etc), so it would be good to get rid of them if we
are not using them. John says this can probably be removed
"	1320838999000000	1320849527000000
1470	defect	dread	amercader	ckan-sprint-2011-11-21	closed	fixed	Check user name in the profile form		1321446143000000	1324473955000000
1473	defect	dread	dread	ckan-sprint-2011-11-21	closed	fixed	max name length validators missing	"Max length of package name and version field is 100 characters, but the validators seem to enforce this, although in the web interface the javascript limits typing in these fields, so it just affects the API really.
"	1321631452000000	1321872717000000
1479	defect	dread	dread	ckan-sprint-2011-12-05	closed	fixed	Can't edit a user with a unicode email address	" 1. Register User with an email address with a unicode char (e.g. u'\u044e')
 2. View the User in the UI (/user/) or with 'user_show' Action API

Exception:
{{{
Module ckan.controllers.user:98 in read
<<          try:
                   user_dict = get_action('user_show')(context,data_dict)
               except NotFound:
                   h.redirect_to(controller='user', action='login', id=None)
>>  user_dict = get_action('user_show')(context,data_dict)
Module ckan.logic.action.get:488 in user_show
<<      check_access('user_show',context, data_dict)
       
           user_dict = user_dictize(user_obj,context)
       
           if not (Authorizer().is_sysadmin(unicode(user)) or user == user_obj.name):
>>  user_dict = user_dictize(user_obj,context)
Module ckan.lib.dictization.model_dictize:189 in user_dictize
<<      
           result_dict['display_name'] = user.display_name
           result_dict['email_hash'] = user.email_hash
           result_dict['number_of_edits'] = user.number_of_edits()
           result_dict['number_administered_packages'] = user.number_administered_packages()
>>  result_dict['email_hash'] = user.email_hash
Module ckan.model.user:59 in email_hash
<<          if self.email:
                   e = self.email.strip().lower()
               return hashlib.md5(e).hexdigest()
               
           def get_reference_preferred_for_uri(self):
>>  return hashlib.md5(e).hexdigest()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u044e' in position 17: ordinal not in range(128)
}}}"	1321960486000000	1321961592000000
1480	enhancement	dread	dread	ckan-sprint-2011-12-05	closed	fixed	Munge functions in util	munge_title_to_name, munge_tag v. useful for client software importing things into CKAN. I want it for ONS.	1321965837000000	1321978546000000
1484	defect	dread	dread	ckan-sprint-2011-12-05	closed	fixed	Creating package with duplicate tags causes exception	"e.g.
{{{
curl http://localhost:5000/api/action/package_create -d '{""name"":""test5"", ""tags"":[{""name"": ""tag6""}, {""name"": ""tag6""}]}' -H 'Authorization:tester'
}}}

Also broken in Web UI (adding a package).

Affects API and Web UI in 1.5 and I think all versions from 1.4.1"	1322052046000000	1322059006000000
1488	enhancement	dread	dread	ckan-sprint-2011-11-21	closed	fixed	List of installed extensions - add to the status_show	Add the list of extensions install (config's ckan.plugins) to status_show logic action (See #1087)	1322130555000000	1322241966000000
1493	defect	dread	dread	ckan-sprint-2011-12-05	closed	fixed	'search-index rebuild/clear' doesn't work if no ckan.site_id	"You can't delete things from the SOLR search index if the ckan.site_id and ckan.site_url are blank.

Should assert that one of these are set up."	1322484422000000	1324474360000000
1497	enhancement	dread	dread	ckan-sprint-2011-12-05	closed	fixed	Suppress benign warnings	We really shouldn't have lots of benign stuff printed on stderr when we run every command. It confuses users and hides important stuff.	1322569640000000	1322569805000000
1499	enhancement	dread	dread	ckan-sprint-2011-12-05	closed	fixed	list/read/delete packages on the command-line	It would be useful to be able to manipulate packages using the paster command-line interface. In particular listing, reading and deleting packages. Seeing which packages were deleted too.	1322591656000000	1322591888000000
1500	defect	dread	dread	ckan-sprint-2011-12-05	closed	fixed	Slug API calls tidy up	" * API call documented but don't exist any more: dataset create_slug
 * API call not documented but exists: is_slug_valid
 * API call not documented, badly sited in API and using custom code in controller (not logic layer): /package/autocomplete"	1322600285000000	1323166918000000
1503	task	dread	markbrough	ckan-sprint-2011-12-05	closed	fixed	Move from Bitbucket to Github	"Reasons: more people familiar with Git, bigger Open Source community on Github.

Keeping track of the move here: https://docs.google.com/spreadsheet/ccc?key=0AvCDqUH8jVN8dGNVNEpXLWk3UTlJZXhFMjBPbXVtRmc&hl=en_GB#gid=0"	1322745084000000	1323793662000000
1504	defect	dread	dread	ckan-sprint-2011-12-05	closed	fixed	Action API given blank data causes exception	"This is what caused the exception:
{{{
curl http://localhost:5000/api/action/package_search -d='{""q"": ""osm""}'
}}}
The problem is the content-type is formencoding, the '=' is the first character of the request, and so internally the data_dict is set to ' ', which needs to be a dictionary."	1322756748000000	1324035928000000
1505	defect	dread	dread	ckan-sprint-2011-12-05	closed	fixed	SearchError and SearchQueryError cause exception in Action API	"This query caused ckan to except because ckan/controllers/api.py doesn't catch SearchError and SearchQueryError:
{{{
curl http://localhost:5000/api/action/package_search -d '{""sort"": ""metadata_modified""}'
}}}"	1322758968000000	1324474577000000
1513	defect	dread	dread	ckan-sprint-2012-02-06	closed	fixed	Logged in/out problems	"Richard Cyganiak: I often see a problem where half the system thinks that I'm logged in, while the other half insists I'm not. Logout then login again may fix it.

David Read: I think I've seen it several times that the ""My Account"" link is given (suggesting you're logged in) but when you click it it asks you to log in, which means you weren't."	1323100353000000	1328205099000000
1529	enhancement	dread	dread	ckan-sprint-2011-12-19	closed	fixed	Display user name when logged in	"Currently when you log in it says ""logged-in"". Most sites show your user-name and this is helpful when you have more than one account or human using the computer."	1323252086000000	1324318628000000
1536	defect	dread	dread	ckan-sprint-2011-12-19	closed	fixed	Group deletion doesn't work	In the Web interface and API, an administrator can set the 'state' field of a group to 'deleted' but this change is ignored in the save/submit and it doesn't get deleted.	1323346766000000	1330020599000000
1548	enhancement	dread	dread	ckan-sprint-2011-12-19	closed	fixed	Simplfy inheritance in functional tests	It's hard to understand what's going on with the functional tests because there is so much going on in inherited classes. Would be better to be explicit in constructors, even if it involves cut & paste.	1323859687000000	1323888271000000
1552	enhancement	dread	dread	ckan-sprint-2011-12-19	closed	fixed	Be able to set password on the command line when creating users	(instead of using the prompt) - needed for deployment	1324056178000000	1324056240000000
1575	enhancement	dread	dread	ckan-sprint-2012-01-23	closed	fixed	tag punctuation lost in ca.ckan.net import	Last week I imported ca.ckan.net datasets into thedatahub.org, but the tags seem to have lost their dashes, underscores and dots.	1324316860000000	1326808657000000
1613	defect	dread	dread	ckan-sprint-2012-01-23	closed	wontfix	Post-dataset-edit URL has #section	" 1. Viewing a dataset, hit 'Edit'
 2. Click on the ""Basic Information"" tab (note: URL has suffix   #section-basic-information
 3. Click 'Save'
 4. URL still has suffix #section-basic-information

Affects 1.5, 1.5.1, 1.5.2a"	1325685555000000	1326813924000000
1623	enhancement	dread	dread	ckan-sprint-2012-01-23	closed	fixed	Dump to exclude deleted objects	"The database dump currently contains all Packages and their associated objects, even those that have been set to state=deleted. We should exclude these from the dump now.

Dump = paster db simple-dump-csv/json

== reasoning ==

The dumps are designed for end-users to be able to run scripts on the mass of data. Since end-users don't see state=deleted packages then they shouldn't need them in the dump. In fact their presence in the dump probably confuses them.

Admins get the full database anyway in the backup pg_dump.

We only included them in the user dump because it was designed before use of state=deleted was established.

Time estimate: 2h"	1326118987000000	1326892264000000
1624	defect	dread	pudo	ckan-sprint-2012-01-23	closed	fixed	Typo in dataset edit mode	"Futher Information -> Further Information
"	1326121197000000	1326216362000000
1626	enhancement	dread	dread	ckan-sprint-2012-01-23	closed	fixed	'About CKAN' page update	thedatahub.org/about contains info that is very general to the CKAN and really quite technical. The text should be changed to be both specific to thedatahub.org and provide the context in a non-technical way. It should be easy to customise the About page to be appropriate for say new-york.ckan.net - a bit of info about who runs it, plus the general stuff about CKAN powering it and it was written by OKF to further open data.	1326205236000000	1326215877000000
1627	defect	dread	dread	ckan-sprint-2012-01-23	closed	fixed	favicon broken	"On thedatahub.org the favicon doesn't display. i.e. the CKAN logo should appear in the browser's tab.

Original ticket #48"	1326207102000000	1326890614000000
1629	defect	dread	dread	ckan-sprint-2012-01-23	closed	fixed	permissions changed during upgrade to 1.5.1	This was seen on datacatalogs. When we upgraded it 1.5 to 1.5.1 we saw some permissions being reset so that it could be spammed. Anonymous and logged in users were given anon_editor and editor permissions. I don't know what the previous config was. Permissions for sysadmins remain unaffected. 	1326215162000000	1326823222000000
1659	defect	dread	dread	ckan-sprint-2012-01-23	closed	fixed	Cannot logout if CKAN mounted at non-root url	"If you set WSGIScriptAlias to mount CKAN at a URL other than / then you cannot logout without adjusting the OpenID logged_out_url to match in who.ini config. e.g.

 [plugin:openid]
 ...
 logged_out_url = /sub/dir/user/logged_out

Note: all the other URLs in who.ini should not have the /sub/dir/ - it is just this one that doesn't take account of the mounting point.

The solution is to fix-up the repoze.who OpenID plugin to take account of the mounting point."	1326716302000000	1326747205000000
1682	enhancement	dread	dread	ckan-v1.6	closed	duplicate	[super] Translatability for EC	"We need to be able to translate:
 * EC extension (templates, form)
   * Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
 * Field values in taxonomy (e.g. country names - Eurovoc)
   * #1665 Research into Eurovoc
   * Display Taxonomy in different languages
 * Field values not in taxonomy (e.g. title & description)
   * Use extra fields e.g. _i18n_title_fr = Le data.
   * (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)

Timescale to complete this - Mar/Apr.

Related work to interface with:

 * Eurovoc - Sean
 * i18n in search index - Adria"	1326911899000000	1330038786000000
1683	defect	dread	dread	ckan-sprint-2012-01-23	closed	fixed	Dataset search results - last item out of order	"On each page of package search results, all the items are neatly sorted apart from the last item of the page. SOLR gets the sorting of the results incorrect. 

This is a known issue: https://issues.apache.org/jira/browse/SOLR-1777 affecting SOLR 1.4 only (which comes with Ubuntu 10.04)

It is highlighted in CKAN test ckan/tests/functional/test_pagination.py:TestPaginationPackage of commit 39096ed54bda86d043521b08b2e14fc5e283a0ff which fails most of the time it is run (passes intermittently)."	1326971864000000	1326976925000000
1688	enhancement	dread	dread	ckan-sprint-2012-02-06	closed	fixed	API for changing the permissions on an object	"We need an API for changing the authorization (permissions) on a model object.

This was neglected in #1253 because of various changes to authz were looming. But six months later it still hasn't been done.

Suggest this is just moving _add_user_object_role from ckan/lib/base.py to the logic layer, but no doubt it will be more complicated.

Estimate: 3 days"	1327061809000000	1328289101000000
1695	enhancement	dread	dread		closed	fixed	Relationships - improvements	"Suggested by Richard Cyganiak in http://lists.okfn.org/pipermail/ckan-discuss/2012-January/001977.html

 * new relationship type links_to / linked_from
   * Maybe with extra 'count' property?
 * creating a new relationship should be post to the relationship register, not the source entity.
 * add missing doc about creating a new relationship
 * 'Model Formats' section seems to be missed out of the docs since release 1.4.2. See: http://readthedocs.org/docs/ckan/en/ckan-1.4.2/api/version2.html#model-formats
"	1327323280000000	1330085001000000
1708	defect	dread	dread	ckan-sprint-2012-02-06	closed	fixed	SOLR configuration lost	"The SOLR url, user and password defined in the CKAN config file are ignored and the default SOLR url is used.

This causes:
 * ""0 datasets"" displayed on the home page
 * Dataset searches result in 0 results and a small message ""There was an error while searching"". (Nothing about it in the logs)

== To reproduce ==

This bug is only visible if your SOLR instance is not at the default place. To quickly reproduce this problem, setup your machine as a SOLR multicore instance and run: ""paster db clean && paster create-test-data && paster serve development.ini"". It quits with the error: ""solr.core.SolrException: HTTP code=400, reason=Missing solr core name in path""

== Code affected ==

* Does not affect any released versions (1.5.1 is clean).
* Introduced to master: https://github.com/okfn/ckan/commit/627382e6ff575146da221efc50321a41152d8b0f by Ross"	1327493428000000	1327580995000000
1709	defect	dread	dread	ckan-sprint-2012-01-23	closed	fixed	Simple_search fails with all packages	"The home page always shows the total number of packages as 0, because the query is '*:*' and SQL search doesn't know how to handle it.

Also, simple_search configuration is not documented.

Also, no tests for sql search."	1327498366000000	1327659722000000
1710	enhancement	dread	dread	ckan-sprint-2012-02-06	closed	fixed	'Announce' email list for thedatahub	"We want to make email announcements to users of thedatahub.org.

The most simple way to do this is to get a dump of the email addresses and subscribe/invite them to a mailman list.

Different ways to do it:
 * 'paster dump-users-csv' creates a CSV file. Apache serves up the directory using password protection. Is this secure enough?
 * CKAN admin pages provides a button to produce the CSV and download it. A config option may be necessary to enable this (disabled by default).

Something more advanced in the future could be:
 * CKAN tells mailman about new subscribers using the mailman web interface. Could be brittle.
 * Mailman doesn't have an Internet API, but it does have a python interface. So CKAN could tell mailman about new subscribers using a python proxy. There may be one out there, or maybe easy to do. But annoying to have to maintain another process running on machine.
 * CKAN sends out the emails itself. A sysadmin is given a form. Users have an opt in/out option in their account. But receiving emails is a sensitive thing so it would need to be right first time, and there are several risks to this approach.
"	1327579259000000	1327583922000000
1725	defect	dread	dread	ckan-sprint-2012-02-06	closed	fixed	Tag autocompletion and commas don't mix	"If you use tag autocompletion, it automatically adds a comma after the tag. This is fine if you are going to add another tag, but not fine if it is the last one. If you don't manually remove the final comma than you get form validation error ""Tag string: Tag """" length is less than minimum 2""

Affects only version 1.5.2a."	1328006709000000	1328204489000000
1783	defect	dread	dread	ckan-sprint-2012-02-20	closed	fixed	Locale change exception when unicode in URL	If the current page URL has unicode in it (e.g. {{{u'/tag/biocombust\xedveis'}}}) then I'm not sure how why it isn't more encoded, but we are seeing links such as {{{http://beta.dados.gov.br/dados/locale?locale=hu&return_to=/dados/tag/biocombust%C3%ADveis&hash=7fd941b5ba1a28bc84406c4191bf07b45a42a251}}} which cause an exception when the foreign character is passed to ckan.lib.hash.get_message_hash().	1328611485000000	1328612744000000
1786	enhancement	dread	mirko.spasic@…	ckan-sprint-2012-02-20	closed	fixed	Relationships lost when editing package	If I have a relationship between two packages, and then I edit any property of one of them (Author email, for example), the relationship will automatically be lost.	1328635226000000	1328640597000000
1791	defect	dread	dread	ckan-sprint-2012-02-20	closed	fixed	Changing locale on /dataset/new causes exception	"When you are on the /dataset/new page and you try and change locale then you get a 500 error. 

This is because it adds the '__cache' parameter, to ensure any proxy cache in the chain does not just send the cached page.

e.g. 
http://127.0.0.1:5000/dataset/new?__cache=37713707
"	1329134556000000	1329138315000000
1799	defect	dread	dread	ckan-sprint-2012-03-05	closed	fixed	To login as a different user or register, you need to logout first	"When you are logged in, you cannot login as a different user before first logging out. (This is a restricting in repoze.who, which gives priority to an existing recognised identity over another one you supply.)

Yet, the CKAN login form appears when you try to do something you're not allowed to, even if you are logged in already. You login with a different identity and nothing happens - you remain as the first identity - this is incorrect.

(It is also a little confusing to be sent to the log-in page, although we do have a flash message explaining, and you might want to relogin as a superuser if you can.)

Whilst in this area, it would be worth fixing the problem for developers where you are logged in, do ""paster db clean"" and then try to log in - AuthTkt recognises your old cookie but the User object doesn't exist, so you get an error logging in. In this instance we should tell people to log-out."	1329402820000000	1330002176000000
1802	enhancement	dread	dread	ckan-sprint-2012-03-05	closed	fixed	Investigate CKAN deployment on Windows	"See how possible it is to run CKAN on a Windows server.

Plan:
 * Get CKAN code and dependencies installed on Windows
   * ideally as similar to standard package install as possible
   * virtualenv/pip should work
   * try Cygwin to help
   * SOLR and Postgres could potentially run on other servers if necessary.
 * Try to run on Apache
 * See if there are any issues"	1329409512000000	1329830097000000
1829	defect	dread	dread	ckan-sprint-2012-03-05	closed	fixed	Changing back to English prints the flash message in the previous non-English language	"On the homepage click ""francais"" and then ""English"". The flash message reads ""Le langage a été fixé à: français"" when it should say ""The language is now: English""."	1330000660000000	1330001990000000
1830	enhancement	dread	dread	ckan-sprint-2012-03-05	closed	fixed	OpenID disablable	"Allow OpenID to be disabled as a login option. 

Currently (CKAN 1.6) there is a config option openid_enabled which stops the login option being displayed, but:
 * In the User edit form you can still add your OpenID
 * The repoze.who.openid middleware is still required - can it be avoided?"	1330002137000000	1330089912000000
2266	defect	dread	dread	ckan-sprint-2012-04-02	closed	fixed	Can't delete all of a package's resources over REST API	Nothing happens if you set resources=[] or resources=null.	1332932504000000	1332932634000000
2281	enhancement	dread	dread	ckan-sprint-2012-04-16	closed	fixed	Pagination styling converted to Bootstrap	"Make the search results pagination styled by Bootstrap. 

Needs putting into a <ul> structure. Current page and '..' need become links that are styled disabled.

Wanted by DGU."	1333967506000000	1333967862000000
2315	enhancement	dread	dread	ckan-sprint-2012-04-30	closed	fixed	Change Cookie expiry	"Change login cookie from a default expiry of 50 years to 2 years. You can also uncheck a 'remember me' checkbox on the login form for the cookie to just last the session.

Background conversation on ckan-dev:

DR: I wonder if anyone objects to the expiry of the login cookie to be
changed from 50 years to 2 years? 50 years might be appropriate for
thedatahub.org, but for government sites it seems (to me) to be too
lax.

Toby: is this the repoze.who cookie?  If so that seems sensible to me.

Rufus: Definitely agree. I would also like to see introduction of a standard
""remember me"" checkbox (set to true by default). At the moment a login
lasts forever (until you logout) automatically."	1334919449000000	1334919522000000
2325	enhancement	dread	dread	ckan-sprint-2012-04-30	closed	fixed	Format alphabet pager (tags page) for bootstrap	Also need to fix bug where 'Other' is never highlighted.	1335366110000000	1335366188000000
2373	defect	dread	dread	ckan-sprint-2012-05-15	closed	fixed	Group autocomplete doesn't work with multiple words	"e.g. 'Cabinet Office' returns 0 results, whereas 'Cabinet' returns 'Cabinet Office'.

{{{
/api/2/util/authorizationgroup/autocomplete?q=Cabinet%20Office
}}}

This feature only seems to be used by DGU at the moment (I think)"	1336139787000000	1336142514000000
2384	enhancement	dread	dread	ckan-sprint-2012-05-15	closed	fixed	Rights tool factored out	The command line tool 'rights' is quite handy but it is glued to the CLI. I'm going to factor out the bit which searches for objects etc so it can be used by CreateTestData etc and will be used by DGU.	1337080794000000	1337100810000000
2390	enhancement	dread	dread	ckan-sprint-2012-05-29	closed	fixed	create_test_data for multiple users, rights	It would be useful (well for DGU, but no doubt for core too) to have be able to create multiple users with full properties using one commit. Also to be able to create rights (UserObjectRoles).	1337098459000000	1337100788000000
2394	enhancement	dread	dread		closed	fixed	Separate off create_test_data CLI	We need to separate the CLI part of CreateTestData from the bit which creates test fixtures. This is because of the problem with imports and logging. Otherwise we have to do unpythonic things to get logging to work when deriving from CreateTestData from an extension such as ecportal / dgu. I've only introduced logging into CreateTestData but I think it is a good thing to have. This change doesn't cause breakages	1337159591000000	1337161917000000
2401	enhancement	dread	dread	ckan-sprint-2012-05-29	closed	fixed	CLI for time/speed profiling	To enable you to easily track down what is taking all the time when you make a request.	1337269571000000	1337273042000000
2942	defect	dread	dread		closed	wontfix	API POST barfs on interesting Content-Type headers	"When POSTing to the API, if specified, the 'Content-Type' header must be blank or 'application/x-www-form-urlencoded'. Otherwise we get an error like: ""Bad request - JSON Error: Could not extract request body data: Bad content type: \'; charset=utf-8\'""""

The problem is that this is a very reasonable header to send. Indeed requests 0.14 sends this particular header.

This affects all versions of CKAN.

This is due to webob/requests.py:1248 being pretty basic."	1348593156000000	1348611144000000
2955	defect	dominik	dominik	ckan 2.0	closed	fixed	Recline should be updated		1349270534000000	1350577952000000
2961	enhancement	dominik	dominik	ckan 2.0	assigned		Preview plugin endpoint	"Users could write ckan extensions that offer previews for a certain datatype. 

Advantages:
 - We can say that you can add your own previews
 - Every preview would have to define it's own dependencies
 - External development (community creates new extensions?)

Questions to answer:
 - Specification of the interface
 - Precedence of previews for the same data types
 - Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible. 

As far as I can see, this should be fairly easy to implement by following these instructions: http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface"	1349696367000000	1350577990000000
3013	defect	dominik	dominik		new		common-error-messages is unreadable	"Since the update of the doc theme, the page became unreadable.

http://docs.ckan.org/en/latest/common-error-messages.html"	1352553505000000	1352553505000000
1558	enhancement	David Raznik	jilly mathews	ckan-future	new		Publisher Tools	Summarise final set of requirements for this and finish development and test. Estimated 10 working days. 	1324291573000000	1324291573000000
1560	enhancement	David Raznik	jilly mathews	ckan-future	new		Follow extension	"Estimate 2 days to finish dev and test.

David can you add any info needed here?"	1324291879000000	1324291879000000
1561	enhancement	David Raznik	jilly mathews	ckan-future	new		To do extension	Can we finish this ready for release on data hub and CKAN Hosted.	1324291972000000	1324291972000000
1563	enhancement	David Raznik	jilly mathews	ckan-future	closed	invalid	Finish Data Storage	Unsure what needs to happen here. Need to list outstanding tasks and implement.	1324292346000000	1324314806000000
1564	enhancement	David Raznik	jilly mathews	ckan-future	new		Structured Data (Data API)	"Basic websotre exists but this may be not what is described yet. 

CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first."	1324292834000000	1324292834000000
1566	enhancement	David Raznik	jilly mathews	ckan-v1.7	closed	fixed	[super] Finalize Google analytics extension	" * ~~(Deploy it)~~
 * Integrate with stats plugin (now that is in core) - #1101
 * Blog about it #1581
 * (Other tickets re upgrading for v1.5.1 - done but where are tickets?)
"	1324293168000000	1338203338000000
1567	enhancement	David Raznik	jilly mathews	ckan-future	new		Finish QA extension	Requires change to celeryd. Estimated 4 weeks.	1324293599000000	1324293599000000
1568	enhancement	David Raznik	jilly mathews	ckan-future	closed	duplicate	Moderated Edits	Can this be released as a standard CKAN feature?	1324293776000000	1325267998000000
1569	enhancement	David Raznik	jilly mathews	ckan-future	new		Wordpressser	How much effort will this be to be ready to use?	1324294056000000	1324294056000000
1570	enhancement	David Raznik	jilly mathews	ckan-future	closed	invalid	Integrated file Storage	Is this ready for release? What needs to be done?	1324294142000000	1324314741000000
1572	enhancement	David Raznik	jilly mathews	ckan-future	new		Meta data Harvester	Need to write custom harvesters for each client. Is it worth having one for data hub?	1324294509000000	1324294509000000
1573	enhancement	David Raznik	jilly mathews	ckan-future	new		Apps and Ideas	Estimate 2 weeks for someone to finish and test.	1324294593000000	1324294593000000
1557	enhancement	David Rasnik	jilly mathews	ckan-future	new		Complete Webstore Preview Extension	"Finish any work out standing on web store preview extension to be able to package and release.

Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted."	1324291253000000	1324291253000000
1553	defect	aron.carroll	dread	demo phase 5	closed	fixed	Upload handles non-logged in user badly	"When you install ckanext-store (Storage extension) then CKAN displays 'Upload a file' button in the dataset/resource edit tab.

If you are not logged in, then this feature doesn't work, but there is nothing to indicate this. The button should be disabled.

Furthermore, it lets you think you are starting the upload process, the spinner spins, it says 'Uploading...' and stays like that for a long while.

This is a problem CKAN 1.4.3(ish) - 1.5.1 (current version) when you install the current ckanext-storage."	1324056405000000	1343226756000000
1718	defect	aron.carroll	dread		closed	fixed	Deprecated JS properties used	"I get this warning when viewing the home page:
{{{
event.layerX and event.layerY are broken and deprecated in WebKit. They will be removed from the engine in the near future.
}}}
CKAN: latest master
Browser: Chrome Ubuntu

Minor, but worth checking?"	1327665842000000	1340703280000000
2225	enhancement	aron.carroll	rgrp	demo phase 2	closed	fixed	Run jshint on our javascript and clean up as needed		1331407316000000	1343220502000000
2267	enhancement	aron.carroll	rgrp	demo phase 5	closed	fixed	Add generator=ckan tag in head	Add generator=ckan tag in head of templates to identify site as generated by CKAN	1333040018000000	1343229146000000
2354	enhancement	aron.carroll	aron.carroll	demo phase 5	closed	wontfix	Front end performance improvements [super]	"Try and increase the page load times for the front end.

= Now = 

 * Always include scripts at the bottom
 * Include CSS at the top
 * Load fonts with JS
 * Concatenate and minify JavaScript files
 * Create a build script to load dependancies seperately
 * Enable Gzip compression on resources
 * Enable future cache headers on resources

= Next = 

 * Support serving assets from a CDN
 * Look into optimising the CSS - Remove large portions of unused CSS
 * Remove extreaneous libraries, can jQuery UI be replaced with bootstrap? 

= Tools = 

 * Webkit Inspector Audits
 * YSlow Firebug Plugin - http://yslow.org/
 * Speed Tracer -  https://chrome.google.com/webstore/detail/ognampngfcbddbfemdapefohjiobgbdl

= Continuous Integration =

YSlow can be hooked up to Jenkins via Phantom JS"	1335885042000000	1343317505000000
2356	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-05-15	closed	fixed	Localise JavaScript strings		1335886092000000	1338205356000000
2357	enhancement	aron.carroll	aron.carroll	demo phase 5	accepted		Create build script for front-end resources	Should minify and concatenate JavaScript and CSS as part of #2354	1335886181000000	1343220524000000
2367	enhancement	aron.carroll	aron.carroll		closed	fixed	Style the analytics view counts on datasets	Currently an eye and a number, needs to look a little nicer.	1335892789000000	1336035031000000
2372	task	aron.carroll	aron.carroll		closed	fixed	Create frontend style guide	"This should cover formatting and best practices for HTML, CSS and JavaScript

Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch.
"	1336071043000000	1343220160000000
2408	enhancement	aron.carroll	markw	demo phase 3	closed	fixed	Only 4 'extras' can be added at a time	The settings > extras page for a dataset allows extra fields to be added, but only four can be added a time in a single atomic update. It would be helpful to have an 'add another' button on the form that can be used any number of times.	1337683639000000	1343657014000000
2418	enhancement	aron.carroll	aron.carroll		closed	fixed	Facets disappear once selected	"Currently facets in the new demo site disappear from the sidebar once they are selected.

http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa

Instead they should be marked with an ""active"" class and clicking the link should remove the filter in the same way as clicking the cross in a token below the search field works."	1337793330000000	1341833880000000
2419	enhancement	aron.carroll	aron.carroll		closed	fixed	Add dataset form needs hooking up to action/controller	Currently for some reason I'm not getting the errors/data dict in the package/new.html template.	1337793401000000	1343220027000000
2421	enhancement	aron.carroll	toby		closed	fixed	demo-theme pills break in dataset search	if pills cross onto 2 lines they split which they shouldn't to see just add some tags/formats till it breaks	1337865196000000	1343220041000000
2441	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo search page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments

Implementation:

http://s031.okserver.org:2375/dataset/"	1338210155000000	1339670118000000
2442	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo dataset overview	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61752357/comments

Implementation:

http://s031.okserver.org:2375/dataset/adur_district_spending"	1338210249000000	1339670612000000
2443	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo dataset data (resource) page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62308209/comments

Implementation:

(to be added once test server is working)"	1338210382000000	1341824663000000
2444	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo dataset related page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments

Implementation:

http://s031.okserver.org:2375/dataset/adur_district_spending/related"	1338210559000000	1339670232000000
2447	enhancement	aron.carroll	aron.carroll	demo phase 5	closed	fixed	Create demo add dataset page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62234769/comments

Implementation:

http://s031.okserver.org:2375/dataset/new
http://s031.okserver.org:2375/dataset/new_resource"	1338210835000000	1343924125000000
2448	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo groups overview page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

http://s031.okserver.org:2375/group"	1338210960000000	1339669678000000
2449	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo group page	"Discussion:

 https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

http://s031.okserver.org:2375/group/test-group"	1338211060000000	1339670041000000
2450	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo add group form	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

http://s031.okserver.org:2375/group/new"	1338211136000000	1339670997000000
2452	enhancement	aron.carroll	aron.carroll	demo phase 5	closed	fixed	Create demo user profile page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

http://s031.okserver.org:2375/user/aron"	1338211295000000	1343924143000000
2453	enhancement	aron.carroll	aron.carroll	demo phase 5	closed	fixed	Create demo edit user profile page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

http://s031.okserver.org:2375/user/edit"	1338211351000000	1343924178000000
2455	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo register page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998362/comments

Implementation:

http://s031.okserver.org:2375/en/user/register"	1338211541000000	1338998214000000
2456	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Create demo forgot password page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments

Implementation:

http://s031.okserver.org:2375/en/user/reset"	1338211604000000	1339670080000000
2458	enhancement	aron.carroll	aron.carroll	demo phase 3	closed	fixed	Create demo statistics page	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998513/comments

Implementation:

http://s031.okserver.org:2375/en/stats"	1338211906000000	1344262256000000
2474	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Trial Jinja 2 as templating system	"Percieved benefits of switching to Jinja 2

 * Block inheritance for base templates which is much simpler to understand.
 * Large number of developers (and designers) are familiar with it due to its
   similarities with the Django templates.
 * Should be much faster at parsing and rendering templates.
 * Has localisation support.
 * Good documentation both on it's website and on sites like Stack Overflow.

We should convert a couple of existing templates to test the switchover and then get peoples feedback."	1338307326000000	1338982780000000
2493	enhancement	aron.carroll	toby	demo phase 5	accepted		Demo site 404 needs some love	"templates/error_document_template.html

I've converted the 404 page to the new style but we probably want to serve this as a 'full page' ie content is 100% page width not sidebar and primary content

just go to a non-existant url to see

Discusion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63374042/

Implementation:

http://s031.okserver.org:2375/404"	1338986081000000	1343135815000000
2504	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Remove all Genshi templates from demo theme		1339156577000000	1339409776000000
2510	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	fixed	Demo tag lists should link to the search page	"Instead of pointing to /tags/{tagname} all tag links should just point to a dataset search filtered by that tag.

Examples are on the homepage and the dataset pages."	1339513217000000	1339670520000000
2511	enhancement	aron.carroll	toby		closed	fixed	header logo should be link to /		1339513880000000	1339517817000000
2522	enhancement	aron.carroll	toby		closed	fixed	view user gatavar not sqaure	the user pic should be square on the user read page	1339604954000000	1341499095000000
2525	enhancement	aron.carroll	aron.carroll	demo phase 1	closed	fixed	Demo theme should auto-generate slugs on forms		1339679954000000	1341497344000000
2526	enhancement	aron.carroll	aron.carroll	demo phase 5	closed	fixed	Demo theme new custom form fields should be created when needed	"Implementation:

http://s031.okserver.org:2375/dataset/new_metadata"	1339680105000000	1343924205000000
2527	enhancement	aron.carroll	aron.carroll	ckan-sprint-2012-06-25	closed	wontfix	Implement a method of loading templates into the demo site	"JavaScript needs to be able to insert html templates into the document.

There are three common solutions to templating at the moment.

 1. Insert the html into <script> tags at the bottom of the page and pull them in using the DOM and innerHTML. This is a very simple solution but not very portable.
 2. Request the HTML snippets as plain text/html using XHR. This adds an extra HTTP request per template but is very clean. It would also be possible to inline the HTML into the JS at the minification stage for production.
 3. Include the HTML in JavaScript strings (horrible to maintain and bloats the js)

It makes sense to keep these with the other templates so that we can take advantage of localisation where possible. This indicates 1 or 2."	1339682884000000	1340116130000000
2528	enhancement	aron.carroll	aron.carroll	demo phase 1	closed	fixed	Create demo edit dataset page	"Implementation:

http://s031.okserver.org:2375/dataset/edit/5b451cab-8671-4513-91c4-257784c1f484"	1339686982000000	1340617176000000
2556	enhancement	aron.carroll	aron.carroll	demo phase 1	closed	fixed	Update module styles on the demo theme	"They've become inconsistent as the design has changed.

 * Remove white stripe
 * Make text sizes consistent
 * Make white space around text boxes consistent"	1340035028000000	1340212090000000
2557	enhancement	aron.carroll	aron.carroll	demo phase 1	closed	fixed	Demo issues on the search dataset page	" * Crosses have disappeared
 * Formats should stretch across the page"	1340035222000000	1340185609000000
2558	enhancement	aron.carroll	aron.carroll	demo phase 1	closed	fixed	Demo site add dataset has missing functionality	" * Needs tags
 * Needs groups
 * Needs slug
 * Licenses links to something
 * Needs tooltips (Ira to write some tooltips)"	1340035280000000	1340615257000000
2561	enhancement	aron.carroll	aron.carroll	demo phase 1	closed	fixed	Demo theme edit dataset is missing functionality	"* Add tags
* Add groups
* JavaScript for custom fields"	1340035457000000	1341497328000000
2563	enhancement	aron.carroll	aron.carroll	demo phase 1	closed	fixed	Add font based icons to the demo theme	"Allowing them to be displayed in many sizes, colours and resolutions.

http://www.zurb.com/playground/foundation-icons"	1340035831000000	1341501320000000
2564	enhancement	aron.carroll	shevski	demo phase 1	closed	fixed	Updates to additional info fields on datasets	"E.g. http://s031.okserver.org:2375/en/dataset/adur_district_spending/resource/04127ad5-77e5-4a08-9f40-12d3c383e460

When there are no maintainer/contact details or extra fields we should hide the additional info box completely. Otherwise hide the last updated and licence fields which are already displayed in sidebar.

Update sidebar to provide context - so 'CSV' -> 'CSV format' + 'unknown' -> 'Last updated: unknown'
"	1340038534000000	1340118952000000
2565	enhancement	aron.carroll	shevski	demo phase 1	closed	fixed	Social links to open in new tab/JS lightbox	Share this (twitter/fb) links open in new tab (no JS) or nice lightbox of some description in demo	1340039322000000	1340116053000000
2567	enhancement	aron.carroll	shevski	demo phase 1	closed	fixed	remove resource type field from add data step	No real need/use for this for demo/ODS yet	1340039447000000	1340120447000000
2568	enhancement	aron.carroll	shevski	ckan-sprint-2012-06-25	closed	fixed	Crop/shorten titles in breadcrumb headings	So that they don't interfere with buttons such as here: http://s031.okserver.org:2375/en/dataset/afterfibre/resource/5d19f9fa-c5a4-426c-84de-c624d6e8c3b3	1340039534000000	1340108132000000
2570	enhancement	aron.carroll	shevski	ckan-sprint-2012-06-25	closed	fixed	add 'back to dataset' button to resource pages	Suggest making this consistent with http://s031.okserver.org:2375/dataset/adur_district_spending/related by adding same button in top right of resource pages	1340039680000000	1340107841000000
2576	enhancement	aron.carroll	toby	demo phase 1	closed	fixed	resource side box in dataset resource view broken	"assuming you have datahub data

http://localhost:5000/en/dataset/2008_facility_ghg_emissions_by_province_territory/resource/28650645-8878-4826-9405-f20b9dfa4c12

also the breadcrumb becomes overly long too

"	1340199857000000	1340617688000000
2584	enhancement	aron.carroll	toby	demo phase 1	closed	fixed	edit profile needs new theme	"

http://localhost:5000/user/edit/<my user name>"	1340359925000000	1340620576000000
2598	enhancement	aron.carroll	toby	demo phase 1	closed	fixed	stages css broken in FF	I see X where none should be	1340708102000000	1341218567000000
2610	enhancement	aron.carroll	shevski	demo phase 1	closed	duplicate	Text / link changes	"In the footer:
Change 'Powered by CityData' to Powered by DataSuite (with link to ckan.org/datasuite) 

Links in left hand side of footer should change to a link to ckan.org, link to OKFN, link to ckan.org/datasuite/about and link to Pricing Page (on ckan.org .. will be /datasuite/solutions)

In create dataset form, prefilled tag examples should include one example with a space, e.g. ""mental health"""	1340810738000000	1341501424000000
2611	enhancement	aron.carroll	shevski	demo phase 1	closed	fixed	Text / link changes	"In the footer:
Change 'Powered by CityData' to Powered by DataSuite (with link to ckan.org/datasuite) 

Links in left hand side of footer should change to a link to ckan.org, link to OKFN, link to ckan.org/datasuite/about and link to Pricing Page (on ckan.org .. will be /datasuite/solutions)

In create dataset form, prefilled tag examples should include one example with a space, e.g. ""mental health""

Change 'Groups' to 'Publishers' everywhere

Homepage copy/images (Ira to supply?)"	1340810740000000	1342008509000000
2613	enhancement	aron.carroll	shevski	demo phase 2	closed	wontfix	Javascript functionality	" 1. file uploads on dataset creation
 2. create dataset validation
 3. create dataset slugs
 4. Add dataset to group process: pop-up allowing you to quickly search for a pick a dataset to add to group. Must check you have necc auth over dataset to be added
 validation of user name
 5. Language box
 6. toggling activity stream
 7. Search sorting
 8. autocomplete on tags
 9. add and remove custom fields on forms
 10. the rest:
 tooltip on popular datasets with number of views

 facets to update automatically

 creating a dataset without reloading page between steps

 hover on licences information

 autocomplete on search terms

 group filtering

 add comment to datasets

 social share buttons in lightboxes

 dataset counts on homepage?"	1340814271000000	1342620543000000
2614	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	improve the documentation around less/node	"needs to move to a better home

better ubuntu install instructions plus test they work

`apt-get install nodejs`

integrate with paster"	1340815928000000	1343685572000000
2617	enhancement	aron.carroll	aron.carroll	demo phase 2	closed	fixed	Move data preview into demo		1340875192000000	1342088744000000
2626	enhancement	aron.carroll	toby		closed	invalid	ff render issue in edit dataset	"http://localhost:5000/dataset/edit/2010-nigeria-budget

the edit url button appears too high up"	1341499419000000	1341499834000000
2636	enhancement	aron.carroll	aron.carroll	demo phase 2	closed	duplicate	Style activity stream in user profile		1341835166000000	1342090614000000
2637	enhancement	aron.carroll	aron.carroll	demo phase 2	closed	fixed	Submit search form when user changes sort order		1341835218000000	1342617226000000
2643	enhancement	aron.carroll	aron.carroll	demo phase 2	closed	fixed	Demo homepage redesign	"Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments"	1341848685000000	1342617247000000
2647	enhancement	aron.carroll	toby	demo phase 5	closed	fixed	api bullshit button	make it as a snippet calling thing like aron would love	1341920530000000	1343229185000000
2659	defect	aron.carroll	shevski	demo phase 2	closed	fixed	HTML preview window in demo too small	"See http://demo.ckan.org/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b

David previously fixed this on some version of demo, seems to have regressed. Just want the window to be longer"	1342010723000000	1342089310000000
2661	enhancement	aron.carroll	toby	demo phase 2	closed	fixed	delete dataset/group javascript confirm	"in 2375-demo-theme-development

templates/package/confirm_delete.html needs to be made to look nice note some writing in black

delete button needs nice home on dataset/edit

you can add js support add ?confirm=yes to url on dataset button to force delete

Please note delete is not yet implemented - but will be in the next few hours"	1342012395000000	1342716070000000
2665	enhancement	aron.carroll	toby	demo phase 2	closed	fixed	add dataset stages need to be form buttonable	"the stages snippet needs to be inside <form> so I can make some links buttons - eg stage 1 -> stage 2 I want to save the dataset not just move to add resource

if you can just move it within the form html leaving everything else the same - I can then add buttons and pass back for styling"	1342023971000000	1342088650000000
2666	enhancement	aron.carroll	toby		closed	fixed	templates/package/resource_edit.html	"There is a new template for editing resources outside of the add dataset logic

this is the template it just throws some crap onto the page at the moment - we need to create a form - the form will end up with the usual data/edit stuff but for now you can use the resource to populate the form and I can then change it to use data - I just want to know which fields we'll have


templates/package/resource_edit.html

when done reassign to me and I'll get it functioning"	1342084373000000	1343220176000000
2669	enhancement	aron.carroll	toby	demo phase 5	closed	fixed	style [Draft]	"I've added [Draft] to partial completed datasets in the package list and read  commit 6a8e74d0


templates/snippets/package_list.html

templates/package/read.html

do you want to style them in some way?"	1342174110000000	1343907572000000
2675	defect	aron.carroll	shevski	demo phase 2	closed	fixed	editing related item uses 'create' button instead of save	"see http://s031.okserver.org:2375/dataset/gold-prices/related/edit/46ba1174-3666-486e-b822-b5b45ffbb703

when editing a record, button at bottom right should change to save instead of create"	1342435576000000	1342461534000000
2677	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	related item links should open in new tab	Clicking on the blue buttons or images on related page http://s031.okserver.org:2375/dataset/gold-prices/related should take you to a new tab instead of navigating you away	1342435923000000	1342461576000000
2681	enhancement	aron.carroll	shevski	demo phase 2	closed	duplicate	autofill on resource format	"This is pretty key for standardization.

Also would like us to change the grey text to be: e.g. CSV, JSON, [more examples... i have no idea what they key ones are tbh]"	1342440406000000	1343135874000000
2684	enhancement	aron.carroll	toby	demo phase 2	closed	fixed	.js leaving forms warning	"I think it would be nice to have all form pages monitor their forms - excluding search ones - and if a field has changed then warn the user that they have not saved the form data but are leaving the page.

This would help with things like changing resources on the add dataset resource form etc."	1342510477000000	1342539637000000
2687	defect	aron.carroll	shevski	demo phase 2	closed	fixed	uploading file breaks add dataset process	"Once you upload a file, all the buttons on the page stop working. You can't complete & go to next step or save & add another or go to previous

Clicking on these just seems to re-load the page

http://s031.okserver.org:2375/dataset/new_resource/super"	1342518422000000	1342534855000000
2689	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	clicking on 'save & add another' doesn't act as expected	"Clicking on 'save & add another' button here http://s031.okserver.org:2375/dataset/new_resource/yo takes you to step 3 instead of leaving you on step 2 when you haven't added any data.

Instead it should show an alert/ message that no data has been added & that you need to link to a file or upload something

Also, can upload be option 2 instead of 3?"	1342538126000000	1342617293000000
2690	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	long URL misaligns edit button for slug	See attachment. Just need to amke sure that there's enough spacing around the edit button and that it lines up so it doesn't look too wierd	1342538301000000	1342542394000000
2692	defect	aron.carroll	shevski	demo phase 2	closed	fixed	'add your own data' link in no search results wrong	"the ""add your own data""  link in ""Try another search term, browse the datasets below or add your own data."" here http://s031.okserver.org:2375/dataset?q=stuff doesn't take you to add a dataset, just back to the search/datasets page"	1342540803000000	1342617328000000
2695	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	autocomplete for tags	in dataset creation / edit form	1342619891000000	1343138240000000
2696	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	autocomplete for resource format	in create/edit dataset forms	1342619938000000	1343138454000000
2700	enhancement	aron.carroll	shevski	demo phase 3	closed	fixed	language box	lang box improvements: styling, images, submit upon selection	1342620223000000	1344505614000000
2706	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	hide all prefilled fileds from resource edit form page	i.e. mimetype, size, last updated etc	1342621381000000	1342707718000000
2713	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	button colour not set properly in variables.css	"to reproduce run

paster color fuchsia -c development.ini

then go to http://localhost:5000/dataset/new

Add button is blue gradient at the top this should be overridden in variables.less

when added send this ticket to me and i'll update the color command to change it"	1342706428000000	1343656865000000
2720	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	no padding around gravitars on all users page	"http://s031.okserver.org:2375/user

needs some spacing between the image & user name

We could also display some stats here (e.g. number of dataset created, timestamp of last activity)"	1342948111000000	1343219999000000
2722	defect	aron.carroll	shevski	demo phase 3	closed	fixed	Ordering or search of datasets of a group brings up 'no results'	"1. go to a group, e.g. http://s031.okserver.org:2375/group/data-explorer
2. Order by anything, e.g. choose 'popular' and click go
3. ""sorry no datasets found"" message appears instead of ordering the group datasets in wanted order

http://s031.okserver.org:2375/group/data-explorer?q=&sort=views_recent+desc"	1342948854000000	1344510010000000
2723	defect	aron.carroll	shevski	demo phase 2	closed	fixed	Search result summary badly displayed	"Text goes over the order by text, with drop down floating on top meaning it's all impossible to read

http://s031.okserver.org:2375/dataset?q=unemployment%2C+data&sort=metadata_modified+desc"	1342949143000000	1343136190000000
2739	enhancement	aron.carroll	shevski	demo phase 3	closed	fixed	Help text for format field	"Can we please add some text next to the format field when user adding a new resource along the lines of:

""This will be generated automatically, but you can edit if you wish""

It always seems to work better when the user does not fill it themselves..."	1343123175000000	1343817919000000
2744	defect	aron.carroll	toby	demo phase 2	closed	fixed	license dropdown is cramped	"http://s031.okserver.org:2375/dataset/new

license dropdown - when open we need some line spacing at least in FF
I think it'd be nice to fix this in phase 2"	1343144543000000	1343219298000000
2749	enhancement	aron.carroll	toby	demo phase 5	closed	fixed	page title issue with super() super()	"title broken for

http://s031.okserver.org:2375/en/dataset/new_resource/gold-prices

we have inherited the title from the extended page not from the base

should be 'add data - ckan' not 'add data - create dataset - ckan'

any idea how we fix this?"	1343215828000000	1343232304000000
2753	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	rename breadcrumb on user page	"Currently lower case 'active' is the name of this page: http://s031.okserver.org:2375/user

Should be 'All Users' or 'Active Users'"	1343225088000000	1343227431000000
2754	enhancement	aron.carroll	shevski	demo phase 2	closed	fixed	Add + icon to two buttons	Specifically the 'add resource' button and the 'add item' button in related section	1343225177000000	1343227446000000
2759	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	resource upload pointer incorrect	"On the add resource forms new or existing dataset

when hovering over `Upload a file` css cursor should be pointer not text"	1343289469000000	1343651082000000
2760	enhancement	aron.carroll	shevski	demo phase 3	closed	fixed	fix spacing between data API button and other buttons on dataset page	"see spacing between 'download' button and 'data api' button
http://s031.okserver.org:2375/dataset/afterfibre/resource/66f5b980-06db-4ccd-a005-c18de4d68a4b"	1343297360000000	1343319140000000
2765	enhancement	aron.carroll	aron.carroll	demo phase 3	closed	fixed	Demo login error messages	"I'd suggest we add a generic ""failed to login, invalid fields"" in the error_summary block. Then more specific ""this field is required"" when a required field is left empty.

So in general, provide an error dict"	1343317842000000	1343817858000000
2767	enhancement	aron.carroll	aron.carroll	demo phase 3	closed	fixed	Upgrade macro form.select to use tuples instead of dicts	This keeps the intended order.	1343319298000000	1343645517000000
2776	enhancement	aron.carroll	shevski	demo phase 3	closed	fixed	add styled text field for header title	"to use on demos instead of the ckan logo & tag line.

Design here: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63387712/comments#190457381
(First, text only one at the bottom)"	1343724603000000	1343824375000000
2778	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	apps dashboard needs some design love	"apps dashboard available at http://localhost:5000/apps in main dev branch


I have got this to work in the demo however


a) it is a bit ugly


b) the template could do with some sorting eg the dropdowns should be being created in a nicer way "	1343734040000000	1343841130000000
2779	enhancement	aron.carroll	shevski	demo phase 3	closed	fixed	add 'deleted' to deleted datasets that admins can view	"Currently, as a logged in admin, you can view deleted datasets just as if they are live datasets. Nothing distinguishes them apart from normal datasets other than they don't show up on the dataset page

This is potentially very confusing
Can we add some kind of banner / label to deleted datasets to show they have been deleted?

& Do you guys need Sam to design this?

Related ticket for undeleting datasets -  http://trac.ckan.org/ticket/2780"	1343737077000000	1343903455000000
2781	enhancement	aron.carroll	shevski	demo phase 3	closed	fixed	add way to edit homepage text to simple theming	"demo.ckan.org has a small field for adding text on homepage
current text:
""Welcome to the CKAN demo
Try out standard CKAN functionality in a sandbox environment. Search for datasets directly from the homepage or by navigating to the Datasets search page where you can facet by tags, groups and format.""

Want to make this editable for demos"	1343738523000000	1344510922000000
2789	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	logo fixups	"With the text logo I'd like the tag line to be under the site name so some css like

`float:left;clear:left;font-size:13px;line-height:0.1;`

Also the site name can be larger if no tagline


commit daac54dbd856  I've added a new global g.header_css that sets the hgroup class to


'header-image'
'header-text-logo'
'header-text-logo-tagline'


I think maybe we should also have 'header-image-css'  for the equivalent of the class=""logo"" in the link to allow/stop the image replacement 
"	1343835141000000	1343835904000000
2791	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	Logo fix	"@aron
 

We should be able to have longish tag lines that look ok currently they wrap and stop the logo text being clickable


Can we have `white-space:nowrap;` for the tagline so long ones don't wrap and maybe look at aligning long ones

"	1343895034000000	1343906891000000
2792	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	dropdown fixes - open correctly	"http://s031.okserver.org:2375/  language dropdown expands downwards it should open upwards

select2 can do this see http://ivaynberg.github.com/select2/#documentation can you see if we can get this in ckan demo
"	1343895296000000	1343901501000000
2794	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	top buttons missing on edit resource	"http://s031.okserver.org:2375/dataset/resource-markdown/resource_edit/ba1b094d-697c-47c4-a72a-6c2557da5409

should have `back to resource` as a minimum"	1343897926000000	1343903064000000
2797	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	plain_textarea styling	"http://localhost:5000/ckan-admin/config uses a plain (non markup) textarea for custom css

I've added form macro plain_textarea to provide one, however it renders without a bottom border due to markup textarea styling can you fix this - thanks"	1344000422000000	1344250973000000
2800	enhancement	aron.carroll	shevski	demo phase 5	closed	worksforme	tooltip on popular datasets with number of views	be good to do this in phase 3	1344243925000000	1344544461000000
2801	defect	aron.carroll	shevski	demo phase 3	closed	fixed	no way to see full description of a related item	"e.g. see here: http://s031.okserver.org:2375/apps
same with related page on a dataset"	1344245809000000	1344531016000000
2802	enhancement	aron.carroll	aron.carroll	demo phase 3	closed	fixed	Need to tweak the JSON formatting of the localisations	"This is what we need the JSON to look like. Taken from the Jed documentation (http://slexaxton.github.com/Jed/).

{{{
{

      // The empty string key is used as the configuration
      // block for each domain
      """" : {
        // Domain name
        ""domain"" : ""the_domain"",

        // Language code
        ""lang"" : ""en"",

        // Plural form function for language
        ""plural_forms"" : ""nplurals=2; plural=(n != 1);""
      },

      // Other keys in a domain contain arrays as values
      // that map to the translations for that key.
      // Gettext suggests that you use english as your keys
      // in case the key isn't found, and it can just pass
      // the value directly through.
      // Note: by convention, the 0-index location of the translations
      // is never accessed. It's just a thing, I guess.
      ""a key"" : [ null, ""the translation"", ""the plural translations"", ... ],

      // The plural form string is converted into a function
      // and the value that's passed into the gettext call
      // is passed into the plural form function. It's result
      // (plus 1) is the index that the translation comes from.

      // We're using sprintf interpolation on our keys so we can
      // then sub in the _actual_ values into the result.
      ""%d key"" : [ null, ""%d key"", ""%d keys"" ],

      // Contexts are keys that are just prefixed with a context string
      // with a unicode \u0004 as the delimiter.
      // You can use it for anything. Usually it's just for being content aware
      // in some way (e.g. male vs. female, product vs. category)
      ""context\u0004%d key"": [ null, ""context %d key"", ""context %d keys"" ]
    }
}}}

This looks like an appropriate python library to do the formatting. http://pypi.python.org/pypi/pojson/ but I'll leave to your discretion.

Finally an API endpoint that returns the appropriate request would save us making requests to the i18n directory.

{{{
/api/i18n/{locale}
}}}

For example if the current url is http://localhost:5000/pl/dataset then the client would request:

{{{
http://localhost:5000/api/i18n/pl
}}}

This should return the Content-Type: application/json and a 200 with an appropriate JSON body (this can be the language requested or an appropriate fallback). I think that if the fallback is the default language (in most cases English) then we just return a 204 No Content to save data. "	1344248847000000	1344269446000000
2803	enhancement	aron.carroll	aron.carroll	demo phase 3	closed	fixed	Convert all JS translations to use keywords		1344251817000000	1344256274000000
2804	enhancement	aron.carroll	shevski	demo phase 3	closed	fixed	please make download button blue	"on resource pages (e.g. http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360), same as old ckan
for continuity and to highlight!"	1344252255000000	1344269865000000
2807	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	autocomplete.js error	"@aron,


edit dataset has a js error


http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada


TypeError: options is undefined
[Break On This Error] 	

this.lookup(options.term, options.callback);

autocomplete.js (line 231)"	1344266228000000	1344270190000000
2808	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	Confirm delete dialog badly placed on small screen	"If you make your browser window short - say half screen height on a page with delete eg http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada


the dialog box is partially off the screen - it should try to stay on screen"	1344333474000000	1345023693000000
2809	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	_ not defined for modules	"eg on confirm delete


http://localhost:5000/zh_TW/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada


_(....) gives [object Object] not translation"	1344335863000000	1344347814000000
2837	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	checkbox styling	Is this done I know checkboxes are a pain but it looks a bit out of place compared to the other items.	1344605993000000	1344851271000000
2840	enhancement	aron.carroll	toby	demo phase 3	closed	fixed	tag line styling	"@aron

Is it possible to get the tagline text to float  the other way

at the moment we have

...........Title
Tag line to here

I'd like 

...........Title
...........Tag line to here

is this possible or too much pain


based partly on this comment
Markw wrote 


A minor problem visible on the home page
<http://s031.okserver.org:2375/> and other pages, when viewing in both
Chrome and Firefox:

if the browser width is not very wide, the tag line 'Open source data
portal' vanishes off the *left* hand side of the screen. For some
reason the browser doesn't even recognise this with a horizontal
scroll bar (as it does when stuff is off the *right* hand side) - it
just chops off the text."	1344851307000000	1344852538000000
2841	enhancement	aron.carroll	amercader	demo phase 3	closed	fixed	Slug generation fails on demo dev	"On the add dataset form, slug generation fails with JS error:

{{{
Uncaught TypeError: Object [object Object] has no method 'incompleteFormWarning'    basic-form.js:5
}}}
"	1344855653000000	1344857415000000
1142	enhancement	annapowellsmith	rgrp	ckan-v1.5	closed	fixed	[super] Major Overhaul and Extension of CKAN Documentation	"Child tickets:

 * #1041 Start Using the CKAN Wiki for Tutorial-style documentation
 * #1192 Convert CKAN Sphinx docs into admin/reference manual


Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927

 * CKAN 1-page overview (for enterprise and for data hackers)
 * Administrator's Guide (including install)
 * Extensions Guide
 * Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: http://wiki.ckan.net/Documentation_Plans

== Minor Items ==

"	1305727452000000	1313775665000000
1192	task	annapowellsmith	annapowellsmith	ckan-sprint-2011-10-28	closed	fixed	Convert CKAN Sphinx docs into admin/reference manual	"As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

 * An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install. 
 * A Reference Manual which is the primary source of reference for CKAN software - this includes API docs. 

The current chapters of the Sphinx docs should be moved as follows:

 * index.rst - copy some stuff from README.txt, keep short
 * README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst 
 * [NEW] install.rst - new file on installation
 * [NEW] theming.rst - move over from wiki
 * api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
 * i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
 * design.rst - REMOVE (can copy some over if you can be bothered to http://wiki.ckan.net/Vision (all philosophical stuff should go!))
 * loaders.rst: move to wiki.ckan.net/Using_Loaders
 * feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
 * importer.rst: remove (don't even copy)
 * forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) http://wiki.ckan.net/Customizing_Forms
 * forms-integration.rst: remove
 * model.rst: leave but move to reference (only for core devs)
 * load-testing.rst -> move to http://wiki.ckan.net/Load_Testing
 * distributed.rst -> remove
 * admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
 * deb.html -> go into reference (query james gardner on list about moving this to wiki?)"	1308324593000000	1312191462000000
728	requirement	amercader	johnbywater	ckan-backlog	assigned		CSW Harvesting shall be optimised in respect of reharvesting only records that have changed	Hi Will, this is important again because some CSW servers we use have over 300 documents in. Could you take a look at modifying the filter please?	1287675340000000	1310124784000000
736	requirement	amercader	johnbywater	ckan-v1.4-sprint-7	closed	fixed	URL validation for CSW and WMS links	"Robust against slightly incorrect URLs (e.g. with trailing slash, or without /csw), and non-CWS service which also return <ows:ExceptionReport ... />.

Both cases would be resolved by checking capabilities (with !CswClient's  check_get_capabilities() method)."	1287749416000000	1304963598000000
794	requirement	amercader	johnbywater	ckan-backlog	assigned		Investigate reconciling UKLP Publisher and Provider with DGU	"This needs more analysis, but the GEMINI2 attribute ""metadata point of contact"" must be reconciled with the registered publisher (or agent).

This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider."	1289227811000000	1311179581000000
1030	defect	amercader	thejimmyg		closed	fixed	Move harvesting out of the rest API		1299776418000000	1303117978000000
1037	defect	amercader	thejimmyg	ckan-v1.4-sprint-6	closed	fixed	More Robust Harvesting for DGU	"CKAN's harvesting facility is now live on DGU but there are some major improvements that could be made to make it more robust and better fit the generic CKAN harvesting framework proposed in #987.

Some of the key issues:

 * Error reports do not currently contain the ID or title of the document with the error.
 * We only have ""added"" and ""error"" logging on jobs when we really need a report of ""added"", ""updated"", ""not changed"" and ""errors"" with the items in each referencing a real metadata document for which harvesting was attempted
 * We need deletion and editing of sources, without deleting the harvested documents or packages
 * We need a more robust harvesting mechanism than a cron job or we need to deal with the case of multiple cron jobs running at once.
 * We need to know the last time a list of documents was scheduled for harvest and the last time each one was fetched.
"	1300197602000000	1304937601000000
1152	enhancement	amercader	amercader	ckan-backlog	new		True support for generic CSW servers	"The CSW harvesters implemented at the moment were developed with the DGU
project in mind, and they assume all remote CSW servers to implement the
Gemini 2 specification. Gemini 2 is the profile defined in the UK for
INSPIRE complying metadata, so obviously catalogs from other countries
or non-INSPIRE complying ones won't be able to be harvested.

The changes needed to support generic CSW servers (i.e. those
implementing the ISO 19139 profile) are:

* Handling the validators (right now are hardcoded in the harvester
code). This probably involves issues discussed in the CREP 3
(ticket #1134)

* Changes in the model to adapt the specification to ISO 19139

* Renaming objects and classes which are now Gemini-centric


List of CSW servers tested:

https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&hl=en_US&authkey=CNu4hsEB#gid=0"	1306141334000000	1313411822000000
1160	enhancement	amercader	amercader	pdeu-1	closed	wontfix	Havest CSW servers on PDEU	"It could be interesting for the publicdata.eu project to add some CSW datasources. Thanks to the INSPIRE directive, there are quite a few national catalogs of geodata that could be worth harvesting.
Work on this should include:
* Compose Endpoint List (1d)
* Harvest on staging system and evaluate which sources have useful material (2d)

First point is being performed in this GDoc:
https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&authkey=CNu4hsEB&hl=en_US#gid=0"	1306407271000000	1309804038000000
1162	enhancement	amercader	amercader	pdeu-1	closed	fixed	Define a common extra for storing the geographic extent	"We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city).

Relevant discussion:
http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html

"	1306408704000000	1308647187000000
1164	enhancement	amercader	amercader	pdeu-1	closed	fixed	Cloropleth Map of European Data Availability for PDEU	A nice map in the homepage showing the availability of data across Europe	1306408824000000	1308647224000000
1166	enhancement	amercader	amercader	ckan-sprint-2011-10-10	closed	fixed	Less stringent filters for displaying the WMS preview	"Right now, for the ""View available WMS layers"" link to be displayed, the package needs to have an ""INSPIRE=True"" extra, a ""resource-type=service"" extra and a resource of type ""WMS"".

This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough.


(The repository for this is ckanext-spatial, but it's not on the list)"	1306422646000000	1317381932000000
1173	enhancement	amercader	amercader	pdeu-1	closed	fixed	Offer a DCat representation for packages in the API	/api/rest/package/foo.rdf should return a DCat representation of the package. To create it, we will use the functions in ckanext-rdf.	1306939639000000	1307615271000000
1253	enhancement	amercader	amercader	ckan-sprint-2011-10-28	closed	fixed	Authz refactor	We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.	1312199491000000	1313401367000000
1301	defect	amercader	amercader	ckan-sprint-2011-09-12	closed	fixed	Give access to extensions from the old formalchemy controllers	This was removed during the auth refactoring, but it's needed in the old controllers	1314882888000000	1314893744000000
1303	defect	amercader	amercader	ckan-sprint-2011-09-12	closed	fixed	User can not login again if he has edited his profile	"Only happens if the user does not update the password.
Most probably introduced during #1229."	1314889472000000	1314978842000000
1304	enhancement	amercader	amercader	ckan-sprint-2011-09-12	closed	fixed	Show more descriptive error messages when sending emails	Sending emails is failing in some cases, and the error message just says 'Could not send reset link'.	1314893206000000	1314893749000000
1333	enhancement	amercader	amercader	ckan-sprint-2011-10-28	closed	fixed	Move form converters to ckan core	Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.	1315909268000000	1316951285000000
1334	defect	amercader	amercader	ckan-sprint-2011-10-28	closed	fixed	"""AttributeError: 'NoneType' object has no attribute 'name'"" exception when logging in"	"On same cases this exception is thrown when logging in:

 {{{
File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'
}}}

"	1315909508000000	1319798961000000
1337	defect	amercader	amercader	ckan-sprint-2011-10-28	closed	fixed	Group extras are JSON endoded in the REST API responses	"Instead of this:
{{{
extras: {
type: """"Primary source""""
license_id: """"localauth-withrights""""
publisher_frequency: """"2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""""
}
}}}

we should return this:

{{{
extras: {
type: ""Primary source"",
license_id: ""localauth-withrights"",
publisher_frequency: ""2Preño € $ àdrià"",
}
}}}"	1315924727000000	1316951259000000
1338	defect	amercader	amercader	ckan-sprint-2011-10-28	closed	fixed	Exception when trying to use a custom form schema from an extension	"Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

{{{
File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'
}}}"	1315925217000000	1317117738000000
1354	enhancement	amercader	amercader	ckan-v1.5	closed	wontfix	Pass the context object to plugins implementing the IPackageController and IGroupController	"Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).
"	1316955680000000	1316970374000000
1375	enhancement	amercader	amercader	ckan-sprint-2011-10-10	closed	fixed	Define new package id before sending it to the IPackageController plugins	"When creating a package, the package_create logic function sends the newly created package object to all plugins implementing the IPackageController interface

https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64

At this point, the changes have not yet been commited and the id of the package is not set (It has a default value of make_uuid, which is set when commited). For this reason, plugins have no way of knowing the package id. E.g. the spatial query plugin needs to know this id to link the package extent to the actual package.

One possible solution would be to set the package id in the data dict received by package_create (if it's not defined):

{{{
from ckan.model.types import make_uuid
if not 'id' in data or not data['id']:
   data['id'] = make_uuid()
}}}"	1317910441000000	1318260008000000
1381	enhancement	amercader	pudo	ckan-sprint-2011-10-24	closed	fixed	Managing groups via package entity API	"It is not currently possible to assign or remove groups by adding or removing their group name from the ""groups"" list on the REST API. This should be allowed, as it is easier than first adding a package and then editing each group it must belong to. 
"	1318266706000000	1324037815000000
1456	enhancement	amercader	amercader	ckan-sprint-2011-11-21	closed	fixed	Use resource description instead of name if both are present	If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those.	1320862619000000	1324472178000000
1462	defect	amercader	thejimmyg	ckan-sprint-2011-12-19	closed	fixed	Multi-Instance CKAN Solr Search Issues	"With the 1.5 package release, CKAN instances over-write each other's indexes. Adria has a fix.

I'd like to see a new CKAN 1.5.1 release with the Solr changes merged and the Solr site ID set up as part of the install ckan.site_id = dgu-os.okfn.org. I'd also like the ability to search on ID and to have the ID directly visible on the package page.

We need to decide how to do this upgrade without affecting existing CKAN solr indexes."	1321375470000000	1330083671000000
1469	enhancement	amercader	thejimmyg	ckan-sprint-2012-03-05	closed	fixed	Geo-spatial search on the front end	Now that we have a geospatial search API, we should build a front-end that integrates with the existing search for displaying results.	1321376324000000	1330102878000000
1490	enhancement	amercader	amercader	ckan-sprint-2011-12-05	closed	fixed	Standardize output from package listings coming from the logic layer	"Right now, the two logic functions that return a list of packages (package_search [1] and group_packages_list [2])use custom functions to generate the output dict. That's suboptimal because:
- The template function that renders the package listing (package_list_from_dict) could end up receiving different dicts.
- Not all package properties are available at the template level (e.g extras, tags...). These can be really useful to customize the listing from an extension.

In general only the functions present in lib/dictization/model_dictize.py should be used to build the output of a logic function, in that case package_dictize. If necessary, they can be modified to include missing properties, like on this particular case the ""isopen"" property, needed by the template renderer.




[1] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L685
[2] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L442"	1322225239000000	1324033557000000
1498	enhancement	amercader	amercader	ckan-sprint-2011-12-05	closed	fixed	Support multiple SOLR schema versions	"At some point the SOLR schema used by CKAN will need to be changed and the changes won't be always backward compatible, so we need a mechanism to support instances that use older versions of the schema.
This process will involve:

 * Better documentation on how to deploy SOLR, as single instance or multicore
 * A convention on how to tag schema versions and SOLR endpoints to ensure that CKAN instances point to the correct SOLR URL.
 * (If possible) Checks in CKAN core that deal with different schema versions."	1322585264000000	1323165876000000
1516	task	amercader	amercader	ckan-sprint-2011-12-19	closed	fixed	Update SOLR schema after 1.5.1 release and set up multicore SOLR instances if needed	"CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like:

{{{http://<solr-server>/solr/schema-<version>}}}

Different scenarios:

 * Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine):
  * Update CKAN
  * Update SOLR schema (symlink to suitable version in CKAN source) 
  * Rebuild search index

 * Single SOLR instance used by multiple CKAN sites.
  * Update CKAN source
  * Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source)
  * Update solr_url on each of the CKAN sites with the suitable core
  * Rebuild search index


These are the SOLR and CKAN instances that need to be updated (Please add any missing ones):

 * Same machine as the CKAN site:
  * ~~test.ckan.net~~
  * ~~iati.test.ckan.net~~
  * data.gov.uk/ catalogue.data.gov.uk (confirm)

 * ~~s046.okserver.org~~
  * ~~iatiregistry.org~~

 * ~~s052.okserver.org~~
  * ~~datacatalogs.org~~

 * ~~s004.okserver.org / eu4.okfn.org / solr.okfn.org~~
  * thedatahub.org (=www.ckan.net)
  * datagm.org.uk
  * ~~publicdata.eu~~
  * hri.fi (dev.fvh.fi?)
  * it.ckan.net
  * ie.ckan.net
  * cz.ckan.net
  * register.data.overheid.nl (nl.ckan.net?)
  * no.ckan.net / datakilder.no
  * br.ckan.net
  * colorado.ckan.net
  * at.ckan.net


TODO: which SOLR server are using these instances?
 * data.norge.no
 * nederland.ckan.net
 * lt.ckan.net
 * pl.ckan.net
 * datadotmontreal.ca/
 * ca.ckan.net / datadotgc.com

"	1323108431000000	1323362689000000
1540	defect	amercader	amercader	ckan-sprint-2012-01-09	closed	fixed	Search API returns an error if empty parameters are provided	"Both in 1.5.1b:

http://thedatahub.org/api/search/dataset?groups=lodcloud&title=

and 1.5.2a (current master):

http://test.ckan.net/api/search/dataset?groups=lodcloud&title=

Although the error message in 1.5.2a is more verbose:


""Bad request - Bad search option: HTTP code=400, reason=org.apache.lucene.queryParser.ParseException: Cannot parse 'groups:lodcloud title:': Encountered \""<EOF>\"" at line 1, column 22. Was expecting one of:     \""(\"" ...     \""*\"" ...     <QUOTED> ...     <TERM> ...     <PREFIXTERM> ...     <WILDTERM> ...     \""[\"" ...     \""{\"" ...     <NUMBER> ...""

Some parameter validation before sending it to Solr should do the trick"	1323359388000000	1326060385000000
1547	enhancement	amercader	amercader	ckan-sprint-2012-01-23	closed	fixed	Add search methods (before and after) to the IPackageController interface	"Time estimate: 1d

Add a two search method to the IPackageController interface so extensions can:

 * Modify the query being sent to Solr 

 {{{

def before_search(search_params):

    return search_params

}}}

 * Modify the results returned (`after_search`)

 {{{

def after_search(search_results, search_params):

    return search_results

}}}


Please see this pad for implementation details:

http://ckan.okfnpad.org/search"	1323778996000000	1330084379000000
1590	enhancement	amercader	amercader		closed	fixed	Create customized feeds for the IATI Registry	"We need a way to track changes on the registry (datasets edited or updated), globally and on a per country/publisher/etc. basis. RSS and Atom feeds are really popular, and after closing #191 and #1498 crating them from the search results should be fairly easy.

The following URLs are pretty self-explanatory:

{{{
http://localhost:5000/feed/registry.rss

http://localhost:5000/feed/country/AF.rss
http://localhost:5000/feed/publisher/worldbank.rss
http://localhost:5000/feed/organisation_type/10.rss
}}}

As we need to implement custom wrappers for countries, publishers, etc, we might as well offer a fully customizable feed, e.g.:
{{{
http://localhost:5000/feed/custom.rss?q=activity_count:[* TO 100]
http://localhost:5000/feed/custom.rss?publishertype=primary_source
}}}

Apart from the actual feeds, there will be a small amount of work at  the template level to add the links to the suitable pages (and maybe a generic page showing all available feeds)
"	1324486965000000	1326711608000000
1592	enhancement	amercader	amercader	ckan-sprint-2012-01-09	closed	fixed	Add metadata_modified and metadata_created to package_dictize output	The dict returned by package_dictize does not include metadata_modified and metadata_created. These are really useful properties, so it's worth having them on the standard package dict representation, which is used in several places, like at the template level.	1324488909000000	1326304321000000
1601	enhancement	amercader	rgrp	ckan-v1.7	closed	fixed	[super] Well integrated Geo-spatial support including search	"Need thoughts from Adria and James. RP thoughts

 * Integration into WUI search: #1469
 * JS Editor for adding/editing spatial attribute: #1818
 * Theme/template hook for adding the map e.g. on dataset page (or elsewhere)
 * Clear Documentation
 * Deployment
 * Announce post
 * (?) Map page for stats section showing location of datasets

Extras (probably separate):

 * csw support ..."	1325259821000000	1338204029000000
1616	defect	amercader	amercader	ckan-sprint-2012-04-02	closed	fixed	Catch exceptions when rebuilding the search index	Right now if an exception is found while reindexing, the whole process stops and the remaining datasets are left out of the index. The process should continue after logging the exception. If more than a certain number of exceptions occur in a row, the process should stop.	1325844669000000	1332327635000000
1640	enhancement	amercader	dread		closed	fixed	Setup publicdata.eu harvester for Serbian CKAN datasets	Set-up publicdata.eu to harvest datasets at rs.ckan.net (Serbian community CKAN).	1326370425000000	1327340939000000
1641	enhancement	amercader	amercader	ckan-sprint-2012-01-23	closed	fixed	ckanext-archiver: Content-length header not reliable to check if resource has been modified	"The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the ""Content-Type"" and ""Content-Length"" headers differ from the values stored to see if the resource needs to be updated [1].

The ""Content-Length"" header, although widely used, is not mandatory and some servers don't provide it, e.g.:

{{{
$ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG
HTTP/1.1 200 OK
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: text/xml
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly
X-AspNetMvc-Version: 3.0
content-disposition: attachment; filename=AFG_IATI_12012012.xml
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Jan 2012 12:36:43 GMT
}}}

Also worth noting that [http://docs.python-requests.org/ requests], the python library that uses ckanext-archiver, sets an ""Accept-Encoding: gzip"" header by default, which depending on the configuration of the remote web server, may prevent the ""Content-Length"" server from being sent, e.g.:

{{{
$ curl -H ""Accept-Encoding: gzip"" -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 12:12:46 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: application/xml

curl -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 11:56:23 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Content-Length: 2686720
Vary: Accept-Encoding
Content-Type: application/xml
}}}

All this can lead to some resources never getting updated, and of course the size property of the resource not being set.

As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it).



[1] https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72"	1326376420000000	1326376777000000
1654	task	amercader	amercader	ckan-v1.7	closed	fixed	[super] Update Publicdata.eu to the latest CKAN stable version	"Tasks include:

 * #1813 Update ckanext-pdeu (4d)
 * #1814 Update harvesters (2d)
 * #1649 Verify ckanext-rdf works with latest CKAN (3d)
 * #1815 Reenable Sparql endpoint (?)
 * #1816 Update ckanext-apps (2-3d)

"	1326710647000000	1338203166000000
1655	task	amercader	amercader	ckan-sprint-2012-01-23	closed	fixed	Setup issues on s025 (Publicdata.eu)	"Time estimate: 2d

 * Fix logs (apache, ckan, harvest): rotate, set suitable levels
 * Fix harvesting jobs: supervisord for gather consumer, cron job
 * Fix backups

Also it may be worth setting up a test instance ( on s023 ?)"	1326710844000000	1327312857000000
1678	enhancement	amercader	amercader	ckan-v1.8	closed	fixed	Make synchronous search the default behaviour	"Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default.
We could keep a config flag to deactivate it, but synchronous search should be the default behaviour."	1326807604000000	1341405749000000
1701	enhancement	amercader	kindly	ckan-sprint-2012-02-06	closed	fixed	Normalize character encoding for ckan search.	"Make sure accented characters are normalized when indexed and when searched for.

"	1327419050000000	1330085360000000
1702	enhancement	amercader	kindly	ckan-sprint-2012-02-06	closed	duplicate	Normalize character encoding for ckan search.	"Make sure accented characters are normalized when indexed and when searched for.

"	1327419369000000	1327419922000000
1726	enhancement	amercader	amercader	ckan-sprint-2012-02-06	closed	fixed	Update harvesting model	"We need:
 * ""current"" column in harvest_object
 * ""harvest_source_id"" column in harvest_object, foreign key to harvest_source
 * ""title"" column in harvest_source

The extension should check on startup if the db needs updating and run the necessary scripts."	1328007127000000	1328523253000000
1727	enhancement	amercader	amercader	ckan-sprint-2012-02-06	closed	fixed	"Add ""current"" property to harvest objects"	"Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors.
A new ""current"" field will be added which will identify unequivocally the most recent harvested object.
Unfortunately this will involve some major changes:
 * Model update (See #1726)
 * Update queries for getting harvest sources' datasets
 * Update ckanext-csw queries for getting the objects
 * Update inspire and ckan harvesters
 * Document new field"	1328007254000000	1328523641000000
1728	enhancement	amercader	amercader	ckan-sprint-2012-02-20	closed	fixed	Update UKLP harvesters	"Three small changes:

 * Set Resource type to None instead of 'Undefined'
 * Set extra 'UKLP' = True instead of 'INSPIRE' = True

These two changes will require manual update of the database.

 * Allow datasets to be updated if the old source is inactive, even if the metadata date hasn't changed:

{{{
    if last_harvested_object.metadata_modified_date is None \
       or last_harvested_object.metadata_modified_date < self.obj.metadata_modified_date \
       or self.force_import \
       or last_harvested_object.metadata_modified_date == self.obj.metadata_modified_date and last_harvested_object.source.active is False:

}}}"	1328007610000000	1329498181000000
1731	enhancement	amercader	amercader	ckan-sprint-2012-03-19	closed	fixed	Publisher support for harvesting	"Estimate: 4-5d

Only sysadmins can manage harvest sources right now, we need to allow publishers to handle their own sources.

We need to decide if there will be two different profiles on ckanext-harvest, the current one with only sysadmins allowed to manage the harvesting and a new one supporting publishers (ckan core can be configured to not use the publisher model)

Tasks include:

 * Auth checks on the different controller actions (and the command line interface?)
 * Modify the index page to just show sources the logged user has permissions on.
(Talk with Ross about the specific functions to use)
 * Create/edit form to show a dropdown with the user available publishers.
"	1328007919000000	1332152540000000
1733	enhancement	amercader	amercader	ckan-sprint-2012-02-20	closed	fixed	Delete (inactivate) harvest sources	"Right now harvest sources can be inactivated (only via the cli), but we need to make sure:

 * Expose this on the web interface (only for sysadmins, see #1731)
 * Inactive sources are not shown in the web interface

Also for deleted packages:

 a) Deleted packages get state=""deleted"" (not actually purged) so they are hidden on front end
 ACTION: need a button on the front end for ""Delete"" for sysamdins
Done in #1425. Not sure if the button is only shown to sysadmins.

 b) No undelete: If you reharvest a deleted package, it remains deleted
   * Publisher should create a new record with new identifier and same/different content as applicable.
   * If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated.
   * You won't get the state where a package is updated in its deleted state
   * If they change the ID and reharvest, a new record appears on DGU.
 c) Update CSW interface so that export doesn't export deleted packages
 d) Check geospatial search doesn't search deleted pacakges
 e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API.

"	1328008287000000	1329754689000000
1734	enhancement	amercader	amercader	ckan-v1.7	closed	fixed	[super] Updates on harvesting for UKLP	"Some changes are needed in ckanext-harvest to support the new piece of work in the UKLP harvesters, mainly related to the deletion of existing sources.

 * #1726 Update harvesting model
 * #1727 Add ""current"" field to harvesting objects
 * #1728 Update UKLP harvesters
 * #1733 Delete (inactivate) harvest sources

At some point we also want to support a publisher model for the harvesting:
 * #1731 Publisher support for harvesting"	1328008474000000	1332167315000000
1813	task	amercader	amercader	ckan-sprint-2012-03-05	closed	fixed	Upadate ckanext-pdeu to work with CKAN 1.6	"Tasks include:

Estimate 4d

 * Update requirements (git)
 * Search facets error
 * Error on dataset read (extras in context)
 * Url generator for rdf
 * Make templates work with latest theme
 * Use dicts in templates (isopen())"	1329757128000000	1330102557000000
1814	enhancement	amercader	amercader	ckan-sprint-2012-03-19	closed	fixed	Check publicadata.eu harvesters	"Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones."	1329757408000000	1332152596000000
1815	task	amercader	amercader	ckan-future	new		Reenable Sparql endpoint on publicdata.eu	"Estimate: ?

The 4store Sparql endpoint has been down for a long time.
It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset."	1329757723000000	1338204641000000
1816	enhancement	amercader	amercader	ckan-sprint-2012-03-19	closed	fixed	Update ckanext-apps to work with CKAN 1.5.1 - 1.6	"Estimate [2-3d]

If possible, it should work with both 1.5.1 and 1.6.

Needed for PDEU, also at some point for IATI"	1329757864000000	1331302835000000
1818	enhancement	amercader	amercader	ckan-future	new		Spatial metadata editor	"Estimate 2.5d

Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset.

We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers [1], [2].

Things to take into account:
 * Integration with the form sections.
 * Allow some degree of configuration (default extent, limit geometry types...)


[1] http://openlayers.org/dev/examples/draw-feature.html
[2] http://openlayers.org/dev/examples/modify-feature.html
"	1329763941000000	1338203195000000
2231	defect	amercader	amercader	ckan-sprint-2012-04-02	closed	fixed	Base harvester broken on 1.6.1	"On its current version, ckanext-harvest passed a custom schema to package_create and package_update to allow defining an id for new packages (the same as in remote ckan instances) [1]

Recent changes in the logic layer have made that the context object is no longer directly read from the context, but only get from plugins [2]. This makes the schema sent from ckanext-harvest to be ignored and the harvesting fails because the id field is not expected

{{{
ValidationError: {'Id': u'The input field id was not expected.'}
}}}

We need a way to either:

 1. Be more flexible accepting schemas on the logic functions
 2. Modify the harvester plugin to implement the necessary interface


[1] https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115

[2] https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38"	1331815499000000	1333374735000000
2232	defect	amercader	amercader	ckan-sprint-2012-04-02	closed	fixed	Unicode Exception when rebuilding the search index	"In some cases this exception is fired when building the search index. As explained in #1616 this makes the whole process stop.

{{{

  File ""/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py"", line 326, in wrapper
    return self._update(content, query)
  File ""/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py"", line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File ""/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py"", line 639, in _post
    return check_response_status(self.conn.getresponse())
  File ""/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py"", line 1096, in check_response_status
    raise ex
SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
Traceback (most recent call last):
  File ""/var/lib/ckan/pdeu/pyenv/bin/paster"", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File ""/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py"", line 84, in run
    invoke(command, command_name, options, args[1:])
  File ""/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py"", line 123, in invoke
    exit_code = runner.run(args)
  File ""/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py"", line 218, in run
    result = self.command()
  File ""/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py"", line 298, in command
    rebuild()
  File ""/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py"", line 134, in rebuild
    {'id': pkg.id}
  File ""/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py"", line 54, in insert_dict
    return self.update_dict(data)
  File ""/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py"", line 79, in update_dict
    self.index_package(pkg_dict)
  File ""/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py"", line 153, in index_package
    raise SearchIndexError(e)
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
}}}"	1331816582000000	1332331660000000
2252	defect	amercader	amercader	ckan-sprint-2012-04-16	closed	fixed	UKLP harvesters returning errors when importing due to schema changes	"See trace below.


As in #2231, the schema passed with the context is no longer used on the logic layer (in this case the one used on the DGU form is used). This can be fixed allowing a schema to be passed with the context.

{{{
  File ""/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py"", line 139, in import_stage
    self.import_gemini_object(harvest_object.content)
  File ""/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py"", line 163, in import_gemini_object
    package = self.write_package_from_gemini_string(unicode_gemini_string)
  File ""/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py"", line 386, in write_package_from_gemin
    package = self._create_package_from_data(package_dict, package = package)
  File ""/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py"", line 486, in _create_package_from_dat
    package_dict = action_function(context, package_dict)
  File ""/usr/lib/pymodules/python2.6/ckan/logic/action/update.py"", line 169, in package_update
    data, errors = validate(data_dict, schema, context)
  File ""/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py"", line 216, in validate
    converted_data, errors = _validate(flattened, schema, context)
  File ""/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py"", line 289, in _validate
    convert(converter, key, converted_data, errors, context)
  File ""/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py"", line 179, in convert
    converter(key, converted_data, errors, context)
  File ""/var/lib/ckan/std/pyenv/src/ckanext-dgu/ckanext/dgu/validators.py"", line 81, in validate_license
    if data[('license_id',)]== '__extra__':
KeyError: ('access_constraints',)
2012-03-19 10:47:41,736 ERROR [ckanext.inspire.harvesters] Error importing Gemini document: ('access_constraints',)
}}}"	1332333907000000	1334567502000000
2263	enhancement	amercader	amercader		closed	fixed	"IATI: Get ""Record updated"" field from the registry"	Don't provide a field in the form or spreadsheet import for this, just use dataset.metadata_modified.	1332766145000000	1332846807000000
2271	defect	amercader	amercader	ckan-sprint-2012-04-16	closed	fixed	IATI: Some datasets not being indexed	"When importing a CSV file for a certain publisher, records don't come up on the search index. 
After some investigation this has tunrn out to be caused by the records being deleted. We need to investigate if this was caused by the import or just made by the users.
"	1333375469000000	1334567582000000
2332	enhancement	amercader	rgrp	ckan-v1.7	closed	fixed	Fixes for v1.7 release	"A place to list crucial fixes for v1.7 release:

Related extension:

All not complete now moved to #2347

 * ~~Related Item has comments on index page (they should not show up here -- why is this happening!)~~
 * ~~Related item should be to left of history (history could in fact move to RHS)~~
   * What about losing icons since we duplicate for related item
 * Related item plus sign is very poorly aligned within box
  * Add related items should be in drop down like for resources (?) to make it easy to add items
  * Hover should show a title that gives some information about what related are (not at all clear)
 * Related item only includes App, Idea, Visualization but not API  | Post | Paper | News Article options as in original ticket #2204

Search results:

 * #2331 - search should add not or
 * ~~Search results should be similar across group pages and normal search results (suggest adopting group page style)~~
  * Search results resource link should link to resource page rather than url for that resource
  * Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?)

Data viewer:

 * ~~Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07~~
  * Ensure still using read-only (this seems to have been lost in upgrade)
 * Do not show failed preview with red-bar (esp when just empty datastore ...)
    a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy
    b) DataStore enabled is off by default and only turned on either explicitly by user or by DataStorer on successful store ...
 * (? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state)
  * Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...)
  * Do #2322 ...
 * Decent powered by icon for the Embed for the DataHub (make this text customizable and allow html ...)
 * Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
 * Data viewer embed url includes all the resource info

Dataset page:

 * Put home page link somewhere more prominent (e.g. the top of the sidebar)
 * Put author more prominent (at top right)
 * (? - new work) List CKAN owners at top right in sidebar
 * Make resources stand out more
 * Rename resources tab and heading to Data & Resources

Dataset edit and create:

 * Corrections to dataset creation form for v1.7 - #2317
 * Better auto-complete for groups on dataset edit/new page - #2319
  * Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?)
 * Description / notes text area is not wide enough.
   * Should also be larger on edit page
 * Problem with date instructions on add/edit resource: ""Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d.""
 * Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either
   * A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar) 
   * Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like ""checking url and gathering url info ...)

Resource view:
 * Do not show no description in no description (just show dataset info)

Theme:
 * ~~Fix spacing above page title (i.e. dataset title etc)~~
 * ~~Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer~~
 * ~~Fix subnav width on resources pages (does not extend all the way to the right like it should)~~
 * ~~Fix spacing of top menu and search box (do not look right)~~
 * Group pages:
  * view page: image / logo is really poorly locate and sized, see e.g. http://datahub.io/group/wikimedia
  * ~~view page: admin heading on RHS sidebar is wrongly indented~~

Miscellaneous:

 * Ensure that we out-of-the-box config for CKAN has (#2388):
  * Only logged in users able to create
  * Creation permissions have only creators able to update a dataset

Deployments (without deployment we cannot know these are working):

 * ~~Social extension~~
 * ~~Analytics~~"	1335644116000000	1340033281000000
2358	task	amercader	amercader	ckan-sprint-2012-05-15	closed	fixed	Finish off Release 1.7	"Various stuff related with release 1.7:

 * Review patches sent to the release branch
 * Regularly update TDH with latest changes
 * write changelog, announce email, etc
 * ...

"	1335887030000000	1336641577000000
2359	task	amercader	amercader	ckan-sprint-2012-05-15	closed	fixed	Investigate pycsw integration	"Tom Kralidis kindly told us on IRC [1] that the latest trunk of pycsw includes support for wsgi and different backends, which sounds like could be really useful for CKAN integration.



[1] http://irclogs.okfn.org/ckan/2012-04-25.txt 17:16"	1335887299000000	1337016578000000
2360	enhancement	amercader	amercader	ckan-sprint-2012-05-15	closed	fixed	Finish INSPIRE support proposal	I've been writing a doc with different issues regarding INSPIRE support in CKAN. It is mostly done, except for some sections on CSW support. Depends on #2359	1335887409000000	1337016284000000
2361	enhancement	amercader	amercader	ckan-sprint-2012-05-15	closed	fixed	Install ckanext-googleanalytics and ckanext-stats on PDEU	ditto	1335887476000000	1337016768000000
2382	task	amercader	amercader	ckan-future	new		Investigate options for basic geocoding	A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)	1337017160000000	1338205325000000
2388	enhancement	amercader	amercader	ckan-v1.8	closed	fixed	"Update ""out of the box"" settings for CKAN"	"
 * Only logged in users able to create
 * Creation permissions have only creators able to update a dataset
 * Synchronous search is enabled (#1678)
"	1337095269000000	1342104475000000
2389	defect	amercader	amercader	ckan-v1.8	closed	fixed	More robust handling of DataExplorer errors	"US-0a

Do not show failed preview with red-bar (esp when just empty datastore ...)
 * Data Viewer checks if datastore is empty nad falls back to ther options if emtpy 
 * DataStore enabled is off by default and only turned on either explicitly by user or by DataStorer on successful store ...

Loading/showing error reports
"	1337095447000000	1341222201000000
2399	defect	amercader	seanh	ckan-v1.8	closed	fixed	Model cleanup in CKAN core has broken ckanext-spatial	"root@lucid32:/usr/local/feature-2375-demo-theme/pyenv/src/ckan# sudo -u www-data /usr/local/feature-2375-demo-theme/pyenv/bin/paster --plugin=ckanext-spatial spatial initdb 4326 --config=/usr/local/feature-2375-demo-theme/pyenv/src/ckan/development.ini 
Traceback (most recent call last):
  File ""/usr/local/feature-2375-demo-theme/pyenv/bin/paster"", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File ""/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py"", line 84, in run
    invoke(command, command_name, options, args[1:])
  File ""/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py"", line 123, in invoke
    exit_code = runner.run(args)
  File ""/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py"", line 218, in run
    result = self.command()
  File ""/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/commands/spatial.py"", line 38, in command
    self._load_config()
  File ""/usr/local/feature-2375-demo-theme/pyenv/src/ckan/ckan/lib/cli.py"", line 53, in _load_config
    load_environment(conf.global_conf, conf.local_conf)
  File ""/usr/local/feature-2375-demo-theme/pyenv/src/ckan/ckan/config/environment.py"", line 204, in load_environment
    plugin.configure(config)
  File ""/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/plugin.py"", line 60, in configure
    setup_model()
  File ""/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/model.py"", line 20, in setup
    define_spatial_tables(srid)
  File ""/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/model.py"", line 66, in define_spatial_tables
    package_extent_table = Table('package_extent', metadata,
NameError: global name 'metadata' is not defined"	1337173470000000	1340706702000000
2407	enhancement	amercader	markw	ckan-v1.9	new		'Access denied' message is unhelpful	"This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied':

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html

The problem was he didn't have access to the group. A message like 'You don't have access to the group ""LODcloud""' would have been clearer."	1337612500000000	1340633440000000
2467	defect	amercader	amercader	ckan-v1.8	closed	fixed	Fix stats extension frontend	After moving to Bootstrap the tables on the frontend look broken.	1338213230000000	1340981885000000
2468	enhancement	amercader	amercader	ckan-sprint-2012-06-25	closed	fixed	Finish off SlickGrid based Recline view	"Please see these GitHub issues:

https://github.com/okfn/recline/issues/130

https://github.com/okfn/recline/issues/131"	1338213375000000	1339757934000000
2472	enhancement	amercader	kindly	ckan-v1.8	closed	fixed	clean up datastore controller to get rid of datastore_url enables checks	"Datastore_url enabled checks are useless.

We should probably change this to datastore_url = 'active' and only update the flag to when a attempt at a post is made. 

This will need to be removed from the form at some point."	1338293407000000	1341222195000000
2505	enhancement	amercader	amercader	ckan-sprint-2012-06-25	closed	fixed	Docs improvements for 1.7.1	"There are some areas where the documentation could use some improvement and it would be good to have it available for 1.7.1.

 * Add an introduction to harvesting support
 * Add an introduction to CSW support
 * Better upgrade docs: Some users suggested improvements to the upgrade docs on the list [1]. Also we should mention how the upgrade would work for point point releases
 * Source install: see #2406

[1] http://lists.okfn.org/pipermail/ckan-dev/2012-May/002178.html"	1339166533000000	1340018770000000
2534	defect	amercader	seanh	ckan-v1.8	closed	duplicate	ckanext-spatial is broken with CKAN 1.8b	"Looks like recent cleanups in CKAN broke an import in the extension:

https://gist.github.com/2935688"	1339754762000000	1340631124000000
2604	enhancement	amercader	amercader	ckan-v1.8	closed	fixed	Add Requests as requirement for tests	"Requests (http://python-requests.org) is really useful to deal with http requests and saves a lot of time.

I propose we include it as a requirement for the tests for the time being, and even for core if necessary at a later stage."	1340790229000000	1340968458000000
2639	enhancement	amercader	amercader	ckan-v1.8	closed	fixed	Update recline version for 1.8 release	New recline version has significant improvements like the SlickGrid based grid view, it'd be good to have it on 1.8.	1341845809000000	1350296205000000
2640	enhancement	amercader	amercader	demo phase 2	closed	fixed	Update recline on the demo site	Depends on #2639	1341845911000000	1342112730000000
2788	enhancement	amercader	amercader	ckan-v1.9	new		Speed improvements on creating/updating and indexing	"Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.
"	1343832992000000	1343832992000000
2949	defect	amercader	amercader		new		Reenable Data API button on the new theme	The checks to show or not the button need to be updated for the latest datastore version	1349107464000000	1349107464000000
3022	defect	amercader	amercader	ckan 2.0	closed	fixed	setup_template_variables method of IDatasetForm never called	On the package controller the package_type is not passed to the lookup function, so the setup_template_variables defined on the extensions is never called	1353602743000000	1358254781000000
3023	enhancement	amercader	amercader		new		New methods on IPackageController to provide access to the data_dict	Extension hooking into the edit and create methods of the IPackageController interface receive the package object. This may not include all the fields that came from the form. The new extension points will pass the validated data_dict so extensions can have access to it	1353603657000000	1353603657000000
3025	enhancement	amercader	amercader		new		Add requests to core requirements	Because yes please	1354052124000000	1354052124000000
3026	enhancement	amercader	amercader		new		Support icons on  nav_named_link	nav_named_link won't include the nice icon even when passing the icon keyword param.	1354301516000000	1354301516000000
1562	enhancement	Adria	jilly mathews	ckan-future	new		Finish Geo Spatial	Estimated 4 weeks of Adria's time. I guess this will need to be broken down into more tickets. This feature is being requested by a number of potential customers and we have some ideas of requirements between Rufus and Jilly for this. This is the most popular new feature we talk about to new clients.	1324292193000000	1324292193000000
2757	enhancement	Adria	seanh	ckan-v1.8	closed	fixed	Write documentation for upgrading a CKAN install from an older to a newer version of CKAN		1343235242000000	1343392006000000
140	enhancement		rgrp	ckan-backlog	new		News section on front page	"Have a news section (suggest as a sidebar item).

News section will link to latest 3/4 blog posts on CKAN from blog.okfn.org.

Details:

  * Suggest pulling via rss or similar.
  * Will want to cache this ...

Cost: 4h?"	1254902541000000	1265625159000000
182	enhancement		rgrp	v1.0	closed	fixed	Improve openness icons	"Tick and cross aren't clear enough. Propose:

  * Icon for downloadability (nothing if absent ...(?))
  * Icon for open license? (Need something if closed ...)

Cost: 2h (?)"	1257534590000000	1270567116000000
253	enhancement		dread	ckan-backlog	assigned		Package relationships	"= Overview =
Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management.

= Examples =

1. There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial.

2. In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.)

3. OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it.

4. IPCC collection of data linked / mirrored. Not sure if there are useful relationships here?

5. Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM.

See more examples discussed here: http://trac.ckan.org/ticket/253

= Implementation =

This is split into four tickets:

 * Model: ticket:254
 * Read in WUI: ticket:255
 * Edit in WUI: ticket:256
 * API: ticket:257

No need for write access to be provided API for the moment.

This ticket also encompasses ticket:169 (Package derivations) and ticket:176 (Package dependencies)."	1266854721000000	1339774726000000
256	requirement		dread	ckan-backlog	assigned		Package relationships - 3. Edit in WUI	"WUI:
  * Editable as part of package or separately? (e.g. like authz)
  * Do we normalize to only one type name of the pair?
  * Do we allow create of relationship from both ends (e.g. only from dependency to dependent or either way?)
"	1266928561000000	1339774714000000
285	enhancement		rgrp		assigned		Paginate list of packages on tag read page	"Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ...

"	1270664606000000	1340631923000000
301	enhancement		rgrp		assigned		Package discussion pages	"A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package.

It provides a way for people to make suggestions about a package without needing access to main package."	1272301033000000	1340632055000000
331	enhancement		rgrp	ckan-backlog	new		Timezone of CKAN timestamps should be configurable	"Revisions are timestamped using the server's clock, which may not relate to the expected timezone for the site. e.g. the Norway site has a server on GMT. No timezone info is displayed either.

Would like to set timezone for a CKAN instance to use in rendering revision timestamps. For example, use CET or EST timezone."	1275302440000000	1339774701000000
333	enhancement		dread	v1.1	closed	wontfix	CKAN front end requirements for package notifications	"== Use case: new package ==

 1. An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package. 

 2. User: clicks 'new package'. 

 3. CKAN presents the package/new form to the user. 

 4. (After a couple of previews) User: clicks 'commit'. 

 5. Notification message goes from CKAN to the front-end detailing the new package.

6. The user is redirected back to the front-end web page displaying the list of packages, which contains the new one.

The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6).

In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages?

== Use case: CKAN imports packages ==

1. CKAN administrator runs a script that adds 100 new packages into CKAN. 

2. CKAN sends notification message to front-end to report the new packages/revisions.

3. Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages.

4. The front-end queries CKAN for each new package one-by-one.

5. A new user request to the front-end will include the info about the new packages.

The package addition could be achieved in 1 revision, 100 revisions or some compromise:

 * If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'. 

 * If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN.

This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages."	1275324042000000	1275407987000000
339	defect		johnbywater		closed	fixed	Remove download_url from package data format in API Version 2, but maintain the old data format in the old version of the interface (API Version 1)		1275901499000000	1278074705000000
341	enhancement		dread		closed	fixed	Web UI accepts package IDs in URLs	"(in the same way that we can refer to packages in the API by ID as well as name.)

== As a ==

client of CKAN

== I want to ==

link to a package page in the UI, referring to it by ID"	1276162400000000	1277483030000000
344	defect		dread		closed	fixed	REST Create package with incorrect format gives 500 error	"PUT to /rest/api/package of {{{ {""name"": ""name"", ""resources"": [""someurl.com/data""] } }}} (i.e. resource is a string, not a dictionary) gives 500 error, when it should give a 400 error and helpful error message.

Exception:
{{{
Module ckan.controllers.rest:154 in create
<<              if register == 'package' and not subregister:
                       fs = ckan.forms.get_standard_fieldset()
                       request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data)
                       fs = fs.bind(model.Package, data=request_fa_dict, session=model.Session)
                   elif register == 'package' and subregister in model.PackageRelationship.get_all_types():
>>  request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data)
Module ckan.forms.package_dict:88 in edit_package_dict
<<                      for res_dict in value:
                               res_dict_str = {}
                               for key, value in res_dict.items():
                                   res_dict_str[str(key)] = value
                               resources.append(res_dict_str)
>>  for key, value in res_dict.items():
AttributeError: 'unicode' object has no attribute 'items'
}}}"	1276341172000000	1277477712000000
350	enhancement		dread	ckan-backlog	reopened		Search engine optimisation	"Need to research what can easily be done to improve CKAN packages in the search rankings.

Comments from Glen Barnes:

We've been pretty successful at SEO without even really trying (see http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (http://www.ckan.net/package/ascoe -> http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly (""Atmospheric Chemistry Studies in the Oceanic Environment""  on the example above). Some basic SEO 101 on page optimisations. 
"	1276594541000000	1339774690000000
351	enhancement		dread	ckan-backlog	new		Homepage: list new, updated and 'hot' packages	"Have a simpler list of exciting data, as opposed to the big revision list.

For example:

{{{
Hot data
===========

New packages: package1, package2, package3
Updated resources: package1, package2, package3
Popular packages: 
}}}"	1276595816000000	1339774677000000
353	defect		dread		closed	fixed	SOLR search indexing	"== As a ==

SOLR instance

== I want to ==

keep my search index of CKAN packages up-to-date 

== Implementation ==

* Using asynchronous event notifications
* Running in a separate process to CKAN"	1277123480000000	1280756399000000
356	enhancement		rgrp	v1.1	closed	fixed	Search box in at top of page (UI)	"A small but useful ui improvement would be to have a search box at top right on every page.

As an example see the one here on trac or on github.com or bitbucket.org.

  * It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration)."	1277235411000000	1278931830000000
364	defect		dread	ckan-v1.3	closed	fixed	Search for 'statistic' returns nothing	"On ckan.net there are plenty of packages (and indeed their tags) with the word 'statistic' in them, but no packages turn up when you search for it:

http://ckan.net/package/search?q=statistic&search=Search+Packages+%C2%BB

(Using Postgres full text search)"	1278949620000000	1291637291000000
368	defect		anonymous		closed	wontfix	500 Server error when creating package	"
I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response:

""publish: New Zealand Coastline (new-zealand-coastline)""
opening connection to nz.ckan.net...
opened
<- ""POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n""
<- ""{\""name\"":\""new-zealand-coastline\"",\""title\"":\""New Zealand Coastline\""}""
-> ""HTTP/1.1 500 Internal Server Error\r\n""
-> ""Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n""
-> ""Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n""
-> ""Vary: Accept-Encoding\r\n""
-> ""Content-Type: text/html; charset=iso-8859-1\r\n""
-> ""Cache-Control: proxy-revalidate\r\n""
-> ""Content-Length: 640\r\n""
-> ""Connection: close\r\n""
-> ""\r\n""
reading 640 bytes...
-> ""<!DOCTYPE HTML PUBLIC \""-//IETF//DTD HTML 2.0//EN\"">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n<hr>\n<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80</address>\n</body></html>\n""
read 640 bytes
Conn close

The server should not respond with a 500 error if the package was created. "	1279622462000000	1291831811000000
369	enhancement		shudson@…	ckan-backlog	new		"""Package Listing Key"" should appear on Tag results"	"Currently there's a nice legend titled ""Package Listing Key"" that appears in right side of ""Browse Packages"" results. The same key should show on other search results like when searching for a tag."	1279821634000000	1339774666000000
370	enhancement		shudson@…	ckan-backlog	new		Use better email encryption for author_email and maintainer_email	"The JavaScript email encryption used is not very reassuring. Google's MailHide is a much better solution that is easily implemented.

http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built."	1279821819000000	1339774649000000
377	enhancement		glen@…	ckan-future	closed	wontfix	API Should return JSON in all cases	"When the API returns any response it should always retunr JSON. For example a 404 response should return something like

{{{
{'status': 404, 'message': 'package somename not found'}

}}}

When errors are encountered when creating a package: 


{{{
{'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be
blank', 'description':'license code must be an integer']}

}}}

Not authorised:

{{{
{'status': 503, 'message': 'You are not authorised to create this package.'}

}}}

The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created.
"	1280268158000000	1338206349000000
381	defect		johnbywater		closed	invalid	Test defect	please ignore	1280345937000000	1294410466000000
395	task		pudo	ckan-v1.3	closed	duplicate	Set up profiling to analyze performance issues	"At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls. 

This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance.

NB: We should also agree on a maximum request latency. 

TODO: Read up on all those QoS tickets to avoid overlapping efforts. "	1280824739000000	1294417538000000
403	requirement		pudo		closed	fixed	Create web service for OFS storage	"Create a thin web service that will allow users to upload files to OFS on S3 without knowing S3 credentials. This could also provide static urls for all stored resources.

* relates to #307"	1281346110000000	1287392451000000
407	requirement		dread		closed	wontfix	Link to Scraperwiki	"== As a ==

user browsing a CKAN package which has a resource in a bad format

== I want to ==

be alerted to the possibility of scraping it on scraperwiki

== Implementation ==
This should be done in a CKAN extension. It should appear in a side-bar on the package read page.

If the package does not have a resource with scraperwiki.com in the url, then it displays this message:

Title: Scrape it[[BR]]
Image: (Scraperwiki dumper truck)[[BR]]
Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can [http://scraperwiki.com/market/request/ request] this data is scraped or even [http://scraperwiki.com/editor/ write the scraper] yourself.
"	1281348368000000	1330769956000000
412	requirement		johnbywater		closed	fixed	Data format for package importing shall be documented	"Requirements:
* cater naturally for multiple packages with multiple resources
* international encoding
* large number of packages"	1281430809000000	1289822092000000
415	task		dread	ckan-v1.2	closed	fixed	Chase Talis about loading RDF from CKAN.		1281431656000000	1288003954000000
418	task		dread		closed	invalid	With COI, get SOLR working		1281431691000000	1291637919000000
422	story		dread	ckan-v1.3	closed	fixed	Administrator imports meta-data into CKAN	"Requirements:

 * cater naturally for multiple packages with multiple resources
 * international encoding
 * large number of packages"	1281433490000000	1292586586000000
428	requirement		dread	ckan-v1.2	closed	fixed	Daily dump of DGU CKAN available	End users want to get hold of a JSON & CSV dump of the DGU data.	1281703251000000	1285757316000000
429	requirement		dread	ckan-v1.2	closed	fixed	ONS data is imported daily to DGU		1281703294000000	1285757274000000
433	defect		wwaites	datapkg-0.8	closed	fixed	Data package metadata in the Egg	"Still not sure if we shouldn't use the existing setuptools machinery to manage this -- there is already a way to get at the metadata. In any event, I've just made an addition to datapkg that makes it possible to put datapkg_sources as a dictionary in your package's setup.py. Afterwards it is possible to pull the metadata out of the egg. Of course this could easily be changed to save the information in whatever form, indeed if you pass it a string instead of a dictionary it will just write whatever you gave it into the datapkg_sources.spec. The point is, I think that the egg is a good place to stuff this information.

For non-python users, it is always possible to simply put up the datapkg_sources.spec somewhere on the web so they can directly retrieve the data files.

From the docstring::
{{{
    This is the implementation for an [egg_info.writers] entrypoint.
    Datapkg adds an argument to setuptools's setup() function called
    datapkg_sources. The argument should be a dictionary of the form:

    .. code-block:: python

        setup(
            ...,
            datapkg_sources = {
                ""cra2009"" : ""http://www.hm-treasury.gov.uk/d/cra_2009_db.csv""
            }
        )

    The result of this is that there will be a file in the egg called
    datapkg_sources.spec that looks like this::

        [sources]
        cra2009=http://www.hm-treasury.gov.uk/d/cra_2009_db.csv

    How do you get at this data? Simple::

    .. code-block:: python

        import pkg_resources
        dist = pkg_resources.get_distribution(""ukgov_treasury_cra"")
        spec = dist.get_metadata(""datapkg_sources.spec"")

    and 'spec' will be the contents of the file as a string.
}}}
"	1282078660000000	1283183647000000
434	defect		dread	ckan-v1.2	closed	fixed	Logged in user to si.ckan.net causes genshi exception	"Here's the trace:

{{{
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] Error - <type 'exceptions.IndexError'>: list index out of range
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] mod_wsgi (pid=6195): Exception occurred processing WSGI script '/home/okfn/var/srvc/si.ckan.net/py
env/bin/si.ckan.net.py'.
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] Traceback (most recent call last):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/paste/cascade.py"", line 
130, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return self.apps[-1](environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/paste/registry.py"", line
 375, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     app_iter = self.application(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/repoze/who/middleware.py
"", line 107, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     app_iter = app(environ, wrapper.wrap_start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/middleware.py"", l
ine 214, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     self.app, new_environ, catch_exc_info=True)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/util.py"", line 94
, in call_wsgi_application
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     app_iter = application(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/weberror/errormiddleware
.py"", line 156, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return self.application(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/beaker/middleware.py"", l
ine 73, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return self.app(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/beaker/middleware.py"", l
ine 152, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return self.wrap_app(environ, session_start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/routes/middleware.py"", l
ine 130, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     response = self.app(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/wsgiapp.py"", line
 125, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     response = self.dispatch(controller, environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/wsgiapp.py"", line
 324, in dispatch
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return controller(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py"", line 73, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return WSGIController.__call__(self, environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core.
py"", line 221, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     response = self._dispatch_call()
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core.
py"", line 172, in _dispatch_call
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     response = self._inspect_call(func)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core.
py"", line 107, in _inspect_call
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     result = self._perform_call(func, args)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core.
py"", line 60, in _perform_call
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return func(**args)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/controllers/error.py"", line 32, in doc
ument
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     ckan_template = render('error_document_template.html')
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py"", line 38, in render
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return render_template()
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py"", line 36, in render_templ
ate
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return literal(stream.render(method=method, encoding=None))
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/core.py"", line 18
3, in render
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return encode(generator, method=method, encoding=encoding, out=out)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 
57, in encode
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return _encode(''.join(list(iterator)))
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 339, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for kind, data, pos in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 670, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for kind, data, pos in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 771, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for kind, data, pos in chain(stream, [(None, None, None)]):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 586, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for ev in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/core.py"", line 288, in _ensure
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py"", line 618, in _include
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in tmpl.generate(ctxt, **vars):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py"", line 618, in _include
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in tmpl.generate(ctxt, **vars):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py"", line 605, in _include
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/markup.py"", line 327, in _match
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py"", line 545, in _flatten
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for kind, data, pos in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py"", line 177, in _generate
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in msgbuf.translate(gettext(msgbuf.format())):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py"", line 1029, in translate
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     parts = parse_msg(string)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py"", line 1143, in parse_msg
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     parts.append((stack[-1], string))
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] IndexError: list index out of range
}}}

ckan: 1e403927f2bb (metastable)
genshi: 0.6
lang: slovenian"	1282130515000000	1283189807000000
436	bug		dread	ckan-v1.2	closed	wontfix	Investigate exception: resource search JSON	"Here's the dump from 22:10 last night:

URL: http://ckan.net/api/search/resource?all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.rest:400 in search
<<                      response.status_int = 400
                           return gettext('Blank qjson parameter')
                       params = json.loads(request.params['qjson'])
                   elif request.params.values() and request.params.values() != [u''] and request.params.values() != [u'1']:
                       params = request.params
>>  params = json.loads(request.params['qjson'])
Module simplejson:384 in loads
<<              parse_constant is None and object_pairs_hook is None
                   and not use_decimal and not kw):
               return _default_decoder.decode(s)
           if cls is None:
               cls = JSONDecoder
>>  return _default_decoder.decode(s)
Module simplejson.decoder:402 in decode
<<          """"""
               obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               end = _w(s, end).end()
               if end != len(s):
>>  obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Module simplejson.decoder:420 in raw_decode
<<              obj, end = self.scan_once(s, idx)
               except StopIteration:
                   raise JSONDecodeError(""No JSON object could be decoded"", s, idx)
               return obj, end
>>  raise JSONDecodeError(""No JSON object could be decoded"", s, idx)
JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
CGI Variables
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, application/json'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING	'gzip,deflate'
HTTP_ACCEPT_LANGUAGE	'en-us,en;q=0.5'
HTTP_CONNECTION	'keep-alive'
HTTP_COOKIE	'__utma=27730403.1245320310.1281386803.1281386803.1282164955.2; __utmz=27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892; __utmb=27730403.3.10.1282164955; __utmc=27730403'
HTTP_HOST	'ckan.net'
HTTP_KEEP_ALIVE	'300'
HTTP_REFERER	'http://jira/browse/PLATFORM-892'
HTTP_USER_AGENT	'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/api/search/resource'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/search/resource'
QUERY_STRING	'all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class='
REMOTE_ADDR	'64.235.97.218'
REMOTE_PORT	'20720'
REQUEST_METHOD	'GET'
REQUEST_URI	'/api/search/resource?all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class='
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/api/search/resource'
SCRIPT_URL	'/api/search/resource'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0xa1c13ec>
beaker.cache	<beaker.cache.CacheManager object at 0xa1c142c>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0xa1c12ac>>
beaker.session	{'_accessed_time': 1282165818.0880959, '_creation_time': 1282165818.0880959}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: __utma='27730403.1245320310.1281386803.1281386803.1282164955.2' __utmb='27730403.3.10.1282164955' __utmc='27730403' __utmz='27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892'>, '__utma=27730403.1245320310.1281386803.1281386803.1282164955.2; __utmz=27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892; __utmb=27730403.3.10.1282164955; __utmc=27730403')
paste.parsed_querystring	([('all_fields', '1'), ('offset', '0'), ('limit', '20'), ('qjson', '<span class=')], 'all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=')
paste.registry	<paste.registry.Registry object at 0x130ed84c>
paste.throw_errors	True
pylons.action_method	<bound method RestController.search of <ckan.controllers.rest.RestController object at 0xe9bbe0c>>
pylons.controller	<ckan.controllers.rest.RestController object at 0xe9bbe0c>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0xe9bbe8c>
pylons.routes_dict	{'action': u'search', 'controller': u'rest', 'register': u'resource'}
repoze.who.logger	<logging.Logger instance at 0xa3cb0cc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 170067148>, 'auth_tkt': <AuthTktCookiePlugin 171739788>}
routes.route	<routes.route.Route object at 0xa102fac>
routes.url	<routes.util.URLGenerator object at 0x13a5a3cc>
webob._parsed_query_vars	(GET([('all_fields', '1'), ('offset', '0'), ('limit', '20'), ('qjson', '<span class=')]), 'all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x12f53530>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0x13a5a3cc>, {'action': u'search', 'controller': u'rest', 'register': u'resource'})
"	1282206959000000	1288003983000000
437	bug		dread	ckan-v1.2	closed	fixed	Buildbot test failures - ascii codec	"On today's buildbot: http://buildbot.okfn.org/builders/buildbot-test/builds/201

2 failures about ascii (ignore other 2)"	1282223640000000	1288004009000000
439	task		dread		closed	fixed	Plan server requirements for DGU	"Using use cases for security etc.
Draw diagram to satisfy use cases.
Communicate to Paul and therefore TSO."	1282224452000000	1294413903000000
444	task		dread		closed	duplicate	Discuss package relationships ideas with JF	"* Create test data on visible ckan
* Discuss with JF"	1282299238000000	1294414008000000
446	task		dread		closed	invalid	Server configuration query (?)	" * Ensure use case is in design - ""an admin writes to the CKAN API"" (and see JB notes on paper)
 * Communicate how this is planned to be achieved in design to JF"	1282299385000000	1294414077000000
450	task		dread		closed	fixed	Assist with proxying of static media for forms	John did this a while ago.	1282299573000000	1287997487000000
452	task		dread		closed	fixed	Assist with using package update error form		1282299647000000	1287997540000000
455	task		johnbywater		closed	invalid	SOLR - suggest 1 pager about how system would work	Either CKAN writes to SOLR and Drupal reads from SOLR, or CKAN writes to SOLR and Drupal reads SOLR via CKAN API (so search resource locations are unaffected).	1282299913000000	1291637172000000
459	bug		johnbywater	ckan-v1.2	closed	fixed	Versions on branches are broken		1282299973000000	1282921783000000
463	defect		dread	ckan-v1.3	closed	worksforme	package diff exception	"Investigate exception:

{{{

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
AUTH_TYPE	'cookie'
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'en-US,en;q=0.8'
HTTP_CACHE_CONTROL	'max-age=0'
HTTP_COOKIE	'__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173'
HTTP_HOST	'ckan.net'
HTTP_REFERER	'http://ckan.net/package/history/dbtune-audioscrobbler'
HTTP_USER_AGENT	'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'92.226.62.168'
REMOTE_PORT	'55102'
REMOTE_USER	'http://pudo.org/'
REMOTE_USER_TOKENS	['']
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0xbb528cc>
beaker.cache	<beaker.cache.CacheManager object at 0xbb528ec>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0xbb5286c>>
beaker.session	{'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: __utma='27730403.1381387877.1281962773.1282248244.1282250173.11' __utmb='27730403.6.10.1282250173' __utmc='27730403' __utmz='27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan' auth_tkt='e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!'>, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0xf702aec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xf6eaf0c>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xf6eaf0c>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0xf6eaa8c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.identity	<repoze.who identity (hidden, dict-like) at 280815340>
repoze.who.logger	<logging.Logger instance at 0xbcdfccc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 196612236>, 'auth_tkt': <AuthTktCookiePlugin 198048140>}
routes.route	<routes.route.Route object at 0xbbc44cc>
routes.url	<routes.util.URLGenerator object at 0xf5e55ec>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0xfe0b0b0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xf5e55ec>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})
}}}"	1282306037000000	1294916148000000
465	requirement		johnbywater	ckan-v1.3	closed	wontfix	Require API key for all API requests.		1282308904000000	1294411534000000
467	story		johnbywater	ckan-v1.3	closed	duplicate	Admin configures CKAN to expect API key in named HTTP header		1282310562000000	1294411681000000
469	task		johnbywater	ckan-v1.2	closed	fixed	Unit test to check API key in correctly named HTTP header is accepted by server		1282310760000000	1282932682000000
471	story		johnbywater	ckan-v1.2	closed		API user sends API key in correctly named header		1282312088000000	1283248729000000
472	story		johnbywater	ckan-v1.2	closed		API user sends API key in incorrectly named header		1282312108000000	1282932740000000
473	story		johnbywater	ckan-v1.2	closed		API user discovers correct header for sending API key		1282312203000000	1283248736000000
475	task		johnbywater	ckan-v1.2	closed	fixed	Unit test to check API key in incorrectly named HTTP header is not accepted by server		1282312802000000	1282932705000000
476	requirement		johnbywater	ckan-v1.3	closed	wontfix	Daily database dumps shall be discoverable and downloadable		1282313433000000	1294411741000000
477	story		johnbywater	ckan-v1.3	closed	duplicate	Discover location of the daily database dumps		1282313788000000	1294411761000000
478	task		johnbywater	ckan-v1.2	closed	fixed	"Unit test for ""interface resource"" in API"	We don't want /api to be an HTML page, but rather provide information about the API in a JSON structure.	1282315067000000	1283248713000000
481	story		johnbywater	ckan-v1.3	closed	fixed	Unrestricted total read-only access to catalogue API		1282422697000000	1294248359000000
483	story		johnbywater		closed	invalid	Restricted total read-write access to catalogue Web UI		1282422801000000	1294417216000000
484	story		johnbywater		closed	invalid	Restricted partial read-write access to catalogue Web UI		1282422858000000	1294417248000000
485	story		johnbywater		closed	duplicate	Performance beats QoS criteria		1282425219000000	1294411946000000
486	requirement		johnbywater	ckan-v1.3	closed	duplicate	Catalogue service shall notify and query SOLR service		1282425790000000	1291639321000000
487	story		johnbywater		closed	duplicate	Notify SOLR service of model events		1282425910000000	1291639404000000
488	requirement		johnbywater		closed	wontfix	Catalogue service shall notify RDF service		1282426021000000	1320930240000000
490	requirement		johnbywater		closed	fixed	The system shall support registering a harvest source via the CKAN API		1282426875000000	1286200804000000
491	story		johnbywater	ckan-v1.2	closed		Get form for creating harvest source entity		1282427008000000	1284493173000000
492	story		johnbywater	ckan-v1.2	closed		Submit harvest source create form response to the API		1282427042000000	1284493145000000
493	story		johnbywater	ckan-v1.2	closed		Get harvest source entity		1282427083000000	1284493130000000
494	story		johnbywater	ckan-v1.2	closed		Get form for updating remote metadata entity		1282427106000000	1285198894000000
495	story		johnbywater	ckan-v1.2	closed		Put form for updating remote metadata entity		1282427150000000	1285198898000000
498	story		johnbywater		closed	invalid	"Search packages within location ""bounding box"""		1282427412000000	1294412520000000
500	defect		dread	ckan-v1.2	closed	duplicate	Exception from diff	"Investigate exception occured occasionally in last couple of days on ckan.net:

{{{
WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'key' 					

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'zh-cn,zh-tw'
HTTP_CONNECTION	'close'
HTTP_HOST	'ckan.net'
HTTP_USER_AGENT	'Baiduspider+(+http://www.baidu.com/search/spider.htm)'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'123.125.66.32'
REMOTE_PORT	'63767'
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x9f603ec>
beaker.cache	<beaker.cache.CacheManager object at 0x9f6042c>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x9f602ac>>
beaker.session	{'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: >, '')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0x104552ec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xfb17aec>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xfb17aec>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x10286d4c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.logger	<logging.Logger instance at 0xa16e0cc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 167584972>, 'auth_tkt': <AuthTktCookiePlugin 169253516>}
routes.route	<routes.route.Route object at 0x9f3690c>
routes.url	<routes.util.URLGenerator object at 0xfd8d7cc>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x103a5bf0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xfd8d7cc>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})



}}}"	1282553033000000	1287747652000000
501	requirement		pudo	ckan-v1.2	closed	duplicate	Read-only maintenance mode	CKAN should have a read-only maintenance mode with a nice little banner on all pages, appropriate REST messages etc. Bonus points if this is triggered via an environment variable and thus can be triggered by the surrounding apache. 	1282554617000000	1282724566000000
507	story		johnbywater		closed	fixed	Upgrade a CKAN service		1282661646000000	1282909852000000
511	requirement		dread	ckan-v1.3	closed	worksforme	Better warnings and errors when using API		1282754677000000	1297075354000000
512	story		dread		closed	duplicate	User creates package via API with incorrect core fields specified		1282754750000000	1294917121000000
513	task		dread		closed	wontfix	Creating a package with API should warn if there are incorrect fields	"When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK.

"	1282754802000000	1302774329000000
514	defect		dread	ckan-v1.2	closed	duplicate	Inconsistent use of 'location' header in API	"When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all.

I've removed the info about the header in the docs in the meantime."	1282757357000000	1282757391000000
516	defect		dread	ckan-v1.2	closed	fixed	Double-escaped text visible - revision page	"At: http://ckan.net/revision
you can see:
""Next &raquo;"" on pager
This doesn't seem to be a problem with earlier (1.0.1) version."	1282829539000000	1288002933000000
533	requirement		johnbywater	ckan-v1.3	closed	invalid	The performance of CKAN shall scale	"CKAN uses a database. We can put the database on another machine and then clone the CKAN machine. Then there are two machines calling the database and converting SQL results into HTTP responses.

How many machines can you have before the database becomes the bottleneck? At that point, what QoS can be obtained with normal hardware?

Afterwards, can CKAN be made to scale further than this with a little development? Can its Postgres database be mirrored in realtime? Or can we change SQL writes (create/updates) to write to many databases?

Are there any other options for scaling the persistence mechanism?"	1282910499000000	1292957374000000
535	defect		dread	ckan-v1.2	closed	duplicate	genshi error when logged into sl.ckan.net	Genshi exception when rendering the page whilst logged in to sl.ckan.net.	1283165774000000	1283167040000000
544	requirement		pudo	ckan-v1.3	closed	duplicate	Backport facet browsing to CKAN 1.2	"This is in IATI, would be nice to have in generic CKAN. 

 * http://bitbucket.org/okfn/iati/changeset/378431974c76"	1283267292000000	1291638966000000
557	defect		pudo	ckan-v1.3	closed	invalid	Remove siblings from link_to/from relationship display	Displaying siblings is not needed for link_to/from.	1283542595000000	1297075159000000
558	defect		pudo	ckan-v1.2	closed	fixed	Sort extras, relationships by key/object alphabetically		1283542680000000	1283781883000000
561	defect		pudo	ckan-v1.2	closed	duplicate	Deleted packages are returned in the API	Anja is reporting this, severe bug, I think.	1283775578000000	1283775711000000
562	defect		pudo	ckan-v1.2	closed	fixed	OpenID Login fails using Yahoo and Google	"reported by LOD folks.

subsidary bug: error messages from OpenID providers are not shown on the form return page. "	1283941119000000	1284063574000000
564	requirement		johnbywater	ckan-v1.3	closed	wontfix	The system shall support location based search		1284033829000000	1294412714000000
567	story		johnbywater	ckan-v1.2	closed		Post new harvest job for given harvest source		1284039943000000	1284689374000000
573	story		johnbywater		closed	duplicate	Add metadata entity to harvesting queue		1284045353000000	1284220987000000
574	story		johnbywater		closed	duplicate	Create UKLII package with attributes from remote metadata record		1284045805000000	1284222410000000
575	requirement		johnbywater	ckan-v1.2	closed	fixed	The system shall notify remote metadata harvesting errors		1284047959000000	1288012749000000
577	story		johnbywater	ckan-v1.2	closed		Get remote metadata harvest job errors		1284209381000000	1285348049000000
578	story		johnbywater	ckan-v1.2	closed		Get remote metadata harvest job		1284209407000000	1284493202000000
579	story		johnbywater	ckan-v1.2	closed		Delete remote metadata harvest job		1284209440000000	1285260356000000
580	story		johnbywater		closed	duplicate	Write (create or update) CKAN package for metadata document		1284210730000000	1284223068000000
582	task		johnbywater	ckan-v1.2	closed	fixed	Domain object class for harvest source		1284211104000000	1284261402000000
583	task		johnbywater	ckan-v1.2	closed	fixed	Define fieldset object for harvest source forms		1284211169000000	1284261424000000
584	task		johnbywater	ckan-v1.2	closed	fixed	Migration script to add harvest source table		1284211204000000	1284483760000000
585	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to return harvest source create form		1284211302000000	1284261448000000
586	task		johnbywater	ckan-v1.2	closed	fixed	Routing configuration for harvest source create form API resource		1284211336000000	1284261447000000
587	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for getting harvest source create form from API		1284211364000000	1284492381000000
588	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to accept valid harvest source create form submission		1284211575000000	1284318555000000
589	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to reject invalid harvest source create form submission		1284211620000000	1284318561000000
590	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for submitting harvest source create form submission to API		1284211659000000	1284492397000000
591	task		johnbywater	ckan-v1.2	closed	fixed	Routing configuration for harvest source entity API resource		1284212082000000	1284342925000000
592	task		johnbywater	ckan-v1.2	closed	fixed	Methods to return harvest source entity as JSON		1284212338000000	1284342904000000
593	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to return remote metadata edit form		1284212498000000	1285198853000000
594	task		johnbywater	ckan-v1.2	closed	fixed	Routing configuration for remote metadata edit form API resource		1284212532000000	1285198859000000
595	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for getting remote metadata edit form from API		1284212572000000	1285198863000000
596	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to accept valid remote metadata edit form submission		1284212662000000	1285198874000000
597	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to reject invalid remote metadata edit form submission		1284212689000000	1285198885000000
598	story		johnbywater	ckan-v1.2	closed		List remote metadata entities for given publisher		1284213730000000	1286200795000000
599	task		johnbywater	ckan-v1.2	closed	fixed	Routing configuration for listing remote metadata entities for a given publisher		1284213814000000	1284347290000000
600	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller for listing remote metadata entities for a given publisher		1284213854000000	1284347296000000
601	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for listing remote metadata entities for a given publisher via API		1284213928000000	1284493045000000
602	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to accept valid harvest job		1284217587000000	1284336801000000
603	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to reject invalid harvest job		1284217624000000	1284338918000000
604	task		johnbywater	ckan-v1.2	closed	fixed	Domain object class for harvest job		1284217689000000	1284326615000000
605	task		johnbywater	ckan-v1.2	closed	fixed	Routing configuration for harvest job register		1284217966000000	1284336788000000
606	task		johnbywater	ckan-v1.2	closed	fixed	Methods to return harvest job as JSON		1284218102000000	1284344127000000
607	task		johnbywater	ckan-v1.2	closed	fixed	Routing configuration for harvest job entity		1284218136000000	1284344113000000
608	task		johnbywater	ckan-v1.2	closed	fixed	Routing configuration for register of harvest jobs with error status		1284218378000000	1285347281000000
609	task		johnbywater	ckan-v1.2	closed	fixed	Methods to return harvest jobs with error status as JSON		1284218437000000	1285347285000000
618	task		johnbywater	ckan-v1.2	closed	fixed	Domain object class for metadata document		1284219669000000	1284598039000000
619	task		johnbywater	ckan-v1.2	closed	fixed	Get HTTP response for given resource locator		1284220327000000	1284675079000000
620	task		johnbywater	ckan-v1.2	closed	fixed	Identify remote metadata entity as metadata document, WAF, or CSW		1284220527000000	1284689400000000
624	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'title' from local copy of metadata document		1284221083000000	1284675746000000
625	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'abstract' from local copy of metadata document		1284221101000000	1284675753000000
626	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'point of contact' from local copy of metadata document		1284221117000000	1285443038000000
627	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'guid' from local copy of metadata document		1284221134000000	1284598028000000
628	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'bounding-box' from local copy of metadata document		1284221156000000	1285443047000000
629	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'keywords' from local copy of metadata document		1284221191000000	1285443055000000
630	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'use constraints' from local copy of metadata document		1284221210000000	1285443069000000
631	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'resource-locator' from local copy of metadata document		1284221223000000	1284675741000000
632	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute 'topic category' from local copy of metadata document		1284221246000000	1285443078000000
633	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute '' from local copy of metadata document		1284221272000000	1285443112000000
634	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute '' from local copy of metadata document		1284221284000000	1285443115000000
635	task		johnbywater	ckan-v1.2	closed	fixed	Read attribute '' from local copy of metadata document		1284221305000000	1285443119000000
636	task		johnbywater	ckan-v1.2	closed	fixed	Method on API controller to delete given harvest job		1284221790000000	1285260342000000
637	task		johnbywater		closed	invalid	Read UKLII package data from given metadata document		1284222536000000	1284223104000000
638	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for getting harvest source entity from API		1284225805000000	1284491185000000
639	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for posting harvest job to API		1284226008000000	1284491239000000
640	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for getting harvest job entity from API		1284226036000000	1284491233000000
641	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for getting harvest job errors from API		1284226049000000	1285348033000000
642	task		johnbywater	ckan-v1.2	closed	fixed	Migration script to add harvest job table		1284226154000000	1284483747000000
643	task		johnbywater	ckan-v1.2	closed	fixed	Migration script to add metadata document table		1284226255000000	1284675721000000
644	task		johnbywater	ckan-v1.2	closed	fixed	Documentation for deleting harvest job entity		1284226402000000	1285260346000000
646	requirement		johnbywater		closed	wontfix	Web UI shall support registering harvest sources		1284740303000000	1294413500000000
648	requirement		pudo	ckan-v1.3	closed	fixed	Enable lock-down of package creation	"    - copy exisiting tests, modify authz in setUp, adapt and extend tests
        * Problem: default_role_actions is read by init_db 
        * Solution: nuke db after monkey-patching
        * role = model.Role('Reader'), del role.actions[...]

        self.PRE_MAUTHZ_RULES = copy(mauthz.default_role_actions)
        mauthz.default_role_actions.remove((Role.READER, Action.CREATE))
        #raise Exception(mauthz.default_role_actions)
        model.Session.remove()
        model.repo.rebuild_db()

3. Start from the functional
  * Both the api controllers and wui controllers to do
  * tests/function/test_authz.py extended to check CREATE in lockedDown mode. fails nicely. 
  http://bitbucket.org/pudo/ckan-authz2

is_authorized(user, Action.Create, model.Package)

-> Doing this will put 'Package' in the context field of the user_object_role table. This will trigger SQLAlchemy to attempt a join towards PackageRole in all queries. Since for class-level role assignments there never is a PackageRole join table entry, this will never return any results. 
    * cf branch ""classes"": http://bitbucket.org/pudo/ckan-authz2/src/6fd0475e0c66 
    * No possibility of including ""instance"" column in inheritance decision.
    * Even if this can be cirumvented, it must happen on a per-query level and would require a major re-modelling. 


* have a ckan install that would not allow visitors to either list packages or list groups 
    * two paths: create the listing, but for each group/pkg decide that you 
      cannot show this
    * lock down the whole page (/package/list) 
        - this is class-based, not object-based 

is_authorized(user, Action.Package_Create, model.System())
is_authorized(user, Action.Group_Create, model.System())

[Separating Package and Group roles may be useful going forward: PackageEditor, GroupEditor etc]

4. Find a standard way to lock down classes
    * possibly add default rows in user_object_role 
    * introduce lock-down into controllers: group new, package new, REST equivalents, 
    * confirm tests"	1284889519000000	1284889656000000
649	story		pudo		closed	fixed	Introduce Authorization Groups in model	"Authorization Groups:
  * Rename UserObjectRole to SubjectObjectRole
  * Create database migration to do the rename in DB 
  * Make sure all the tests are passing again

  (Doing this first seems more stable since we first want to ""re-gain"" exisiting functionality) 
  
  * Create authz.py and model tests for authorization_group-based auth
      * tests: functional: create fixtures, do actual requests
                 lib-tests: test for membership addition, removal, group-right additions affecting all members, group-right removal, group rights affecting non-members. 
#  * Adapt model to get these passing
  * Integration into existing controllers should be negligible but may want ot have tests
      - do not want that. would mean design is broken 
  * Need to create a new controller for editing authz groups (will have same permissions as any other object -- Reader, Editor, Administrator)
"	1284889884000000	1284889983000000
651	requirement		pudo		closed	fixed	Improve javascript helper for package/user addition on groups/authzgroups	This needs to be abstracted and we still need to be able to remove items before saving. 	1284972211000000	1311171104000000
652	requirement		pudo	ckan-v1.2	closed	fixed	Allow for addition and removal of groups on package/edit	Much more natural place to do this, even at the risk of overloading the package edit form. 	1284998134000000	1284998201000000
653	requirement		dread	ckan-backlog	new		Trackback links for packages	"When people link to a package, a track-back link is auto-created. (Similar system as for blogs). 

As suggested by Tim Davies: 

Allowing some form of ‘track back’ against datasets
When a non-technical user comes to look at a dataset it would be really useful for them to be able to see if anyone has created an interface interpretation of it already.


I found quite a few cases in research of end-users struggling to make sense of a dataset when good interfaces to that data had already been built and blogged about, but without there being any link from the dataset listing to those data uses.
 
Accepting track backs could also make it easier for technical users to find blog posts / shared code etc. relating to a given dataset.
"	1285062025000000	1339774636000000
654	bug		johnbywater	ckan-v1.2	closed		Harvest sources and jobs should return 404 when missing (not 500)		1285170717000000	1285254097000000
655	story		johnbywater	ckan-v1.2	closed		Return status code 404 when harvest source is not found		1285252399000000	1285254088000000
656	story		johnbywater	ckan-v1.2	closed		Return status code 404 when harvesting job is not found		1285252429000000	1285254084000000
657	task		johnbywater	ckan-v1.2	closed	fixed	Fix harvesting job entity get method to default to returning 'None' rather than raising an exception		1285252536000000	1285254068000000
658	task		johnbywater	ckan-v1.2	closed	fixed	Fix harvest source entity get method to default to returning 'None' rather than raising an exception		1285252579000000	1285254072000000
660	requirement		dread		closed	invalid	Check CKAN instance works	As an admin I want to check a CKAN instance works having just upgraded it or configured it.	1285348463000000	1311183115000000
661	task		dread	ckan-v1.3	closed	fixed	Create CKAN site tester	"From Rufus: 

{{{
'blackbox' testing command (read-only only would be fine) -- very
useful for upgrade on sunday

On blackbox thinking of something like:

paster test-site http://hmg.ckan.net {package-name}

Or (maybe better):

paster test-site --site-spec {myconfig.json}

my-config is a json file with info for testing (could overload
--config rather than site-spec):

site-url: http://hmg.ckan.net/
package-name:
package-title:
...etc ...

Then:

Test front-page
Test /revision/
Test /package/{page}  -- check title
Test /package/history
Test /api/rest/package/ ...
Test /api/search/package ...
etc
}}}"	1285348558000000	1288271266000000
666	requirement		johnbywater	ckan-v1.2	closed	fixed	The system shall support updating packages from new versions of previously harvested dataset or service metadata		1285590374000000	1288013526000000
673	task		johnbywater		closed		Construct and send CSW GetRecordById request		1286214712000000	1286786758000000
674	task		johnbywater		closed		Extract metadata documents from CSW GetRecordById		1286214780000000	1286787032000000
693	requirement		wwaites	ckan-v1.2	closed	fixed	Support ETag in cache decorator	... and make cache-control header optional	1286831051000000	1286831188000000
714	bug		johnbywater		closed	fixed	DGU package form shall have a read only field for ??? attribute	So that Drupal doesn't need to make any adjustments to the form, and risk data being lost on submission.	1287581408000000	1291733788000000
729	bug		pudo		closed	fixed	Assets to be loaded from assets.okfn.org, not m.okfn.org	Move from hetzner box to s3. 	1287736519000000	1288012854000000
730	task		rgrp		closed	duplicate	Back up package data from all CKAN packages to storage.ckan.net	"Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net.

  * Naming scheme?
    * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
      * What happens if names change
    * File: filename? hash?
  * Store hash back on ckan instance?"	1287737109000000	1291139609000000
737	enhancement		dread	ckan-backlog	new		Markdown syntax summary page	"I suggest we produce a quick Markdown cheat-sheet page, showing the key runes: e.g. create a title and quote some text. This page can link to the full Markdown docs for advanced users.

A user going to the Markdown docs that we link will have to read a couple of pages of the raison-d'etre of Markdown before he gets to the syntax. And it's not very easy to read, and being white on black it looks like proper geek stuff.
"	1287766749000000	1323170239000000
741	enhancement		ollyc	ckan-v1.2	closed	fixed	Extension / Plugin system for CKAN	"Create plugin system using
<http://pypi.python.org/pypi/pyutilib.component.core/>

Specification in CEP-003 (browser:doc/cep/cep-003.txt) and <http://okfnpad.org/ckan-plugins>
"	1287996890000000	1291989194000000
743	story		johnbywater		closed	wontfix	Create harvest source registration via Web UI		1288012860000000	1294412807000000
744	story		johnbywater		closed	wontfix	Update harvest source registration via Web UI		1288012960000000	1294412813000000
745	requirement		johnbywater	ckan-v1.3	closed	wontfix	Web UI shall accept requests for harvesting jobs		1288013148000000	1294412820000000
746	story		johnbywater		closed	wontfix	Request new harvesting job via Web UI		1288013198000000	1294412827000000
747	story		johnbywater		closed	wontfix	Track harvesting jobs via Web UI		1288013217000000	1294412834000000
748	story		johnbywater		closed	duplicate	Link new sample package to previous sample package of continuous series		1288013849000000	1294412976000000
749	story		johnbywater		closed	duplicate	Fold up continuous series in search results behind newest sample package		1288014002000000	1294412986000000
751	story		johnbywater		closed	duplicate	Get harvested document for a given package		1288014518000000	1288014616000000
761	story		dread		closed	invalid	Read-only CKAN for maintenance	Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done.	1288090334000000	1288091977000000
762	story		dread		closed	invalid	Permanently Read-Only CKAN instance	"A CKAN is used just for distributing metadata. Updates may still arrive through direct db manipulation, e.g.:
 * another (but writable) CKAN instance is connected to the same db
 * restoring database dumps from another CKAN db

"	1288090578000000	1288091982000000
763	enhancement		dread	ckan-future	assigned		Read-only mode - Setup	"Admin configures entering read-only mode in one of two places:
 * CKAN config file (e.g. ckan.ini)
 * environment variable from Apache config

Once enabled, no writes can occur to the database (including user ratings and other usage stats)."	1288091506000000	1338206204000000
765	enhancement		dread	ckan-backlog	assigned		Read-only mode - API usage	"All writes to the API are captured and you are returned an error explaining the reason.

Possible errors: 
 * 503 temporary maintenance
 * 403 forbidden (if server if permanently read-only)"	1288091897000000	1338206123000000
771	story		johnbywater	ckan-v1.3	closed	worksforme	Run CLI help command without arguments	Should return something sensible (currently reports an error).	1288190318000000	1294413256000000
772	story		johnbywater	ckan-v1.3	closed	duplicate	Run CLI harvester command without arguments	Should return help for the harvester (currently crashes).	1288190373000000	1294413313000000
773	task		johnbywater	ckan-v1.3	closed	duplicate	Fix CLI help command to return CLI help message		1288190475000000	1294413400000000
774	task		johnbywater	ckan-v1.3	closed	duplicate	Fix CLI harvester command to return harvester help message		1288190522000000	1294413412000000
796	defect		pudo	ckan-v1.3	closed	invalid	AlphaPage for non-latin languages	"The AlphaPage implementation assumes that all packages start with a latin character. This is not true for Russian, Chinese, Arabian and Persian languages etc. We should either use a unicode solution (i.e. display a full class of characters based on the current locale) or use generic paging instead. 
"	1289296662000000	1291640040000000
807	enhancement		rgrp	ckan-v1.3	closed	wontfix	Handle robot traffic better	"We get a lot of robot traffic. Notice that robots are visiting package edit pages. Suggest:

  * Add rel=nofollow to various links
  * Throttle some robots (not all seem to respect robots.txt)
"	1289899826000000	1297075372000000
809	requirement		pudo	ckan-v1.3	closed	wontfix	Link to the Wiki pages in package form	"This would allow for a community-driven definition of field values, conventions, etc. 

Plus: create wiki.ckan.net "	1289992231000000	1297075561000000
811	defect		cygri	ckan-backlog	new		Extra field editing form layout breaks when there are long field names	The layout of the editing section for extra fields breaks when a field name is slightly too long. Field names jump over to the right. See http://ckan.net/package/edit/dbpedia for examples.	1289994812000000	1323170289000000
812	defect		cygri	ckan-backlog	new		Package edit form only allows three extra fields	"= Rationale =
The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview).

= Implementation =
A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has)

Nice to have: a blank field is added when you tab from the last filled-in field in the table."	1289995010000000	1311176917000000
813	requirement		cygri	ckan-v1.3	closed	fixed	Add link to “Register new package” to the homepage	"A link to the “register new package” page would be handy somewhere on the front page. As a frequent editor, I often go to http://ckan.net in order to create a new package. I don't want to hunt around for the link to the “new package” form.

The link could go into the bar next to “About / Statistics”. Or into the “More information” sidebar. Neither of these truly make sense, but it's a frequently accessed link and that's reason enough to put it somewhere onto the frontpage.
"	1289995253000000	1294411047000000
814	requirement		cygri	ckan-v1.3	closed	fixed	Have an “About CKAN” link on every page	"A link “About CKAN” pointing to http://ckan.net/about should be visible on every page. This is important because people often land on subpages and may have trouble figuring out what CKAN is.

I would put this link into the main navigation bar, on the very right next to “Revision History”. Then I would also remove the “Home” item from the main navigation bar because it is redundant. The CKAN logo is already a link to the homepage. The only other subpage in the “Home” section is “Statistics” and that's already linked from the homepage sidebar.

But anywhere else would be fine as well."	1289995821000000	1294411109000000
815	requirement		cygri	ckan-v1.3	closed	duplicate	Autocomplete for the search field	The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page.	1289995951000000	1294410951000000
817	requirement		cygri	ckan-v1.3	closed	worksforme	Proposed redesign of Resources table on package pages	"There are some problems with the Resources table on package pages:

1. The column labelled “URL” doesn't contain URLs, but just the word “Download”.
2. The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links).

Attached is a screenshot for a proposed redesign. The changes are:

1. Move the Format column to the right of the Description column (description is more important)
2. Make the description a clickable link
3. Drop the “URL/Download” column because it's now redundant
4. Rename “Description” to “Resource” (not that important)"	1289997384000000	1297073724000000
818	requirement		cygri	ckan-backlog	new		Rethinking the author and maintainer fields	"The semantics of the Author and Maintainer fields are really unclear at the moment. This leads to very inconsistent usage. Also, perhaps Name and Email are not the only fields that are needed for a contact.

Here is a table that shows the current usage of these fields in CKAN:
http://richard.cyganiak.de/2010/ckan/ckan-ppl.html

We note several problems:

    * Author and Maintainer are often the same
    * Author and Maintainer are often used interchangeably
    * People really want to specify URLs for the contacts and stick them into random places because there is no field for it
    * Multiple comma-separated names in a single field

I'm not sure what to do about this, but a redesign is necessary in my opinion.

Some ideas:

    * Remove the maintainer field?
    * Make really clear that Author doesn't refer to the metadata on CKAN, but to the original data
    * Add an “author URL” field?"	1290003524000000	1339774621000000
819	defect		cygri	ckan-backlog	closed	fixed	Tag autocompletion widget broken	"The widget for tag autocompletion is broken in various ways.

For example, if I edit a package that is tagged ""music"", and just tab through the form fields to get to the Author field, then the widget changes to ""industrial-music"" as I tab through it.

Or if the tag is ""foo bar"" and I hit alt-right to jump to the end of the text field (with the intent of adding a third tag), then the contents change to ""foo barbecue""."	1290004121000000	1323171435000000
820	defect		dread		closed	fixed	Exception browsing /user/apikey when not logged in	"Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in.
{{{
WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'apikey' 						

URL: http://test-hmg.ckan.net/user/apikey
}}}"	1290006079000000	1311325226000000
821	defect		dread	ckan-v1.4-sprint-2	closed	worksforme	Some resources have disappeared in CKAN	"Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end.

Here are the details:
 * in CKAN's package view (in web interface, API and dump) no resources display
 * but they WERE created in CKAN, as shown by the revision diffs.
 * they are in the Drupal front-end

So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal.

Packages affected: anti-social-behaviour-orders-1999-2007
asylum-applications-jan-mar-2009
control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december
coroners-statistics-england-and-wales
courts-statistics-user-survey-england-and-wales
court-statistics-company-insolvency-and-bankruptcy-england-and-wales
court-statistics-england-and-wales
court-statistics-mortages-and-landlord-possession-england-and-wales
crime-in-england-and-wales
crime-statistics-local-reoffending-england-and-wales
crime-statistics-prison-and-probation-england-and-wales
crime-statistics-reoffending-of-adults-england-and-wales
crime-statistics-reoffending-of-juvenilles-england-and-wales
data_gov_uk-datasets
digest-uk-energy-statistics-2008
directgov-central-hottest-pages-monthly
directgov-central-internal-search-terms-monthly
directgov-section-visits-monthly
electricity-consumption-2007
electricity-gas-consumption-2007
energy-consumption-uk-2008
final-energy-consumption-2007
foi-statistics-uk-central-government
fuel-poverty-statistics-2007
gas-consumption-2007
gb-reported-bicycling-accidents
gb-road-traffic-counts
gb-traffic-matrix
greenhouse-gas-emissions-2008
high-level-indicators-energy-use-2006
judicial-and-court-statistics-england-and-wales
laboratory-tests-and-prices
local-authority-carbon-dioxide-emissions-2007
magistrates-courts-statistics-survey-england-and-wales
monthly-energy-prices
monthly-energy-trends
ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig
ni_013_migrants_english_language_skills_and_knowledge
ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration
ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour
ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so
ni_026_specialist_support_to_victims_of_a_serious_sexual_offence
ni_029_gun_crime_rate
ni_031_re-offending_rate_of_registered_sex_offenders
ni_032_repeat_incidents_of_domestic_violence
ni_034_domestic_violence_-_murder
ni_036_protection_against_terrorist_attack
ni_038_drug_related_class_a_offending_rate
ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at
ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an
ni_109_delivery_of_sure_start_childrens_centres
ni_126_early_access_for_women_to_maternity_services
ni_127_self_reported_experience_of_social_care_users
ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment
ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events
ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law
ni_185_co2_reduction_from_local_authority_operations
ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health
ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an
other-fuels-consumption-2006
police-use-firearms-england-wales-2007-2008
prison-end-of-custody-licence-releases-and-recalls-england-and-wales
prison-population-england-and-wales
probation-offender-management-caseload-statistics-england-and-wales
probation-statistics-quarterly-brief-england-and-wales
quality-indicators-energy-data-2007
quarterly-energy-prices
quarterly-energy-trends
road-transport-energy-consumption-2007
sentencing-statistics-england-and-wales
statistics-terrorism-arrests-outcomes-2001-2008
ukba-control-of-immigration-statistics-2008
ukba-control-of-immigration-statistics-2008-supplementary-tables
uk-energy-in-brief-2008
uk-energy-sector-indicators-background-2008
uk-energy-sector-indicators-key-supporting-2008
uk-exportcontrollists
uk-exportcontrol-sanctions
uk-export-control-statistics
uk-glossary-exportcontrol
uk-ipo-offences
weekly-fuel-prices"	1290015776000000	1298486642000000
822	enhancement		Javier de la Cueva <jdelacueva@…>		closed	fixed	Docs: Configuration as site-available instead of site-enabled for apache	"The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net
"	1290107103000000	1290506354000000
830	enhancement		rgrp		closed	fixed	Themes for CKAN	"Already have the basic capacity using *_extras paths for templates and public directory (see http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: http://packages.python.org/Flask-Themes/"	1290760612000000	1311180263000000
834	task		Alexander	ckan-v1.3	closed	worksforme	Searching in CKAN	"Hello.

I've installed stable CKAN v1.1 from PyPI.

I can't find any docs about using CKAN API in order to query packages.

Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible?

Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&open_only=1&downloadable_only=1 and Ckanclient:

result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1})

As result nothing found.

Thanks."	1290769564000000	1291633657000000
836	enhancement		rgrp	ckan-v1.3-sprint-1	closed	fixed	Use site_url config option in templates	"A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html

Instead we should use the site_url config option where necessary.

Cost: 1h"	1291132434000000	1302882808000000
837	enhancement		rgrp	ckan-backlog	new		CKAN integration with freebase gridworks / google refine	"Thread: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

=== Scenario 1 ===

 1. User installs Refine and CKAN extension for refine
 2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
 3. They edit the dataset on Refine
 4. On save (or perhaps as a separate option) they are prompted as to whether they wish  to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)

NB: for the dataset sync back some form of ""CKAN"" storage would be
required (we already have storage.ckan.net running but a closer
integration would be required)

=== Scenario 2 ===

 1. User visits a package on CKAN.net (or another CKAN instance)
 2. There is a button on the page ""View and edit this dataset in Google Refine""
 3. Click button -- ask them if they have Google refine installed
  * Yes: instructions for loading dataset into refine
  * No: load dataset in hosted version of google refine (we could run this)
 4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset."	1291140609000000	1339774605000000
843	enhancement		wwaites		closed	fixed	user list ui for group curation	"  * perhaps we should show the openid as well to distinguish between users with the same name.

  * perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name.

  * also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it."	1291484245000000	1319721601000000
845	enhancement		dread	ckan-v1.3-sprint-1	closed	fixed	Required fields	"Allow a form to specify fields that are required. Currently name and resource url are required.

* Show required fields with asterisks by the input box.
* Resource fields to be included in this."	1291634995000000	1291723492000000
848	enhancement		pudo	ckan-v1.3-sprint-1	closed	fixed	Ability to hide extras fields on package read	for helsinki regional infoshare. 	1291714316000000	1291751275000000
849	defect		pudo		closed	duplicate	Import extra CSS from main template	The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import. 	1291714923000000	1291715179000000
857	defect		dread		closed	invalid	Tests for CLI	Test coverage for ckan.lib.cli is 34%. Need to improve that.	1291724131000000	1311182581000000
870	defect		memespring	ckan-v1.3-sprint-1	closed	fixed	Menu items not selecting correctly	"Since the main menu items have changed (part of the UI redesign) the selected items are not functioning correctly.

Specifically if Home / About or Add Package / Search are selected both menu items are highlighted.

This is because the code assumes only one top-level menu item per controller."	1292243583000000	1294914243000000
871	defect		nils.toedtmann		closed	invalid	Check whether localhost-only exim installtions need upgrading too	"The infamous [http://www.exim.org/lurker/message/20101207.215955.bb32d4f2.en.html exim bug] only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers. 

We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not.

BTW did we already run a rootkit checker like [http://rkhunter.sourceforge.net/ Rootkit hunter] on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could. 
"	1292264117000000	1296340558000000
874	enhancement		pudo	ckan-v1.3-sprint-2	closed	fixed	Extract solr search backend into an extension	Move the solr SearchBackend to a seperate code base, remove any mentions of the CKAN model. 	1292844422000000	1293057151000000
880	defect		dread	ckan-v1.3-sprint-2	closed	fixed	ONS loader tripping up on deleted packages	Deleted packages with the same title and extra fields as an active package trip up the loader. It finds them in the search, but due to ckanclient not using the apikey when you 'get' the package to examine it further, it causes an exception.	1292939379000000	1292945137000000
893	defect		pudo	ckan-v1.4	closed	wontfix	ExtrasField should not overwrite more specific extras	At the moment, ExtrasField cannot be used in conjunction with more specific extra fields, such as TextExtraField or SuggestTextExtraField.	1294135783000000	1298293527000000
895	defect		memespring	ckan-backlog	new		Add version number (or simular) to css/js includes query string	"Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g.

mycssfile.css?v=12345678"	1294343382000000	1339774593000000
896	story		thejimmyg		closed	duplicate	Distributed Data and Syncing Between CKAN Instances	"Based on the ideas in:

 * DistributingChanges
 * SyncingInstances

Merging the tickets #296, #297, #298, #299 into this one.

Related ideas include:

 * Using dcat to exchange data
 * Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance

This should be considered a wishlist item for the timebeing.
"	1294406999000000	1300217375000000
900	defect		dread		closed	fixed	Intermittent error with new revision not having id	"There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision.  We add the uuid earlier to fix this.    https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e.

From David Raznick"	1294659696000000	1294916587000000
907	enhancement		rgrp	ckan-v1.3-sprint-3	closed	fixed	Data previews: preview package resource data in the WUI	"As a user I want to preview a dataset ('resource') before downloading it.

== Implementation ==

This will be implemented as an [wiki:Extensions Extension]

NB: This ticket requires a way to get data out of a remote url (the dataset). We do '''not want to put this data load/streaming into ckan core''' due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

  1. Need a 'data proxy' - see ticket:698 and ticket:888
  2. Preview using javascript in WUI - do this in an extension http://bitbucket.org/okfn/ckanext-dataapi

"	1295261173000000	1297072303000000
909	enhancement		pudo	ckan-backlog	assigned		DCat importer for CKAN	"Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported: 

 * CKANrdf generated exports
 * opengov.se RDF (not really DCat) 
 * Sunlight Nationdatacatalog as harvested by dcat-tools. "	1295265958000000	1346669602000000
924	enhancement		dread	ckan-backlog	new		Search box has no search button	"The search box at the top-right of CKAN's page doesn't have a 'go' button. I feel that a larger percentage of users expect a 'go' or 'search' button on the right-hand side of the box to press to start searching. Techies tend to know the keyboard shortcut of pressing 'carriage-return' but it might be better to follow standard practise on this.

Examples with 'search' button: Internet Explorer, Firefox, Google, Amazon, trac
Examples without: ?"	1295867533000000	1323170436000000
925	defect		dread	ckan-backlog	closed	fixed	Change the search box icon to remove the down arrow	Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed?	1295867593000000	1323168588000000
933	enhancement		kindly	ckan-v1.4	closed	fixed	get rid of self when it should be cls in classmethods	use pylint	1296174526000000	1297348975000000
934	enhancement		kindly		closed	invalid	Key value store for extensions to store data	"We need to allow plugins to store information.

Each plug-in should have its own name space and allow any data to be set.

There are two options.  

 * make a key value table in ckan that will hold random information.
 * use redis (or other key value store)

If in sql suggest schema of form:

 * (namespace, obj_id, key, value [, value_type])
 * value is json"	1296228758000000	1323171047000000
939	enhancement		rgrp		closed	fixed	Notification bar at top of site shown to new users pointing them to simple instructions	"On http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions.

Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better.

== Implementation ==

 * Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing)
 * FAQ/instructions page (use the new wiki?)

== Depends ==

 * ticket:938

== Estimate ==

 * Cost: 1h"	1296399008000000	1323171158000000
947	enhancement		rgrp		closed	duplicate	Move the assignment of roles to a package on package creation to an extension	"At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own)."	1296557478000000	1314031310000000
948	enhancement		dread	ckan-future	assigned		Highlight (to a sysadmin) which packages are deleted	"When a customer logs in as a sysadmin then he/she see all packages, including deleted and pending ones. These are hidden to the average user, but the sysadmin has no idea of this until he clicks on the package and sees at the bottom 'state: deleted'. 

It should be more obvious than that on the search view - an icon, message or crossed-out name to packages are deleted."	1296646369000000	1338206280000000
951	defect		adrian.pohl@…		closed	invalid	Can't add a package to group	"I can't add a package (e.g. http://ckan.net/package/ub-konstanz) to a group (e.g. http://ckan.net/group/bibliographic). It's neither possible when editing  a package (the only group in drop down menu is ""history"") nor on the group page."	1296726886000000	1314031006000000
958	enhancement		kindly	ckan-v1.5	closed	duplicate	ui for new kind field	Ui for related new field in resource (#957)	1296822896000000	1320664462000000
974	defect		dread	ckan-v1.3	closed	fixed	Document site_description	Configuration option 'ckan.site_description' isn't documented	1297342254000000	1297342599000000
976	defect		dread	ckan-v1.3	closed	fixed	Documentation references knowledgeforge.net		1297346649000000	1297346954000000
977	defect		dread	ckan-v1.3	closed	fixed	db_upgrade errors	" * db_upgrade (in ckan/model/__init__) makes call to validate_authorization_setup but forgets about rest of initialisation.

 * it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?)"	1297420558000000	1297420742000000
990	defect		rene.kapusta		closed	fixed	umlauts in title / text	"It's not possible to use umlauts (like ""CKAN Österreich"") in the title / text."	1297801036000000	1311180850000000
991	defect		dread	ckan-v1.3	closed	fixed	Checkbox defaults to True	"Form for new package has CheckboxExtraField checked, when the value is False.
(as used in ckanext-dgu package v3 form)"	1298035175000000	1298037717000000
1020	defect		kindly		closed	fixed	harvesting doc revision table	add revisioning to harvested document table	1299205012000000	1300196215000000
1029	defect		kindly	ckan-v1.4-sprint-3	closed	fixed	synchronous search erroring when harvesting is run.	Errors are caused when harvesting documents. This also makes the count not show up correctly on the ckan search page.	1299768337000000	1301311643000000
1034	defect		dread		closed	duplicate	Flash message cached	" 1. Login
 (shows ckan home page with flash message ""Welcome back xyz"")
 2. Click away, to ""Add package"" say
 3. Click back to ckan ""Home"". 
 It shows ""Welcome back xyz"" again - it shouldn't, I assume?
 "	1299845308000000	1320174353000000
1043	defect		kindly		closed	fixed	stop sqlalchemy message saying at least one scoped seession already present	see summary.	1300310326000000	1300321033000000
1049	defect		dread		closed	invalid	Fix database errors on offener.datenkatalog.at etc.	"We're getting these sorts of exceptions from  http://offener.datenkatalog.at/ like we had from ckan.net, which was due to small errors in migration scripts. Is it worth fixing this database, and in fact all the ckans on eu3 at once?

{{{
WebApp Error: <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint ""group_revision_pkey"" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, ...
}}}

Thoughts James and Rufus?"	1300444912000000	1323169424000000
1057	defect		dread		closed	fixed	JSONP parameter isn't escaped	"{{{
$ curl ""http://127.0.0.1:5000/api/rest/package/annakarenina?callback=<script>jsoncallback""
}}}
gives:
{{{
<script>jsoncallback({""id"": ""c10ebd31-5b45-4f6f-885d-dca9b18caec4"", ""name"": ""annakarenina"", ""title"": ""A Novel By Tolstoy"",
}}}
which could run script code in the client who made the call.

One idea for filtering: http://tav.espians.com/sanitising-jsonp-callback-identifiers-for-security.html
Maybe just better to have a restricted whitelist of characters to be even more sure.

Same as: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/906"	1301078389000000	1329150236000000
1060	defect		dread	ckan-v1.4-sprint-5	closed	fixed	Spreadsheet importer tries to import readonly keys	"e.g. we just added notes_rendered and that is read in as an extra field.
Tests failing in ckanext-importlib

Also related: we are missing lost metadata_created and metadata_modified in the dumps."	1301312210000000	1301312487000000
1064	defect		amercader		closed	duplicate	Remove Workers from ckanext-queue	"The current implementation of Workers in ckanext-queue is broken. Basically the various consume / callback functions expect three arguments (routing_key, operation, payload) when they are in fact receiving only two of them (message_data, message).
This is fairly easy to fix, but the question is if Workers add an extra complexity to use the messaging library directly."	1301417891000000	1323169787000000
1082	defect		johnlawrenceaspden		closed	fixed	language changes behave strangely	"Set language to Greek, flash message says 'Language set to: English', but page is now about half in Greek.

Set language back to English causes server error:

AttributeError: 'NoneType' object has no attribute 'path'

Module ckan.controllers.error:29 in document          view
>>  if original_request.path.startswith('/api'):

However going to a new page reveals that it's back to English"	1302541989000000	1315917217000000
1087	enhancement		wwaites	ckan-sprint-2011-11-21	closed	fixed	version and contact info api call	"a simple api call that returns data like this:

{{{
{ ""version"": ckan_software_version,
  ""contact"": { ""name"": ""Some Admin"", ""mbox"": ""admin@exmaple.org"" },
  ""description"": ""Site Description"",
  ""url"": ""http://canonical.name.ckan.net/""
}}}"	1302628944000000	1320866159000000
1088	defect		wwaites	ckan-v1.4	closed	fixed	content-type autonegotiation is wonky	"in ckan/controllers/package.py around line 130 it does some strange things...

perhaps replace with https://github.com/wwaites/autoneg

and handle redirection of these content types:
{{{
application/rdf+xml
application/turtle
text/plain
text/x-graphviz
}}}
"	1302630261000000	1303035487000000
1091	defect		johnlawrenceaspden		closed	wontfix	usernames of users logged in using open ids are strange	"If I use my gmail openID to log into a CKAN instance, then my username is:

https://www.google.com/accounts/o8/id?id=AItOawnduohQ5RgXdPJKHiq-SIPbvCBqUaERuEQ

This seems a bit odd."	1302701460000000	1323102767000000
1099	defect		johnlawrenceaspden		closed	wontfix	strange interactions between two browsers while playing with authz groups	"While playing with the authorization groups, trying to design tests, I found that it was necessary to log in as two different users with two different browsers. Often actions of one user would cause server errors in the other user's browser.

I don't have a reproducible test case, but it happens fairly often so it shouldn't be too difficult to get one."	1303380824000000	1324057106000000
1100	defect		dread	ckan-v1.4-sprint-6	closed	fixed	Get buildbot running on ckan branches	Need some changes to pip-requirements files in release branches.	1303385267000000	1303406103000000
1101	enhancement		sebbacon	ckan-backlog	new		Integrate googlanalytics into site nav	"There's a stats plugin (e.g. at http://trac.ckan.org/ticket/832).

Output from the googleanalytics plugin should append to that page, if the stats plugin is present.

Possibly the stats plugin and the googleanalytics plugin should be merged?

Finally, if the stats plugin is active, then a link to the stats page should be added to the main site footer."	1303393926000000	1339774582000000
1102	defect		johnlawrenceaspden		closed	duplicate	searching broken in development setup	"With the default test data created by

paster db clean
paster db init
paster create-test-data

going to the front page shows two recently changed packages
A Wonderful Story
A Novel by Tolstoy

But none of those words ""Wonderful"", etc produce search hits.
In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.
"	1303491912000000	1303744552000000
1103	defect		johnlawrenceaspden		closed	duplicate	searching broken in development setup	"With the default test data created by

paster db clean
paster db init
paster create-test-data

going to the front page shows two recently changed packages
A Wonderful Story
A Novel by Tolstoy

But none of those words ""Wonderful"", etc produce search hits.
In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.
"	1303494538000000	1303744575000000
1105	defect		nils.toedtmann		closed	invalid	test ticket, please ignore	.	1303508261000000	1303508330000000
1115	defect		johnlawrenceaspden		closed	wontfix	can have two authzgroups with the same name	"If you've got edit permission on an authzgroup, then you can change its name to be the same as another existing authzgroup.

This causes some strange UI effects at worst, and probably causes worse problems somewhere else.

Is there any reason why changing the names of existing authzgroups should be allowed? And if so, name collisions should presumably be guarded against in both the name-changing and creation functions"	1304085120000000	1324054704000000
1118	defect		johnlawrenceaspden		closed	invalid	tests are testing something other than the behaviour seen in the browser	"I'm finding that if I try to take an action with insufficient credentials from a test then I often (but not always) get a 401 error, whereas in the browser I get redirected to the login page.

It's a bit worrying that the program in its test environment doesn't behave like it does in the browser."	1304093017000000	1311174062000000
1120	enhancement		tsm	ckan-backlog	new		Atom feeds of each tag	"Tags could/should have an Atom feed. This would mean that every edit to relevant packages could be easily monitored. See [1].


[1] http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001162.html"	1304309285000000	1339774568000000
1121	enhancement		dread		closed	wontfix	JSON extras appear in package edit form mangled	"It is possible to use the CKAN API to insert JSON format data into package extra values, but this data is displayed in the package edit form.

Example: Package http://ckan.net/package/hbz_unioncatalog in the API the extra value as a list:
{{{
""extras"": {""publishingInstitution"": [""http://lobid.org/organisation/DE-605"", ""http://lobid.org/organisation/DE-290""...
}}}
yet when you edit the package it loses all the quotes and brackets: http://ckan.net/package/edit/hbz_unioncatalog
{{{http://lobid.org/organisation/DE-605http://lobid.org/organisation/DE-290...
}}}
so when you save the package, the list is mangled into a bad string."	1304367504000000	1307358426000000
1127	CREP		sebbacon		closed	fixed	CREP0001: Formalise new feature discussion and definition using CREPs	"'''Proposer:''' Seb Bacon[[BR]]
'''Seconder:''' Rufus Pollock[[BR]]
[[BR]]

== Abstract ==

When adding major new features to CKAN, a longer, more formal discussion 
will improve software design quality and documentation, better engage
the wider community, and ensure the core team are up to date with
latest developments.  

I propose a formal process (CREP -- CKAN Revision and Enhancement Proposal) for making this happen.

== The Problem ==

The current workflow for introducing major new features into CKAN is
very informal, typically based around one person's great idea, which
they've discussed with one or two other people in the team.  The
originator of the idea is typically the only person with access to 
all the input they've had through such discussions.  Often, the only 
location of this information is  in that person's head.

However, there is a lot of experience embodied in the CKAN community
which should be drawn on before making large design decisions.  This
will lead to better software.  Additionally, building consensus in the
community around a proposal before implementation ensures positive
community engagement and buy-in to new features, making them more
likely to be a success.

We aren't great at documenting new features.  Documentation after
coding is complete is an unrewarding experience for most programmers.
Requiring skeleton documentation before code is written is a good
discipline that can form the basis of better documentation in the
future (e.g. by a writer rather than a programmer).

== Specification ==

Minor features don't require a CREP, and can just be entered in the
issue tracking system as a bug or feature.  As a rule of thumb, a
feature is major if it will take more than a day to implement, or is
likely to involve matters of opinion in its design.

A developer may decide that a CREP is too formal and long-winded. The 
decision to write a CREP is at at their discretion; however, new 
features MUST always be proposed via email, even if this is just a 
couple of sentences.

If a feature requires a CREP, the proposer should find a seconder
for their idea.  This sanity check step happens before a CREP is
written to ensure at least the possibility of consensus on the CREP.

Next the proposer should write a CREP, starting by copying and pasting
the [http://wiki.ckan.net/CKAN_Revision_and_Enhancement_Proposals template on the wiki] into a new Trac ticket.  This will be with a status of ""new"" and Type of ""CREP"".  The proposer should notify
the [http://lists.okfn.org/mailman/listinfo/ckan-dev ckan-dev] mailing
list, and possibly the
[http://lists.okfn.org/mailman/listinfo/ckan-discuss ckan-discuss]
list for less technical CREPs.

The draft can be discussed via email, verbally, or via the trac
ticket.  In any case, it is the proposer's responsibility to keep the
CREP updated to reflect the current consensus.

Once consensus has been reached, the ticket should be marked with the
""accepted"" status and assigned to a CKAN release milestone.

When an accepted CREP has been implemented, it should be resolved as
""fixed"".

If no consensus can be reached on a draft CREP, or for some reason an
accepted CREP doesn't get completed, it should be marked as or ""wontfix"".

If a completed CREP becomes obsolete, it should be marked as ""invalid"", 
with a note pointing to the obsoleting ticket(s)

== Why do it this way == 

Given the distributed nature of the core team plus other volunteers,
some kind of written procedure is necessary to ensure a fully
documented and discussed proposal.

The idea of ""Enhancement Proposals"" which can be semi-formally
proposed and discussed prior to implementation is common in the Open
Source world ([http://www.python.org/dev/peps/pep-0001/ PEPs],
[http://dep.debian.net/ DEPs],
[http://plone.org/documentation/glossary/plip PLIPs], to name three).

Existing historic proposals exist, called CEPs.  The proposed system
is called CREP (CKAN revision or enhancement proposal) to disambiguate
it from the legacy proposals, and from the delicious fungus ''Boletus
Edulis''.

Giving a formal structure to the proposal is useful as it gives the
community a means to identify a CREP that's not had sufficient thought
or discussion.  An informal email thread can easily be lost and
important questions (such as backwards compatibility) overlooked.  The
use of the proposed template empowers any community member to ask the
proposer to expand on rationale, deliverables, etc.

The structure chosen is somewhere between Debian's and Plone's.  It
aims to give a structure to the debate, a clear start at
documentation, and also prompt some thinking about implementation and
timescales.

All this policy about structure should not be construed as mandatory.  
In particular, the later fields in the CREP template regarding 
Implementation Plan may be omitted if the author doesn't find them 
helpful.

Some projects (e.g. Debian) keep their enhancement proposals in a
versioning repository; others (e.g. Plone) keep them in an issue
tracking system.  Trac is proposed for CKAN because we already use it
for small feature proposals and for team planning.  It seems unlikely
that change tracking on an individual CREP will be useful; a CREP that
changes sufficiently from its original form should probably be marked
""obselete"" and a new CREP started.  Using an issue tracking system also
means we can easily track CREPs by state.

== Backwards Compatibility ==

Some [https://bitbucket.org/okfn/ceps/src/76b274888bcf/cep/ legacy 
enhancement proposals], called CEPs, have previously been started.

They are currently all marked as ""active"".  Any which require
discussion should be altered by the proposer to match the new CREP
specification and submitted to trac.  The original CEP should be
updated with a banner at the top pointing a reader to the new CREP.

Any that are now obselete should be clearly marked as such in a banner
at the top, pointing a reader to the trac for new CREPs.

== Implementation plan ==

=== Deliverables ===

 * This CREP, agreed
 * Support for proposed statuses in Trac
 * Canned reports for listing CREPs in Trac

=== Risks and mitigations ===

 * That this CREP is agreed, but rarely acted on.  This risk can be
   mitigated by nominating a CREP champion in the community or core
   team, whose job it is to say ""where's the CREP for that?"" and
   generally own the quality of CREPS

=== Participants ===

Seb Bacon: as current Documentation Czar (May 2011), responsible for 
ensuring CREPs are up to date.

=== Progress ===

This document is the entire proposal.
"	1304601313000000	1305622850000000
1129	CREP		kindly	ckan-v1.5	closed	fixed	CREP0002: Moderated  Edits	"'''Proposer:''' David Raznick[[BR]]

== Abstract. ==

We are trying to achieve these goals.

 * To get people involved with making edits to CKAN metadata. 
 * To have an ownership model as to who can moderate and validate these changes
 * To not put too huge a burden on these owners.

In order to achieve this, a feature which lets anyone edit a package but only let the moderator/owner accept it.  The moderator should be able to look at a list of changes and accept the ones that

This cep is not about 'if' we need such a feature, it is about 'how' we go about implementing it. Another cep may needed for the 'if' case.

== The Problem ==

We need the following to be possible.

 * Storing revision of objects that are not the current active one.
 * A way of the user viewing past revisions.
 * Accessing not only the history of a particular object but also of related objects at that time. i.e If a resource related to a package changes we need a way to see this when looking at the package.
 * A robust way of doing this in the face of database schema changes.
 * Make sure database queries are quick.

== Solutions. ==

1.  Store the whole dictization of the package and all its related objects every time you change anything in its dictized representation and only save to the database proper if accepted.

 Pros
 * Easy to implement, we already have a preview which makes the dictized form of a package without actually saving it.  This will just need to be persisted in some way.
 * Fast retrieval.
 * Potential to store a branching revision tree of changes.

 Cons
 * No easy way to remake the dictized packages historically or if there is an there a change in the way we represent packages, i.e schema changes.
 * Will only work for the particular objects we decide to store these changes for.
 * Stores a lot of repeated information


2.  Write specialized queries for every read of the database looking only at the revision tables.

This method requires there to be a change in the way we use VDM, so that we manage statefulness ourselves. We will need to add other states such as 'waiting for approval'.

 Pros

 * No specialized storage required
 * Only need to change queries when schema changes
 * Can be made to work easily for other objects

 Cons

 * Slower query time on read, as even looking at the last active package will need to do a fairly complicated query.

== Implementation details. ==

1.
    A new table with columns  id, user, package_id, timestamp, revision_id, parent_id, dictized_package.
    revision_id should be null unless it is actually persisted to the database.  parent_id is the id that this package_dict was changed from.

    We could store only the diffs of the dictized_package as long as we assure that everything inside the json is stably sorted, this will make getting the historical data out slower.

    Getting out the history of the dictized packages is an intensive task, as it will require replaying the whole history of all the changes and creating the dict for each change.  This re-caching will need to be redone for every change we make to dictized representation of a package.

2.
    Every normal packages read needs to look at the revision table to see the last accepted change in the dictized representation of the package. 
    We also need to way to get what the dictized representation of the package was like at any point of its revision history.  This querying is non-trivial in sql.

=== Participants ===

David Raznick to do it.

== Progress. ==

Decided to go with option 2. However we will change the revisioning system to be like the schema attached.
This gets rid of difficult querying problems caused by querying the revision tables by adding an end date, meaning you can do range queries.

The better and more normalized version of a revisioning system is outlined [https://docs.google.com/drawings/d/1Y7nMgVsrs081Pame2RdbZHlCAlV33ddTZ8VAsab1j-0/edit?hl=en_GB&authkey=CJfd8vsB]. 
We will be a step closer to that, with this change, but we will keep the current vdm more or less, intact.

"	1304851498000000	1325268100000000
1130	enhancement		lucychambers		assigned		First time users	Send users to FAQ first time on CKAN	1304938761000000	1340633514000000
1132	defect		johnlawrenceaspden	ckan-backlog	closed	invalid	test_authz doesn't run	"Trying to run the tests in test_authz.py with

$ nosetests --ckan ckan/tests/functional/test_authz.py
results in no tests being run:

----------------------------------------------------------------------
Ran 0 tests in 0.840s

OK (SKIP=3)"	1304966923000000	1307352675000000
1133	defect		johnlawrenceaspden		closed	worksforme	command line rights manipulation doesn't work	"It appears that the command

$ paster rights add russianfan admin warandpeace

has no effect, even though

$ paster rights remove russianfan admin warandpeace

works fine. This may be specific to something I've done, could someone confirm?

If it's the case more generally, then I'm assuming this behaviour is untested? Tests should probably be added."	1305054948000000	1324057072000000
1134	CREP		amercader	ckan-backlog	new		CREP0003: Description and Configuration of Harvesters	"'''Proposer''': Adrià Mercader

== Abstract ==

The new harvester interface allows to create harvesters for different
sources, but right now harvesters don't have many ways to describe and
configure themselves. We need a way of allowing them to:

    * Expose their type and other details so they can be used internally
      and on the UI.
    * Define configuration settings for particular harvester instances.


== The Problem ==

=== Harvester description ===

The current UI for adding and editing harvest sources is the same used
in ckanext-dgu, and thus the 3 harvester types used in DGU to harvest
various GEMINI realted sources are hardcoded in the form. The form will 
be migrated to a DGU-independent one, so we need the harvesters to 
provide all the necessary data. There is a current {{{get_type}}} method
that returns the harvester type, but for make it compatible with the DGU
forms, it returns a machine-readable string (e.g. ""CSW Server""), making
it error prone.


=== Arbitrary configuration ===

In the current implementation, when the harvest process is started,
ckanext-harvest looks for all the available plugins that implement the 
{{{IHarvester}}} interface and calls the appropiate methods for the
current stage ({{{gather_stage}}},{{{fetch_stage}}},{{{import_stage}}}).
At these stages, harvesters have no way of applying arbitrary
configuration options, so all harvesters of the same type behave on the
same way.
For instance, the CKAN harvester needs a way to define the API version
to use when harvesting remote instances (Right now, the version 2 is
hardcoded on the code).


== Specification ==

=== Harvester description ===

Harvesters will need to provide the following information so the UI form
can be built:

    * name: machine-readable name (e.g. ""waf""). This will be the value
      stored in the database, and the one used by ckanext-harvest to
      call the appropiate harvester.
    * title: human-readable name (e.g. ""Web Accessible Folder (WAF)"").
      This will appear in the form's select box.
    * description: a description of what the harvester does (e.g. ""A Web 
      Accessible Folder (WAF) displaying a list of GEMINI 2.1 
      documents""). This will appear on the form as a guidance to the
      user.

The way to provide it will be an {{{info}}} method that all harvesters
must implement, which will return a dictionary with the previous
elements:

{{{
    {
        'name': 'csw',
        'title': 'CSW Server',
        'description': 'A server that implements OGC's Catalog Service 
                        for the Web (CSW) standard'
    }
}}}

=== Arbitrary configuration ===

As different harvesters will have very different needs, we need to
provide a way to persist arbitrary configuration flags for each harvest
source. The more flexible way given the current architecture in my
opinion would be to store the configuration options as a JSON encoded
object as a property of the harvest source (There already is an unused
DB field called {{{config}}} in the database) (Maybe using JsonType?).

This will mean adding an extra field in the harvest source form to allow
entering the configuration. This could be just a simple text field where
users enter the JSON encoded object or a more clever mechanism (i.e an 
""Add a configuration flag"" link that adds two new text fields for the
key and value for each flag, and a mechanism to later build the JSON 
object). In any case, this should probably be hidden in an ""Advance
options"" section.


== Why do it this way == 

=== Harvester description ===

The {{{info}}} method would provide a single point to get all the
information related to the harvester, and future properties could be 
added to the dictionary returned without having to modify the interface.


=== Arbitrary configuration ===
There is an already existing {{{config}}} field in the database, so we
won't need to change the model. 
Harvesters could access the config object at any of the stages. Of
course they could provide default values in their implementations so
users don't need to enter them everytime.

== Implementation plan ==

=== Deliverables ===

=== Risks and mitigations ===

The highest risk on the harvesters {{{info}}} method side is that 
harvester implementation don't offer one of the necessary properties
(namely name and title). This could fire a warning when showing the 
UI form or using the CLI.

=== Participants ===

Adrià Mercader to do it.

=== Progress ===

None yet."	1305108868000000	1339774554000000
1138	enhancement		johnlawrenceaspden		closed	invalid	minor navigations behave inconsistently	"For Authorization Groups, if you have admin privileges you see view, edit and authz tabs, and if you don't have the necessary privileges you only see the view tab.

For Packages, you see all tabs whatever your permissions, so there's a link you can click on which will redirect you to the login page."	1305279888000000	1316965357000000
1140	defect		dread	ckan-v1.5	closed	fixed	Adding the package to the group is not search indexed	"To reproduce:

 1. {{{paster db init}}}
 2. {{{paster create-test-data}}}
 3. In Web UI: create new group 'tilo', which includes package 'annakarenina'
 4. {{{curl http://localhost:5000/api/search/package?groups=tilo}}}  results in 0 results (WRONG)
 5. {{{paster search-index rebuild}}}
 6. {{{curl http://localhost:5000/api/search/package?groups=tilo}}}  results in 1 result"	1305718290000000	1312537257000000
1141	CREP		johnglover	ckan-backlog	closed	fixed	[super] Moderated Edits User Interface	"'''Proposer''': John Glover[[BR]]
'''Seconder''': James Gardner

== Abstract ==

We are trying to achieve these goals:
 * To get people involved with making edits to CKAN metadata.
 * To have an ownership model as to who can moderate and validate these changes
 * To not put too huge a burden on these owners.

This feature allows anyone to edit a package and create a new revision, but requires an owner/moderator to approve a revision before it is are made ""official"".

There have been a lot of discussions around the revisioning system side of this ticket (CREP 0002) and I think these are now largely resolved. We now want to discuss the user interface.

== The Problem ==

We require the following functionality:

 * Allow a group of changes to be stored as a new revision.
 * Allow a linear stack of ""community"" revisions. 
 * Provide a way for the editor and moderator to compare previous revisions to the current one.
 * When a moderator approves a change it creates a new revision flagged ""moderated"" (this is analogous to a merge commit)
 * Provide a way for the editor and moderator comment on revisions if necessary.

Extra features:
 * Need a way to summarise the changes (as part of the preview perhaps)
 * Sysadmin needs to purge a revision completely

== Specification ==



=== UI/UX ===

UI Mockup:

[[Image(ModeratedEdits2.png)]]


Revisions:

 * Revisions are per package rather than per field.
 * Internally CKAN has separate revisions for resources, extras and package metadata.  From a user's point of view this could be confusing to expose, so everything that they see on a package form when they hit save is a single revision.

On the Edit page:

  * We have a panel on the right, listing all the revisions with the current moderated one selected. Moderated revisions are highligted in some way (red and bold?).
  * The values displayed in the form are by default populated from the latest revision (whether community or moderated)
  * Under each field is a ""shadow"", showing the value of the field in the revision selected in the panel, if it is different from the value in the field. By default the shadow values are populated from the latest moderated revision which is the one selected in the revision panel by default too.
  * When you change the value of a field, a shadow may appear or disappear accordingly. If they disappear a box saying that they are the same replaces it
  * If you want to edit values from a previous revision, you first select that revision to get the shadows populated. There is a button named ""Replace fields with values from this revision"" under the revision list. You click this, a warning pops up and then you say ""Yes"". You then select the moderated revision again.
  * We also allow package comments the same way as the todo extension works at the moment. Additionally, we need to be able to differentiate between what the moderator wrote and what a community member wrote, and so we may need to make a small change to the todo extension to facilitate this.
  * In addition to package comments, each revision will have a revision log (analogous to a commit message).

=== Technical Details ===

 * This CREP will result in a new CKAN extension.
 * It depends heavily on the new revisioning system (CREP0002), some of the details of which are yet to be finalised. 
 * This CREP therefore requires working closely with David Raznick to come up with an API that the UI AJAX calls can use.
 * We will then use suitable test data to mimic these API calls until CREP0002 is ready.

== Why do it this way == 

This hopefully provides a clear and consistent mechanism allowing both a community member to make new revisions and a moderator to view and approve revisions, with largely the same UI/UX.

== Implementation plan ==

=== Deliverables ===

A new CKAN extension, consisting of:

 * Code: Python, HTML, CSS, Javascript
 * Unit tests
 * Localization
 * Documentation

=== Participants ===

John Glover to do it.

=== Progress ===

John has implemented the bulk of this UI. Just some things to tidy up before it is complete:

 * Genshi stream filters to be updated with CKAN 1.5 / 1.5.1 templates
 * history_ajax / read_ajax to be replaced with calls to Action API (or Util REST API)

I've split these two off into a new ticket #1604.

=== Related Progress ===

The Todo extension is written and available at: [https://bitbucket.org/johnglover/ckanext-todo].

In the section 'The Problem', under extra features, we mention a need for the sysadmin to be able to purge a revision already. This is already done.


=== See also ===


#1129 Backend work
"	1305721003000000	1325352507000000
1144	enhancement		timmcnamara	ckan-backlog	new		Support DSPL	"DSPL, the Dataset Publishing Language, is being promoted by Google for its ""Google Public Data Explorer"" system. It is an XML format with metadata.

The format is described on the [http://code.google.com/apis/publicdata/docs/developer_guide.html developer docs ofthe Google Code site].

Google provides a [http://code.google.com/apis/publicdata/docs/dsplgen.html Python script which reads CSV data and generates DSPL]


Sample from http://code.google.com/apis/publicdata/docs/dspl_sample.html:

{{{
#!xml

<?xml version=""1.0"" encoding=""UTF-8""?>
<dspl xmlns=""http://schemas.google.com/dspl/2010""
    xmlns:geo=""http://www.google.com/publicdata/dataset/google/geo""
    xmlns:geo_usa=""http://www.google.com/publicdata/dataset/google/geo/us""
    xmlns:time=""http://www.google.com/publicdata/dataset/google/time""
    xmlns:quantity=""http://www.google.com/publicdata/dataset/google/quantity""
    xmlns:entity=""http://www.google.com/publicdata/dataset/google/entity"">

  <import namespace=""http://www.google.com/publicdata/dataset/google/time""/>
  <import namespace=""http://www.google.com/publicdata/dataset/google/quantity""/>
  <import namespace=""http://www.google.com/publicdata/dataset/google/entity""/>
  <import namespace=""http://www.google.com/publicdata/dataset/google/geo""/>
  
  <info>
    <name>
      <value>My statistics</value>
    </name>
    <description>
      <value>Some very interesting statistics about countries</value>
    </description>
    <url>
      <value>http://www.stats-bureau.com/mystats/info.html</value>
    </url>
  </info>

  <provider>
    <name>
      <value>Bureau of Statistics</value>
    </name>
    <url>
      <value>http://www.stats-bureau.com</value>
    </url>
  </provider>

  <topics>
    <topic id=""geography"">
      <info>
        <name><value>Geography</value></name>
      </info>
    </topic>
    <topic id=""social_indicators"">
      <info>
        <name><value>Social indicators</value></name>
      </info>
      <topic id=""population_indicators"">
        <info>
          <name><value>Population indicators</value></name>
        </info>
      </topic>
      <topic id=""poverty_and_income"">
        <info>
          <name><value>Poverty & income</value></name>
        </info>
      </topic>
      <topic id=""health"">
        <info>
          <name><value>Health</value></name>
        </info>
      </topic>
    </topic>
  </topics>

  <concepts>
    <!-- As noted in the tutorial, this concept should extend quantity:amount.-->
    <concept id=""population"">
      <info>
        <name>
          <value>Population</value>
        </name>
        <description>
          <value>Size of the resident population.</value>
        </description>
      </info>
      <topic ref=""population_indicators""/>
      <type ref=""integer""/>
    </concept>

    <!-- This country concept is defined for educational purposes only. A country
    concept exists in the Google geo dataset. See:

    http://code.google.com/apis/publicdata/docs/canonical/geo.html --> 
    <concept id=""country"" extends=""geo:location"">
      <info>
        <name>
          <value>Country</value>
        </name>
        <description>
          <value>My list of countries</value>
        </description>
      </info>
      <type ref=""string""/>
      <property id=""name"">
        <info>
          <name><value xml:lang=""en"">Country name</value></name>
          <description>
            <value xml:lang=""en"">The official name of the country</value>
          </description>
        </info>
        <type ref=""string""/>
      </property>
      <table ref=""countries_table""/>
    </concept>

    <!-- This US state concept is defined for educational purposes only. A US state
      concept exists in the Google geo US dataset. See:

      http://code.google.com/apis/publicdata/docs/canonical/geo.us.html --> 
    <concept id=""state"" extends=""geo:location"">
      <info>
        <name>
          <value>State</value>
        </name>
        <description>
          <value>US states</value>
        </description>
      </info>
      <type ref=""string""/>
      <property concept=""country"" isParent=""true""/>
      <table ref=""states_table""/>
    </concept>

    <concept id=""gender"" extends=""entity:entity"">
      <info>
          <name>
          <value>Gender</value>
        </name>
        <description>
          <value>Gender, Male or Female</value>
        </description>
        <pluralName><value>Genders</value></pluralName>
        <totalName><value>Both genders</value></totalName>
      </info>
      <type ref=""string""/>
      <table ref=""genders_table""/>
    </concept>

    <concept id=""unemployment_rate"" extends=""quantity:rate"">
      <info>
        <name>
          <value>unemployment rate</value>
        </name>
        <description>
          <value>The percent of the labor force that is unemployed, not seasonally
            adjusted.</value>
        </description>
        <url><value>http://www.bls.gov/cps/cps_htgm.htm</value></url>
      </info>
      <topic ref=""social_indicators""/>
      <type ref=""float""/>
      <attribute id=""is_percentage"">
        <type ref=""boolean""/>
        <value>true</value>
      </attribute>
    </concept>

  </concepts>

  <slices>
    <slice id=""countries_slice"">
      <dimension concept=""country""/>
      <dimension concept=""time:year""/>
      <metric concept=""population""/>
      <table ref=""countries_slice_table""/>
    </slice>

    <slice id=""states_slice"">
      <dimension concept=""state""/>
      <dimension concept=""time:year""/>
      <metric concept=""population""/>
      <metric concept=""unemployment_rate""/>
      <table ref=""states_slice_table""/>
    </slice>

    <slice id=""countries_gender_slice"">
      <dimension concept=""country""/>
      <dimension concept=""gender""/>
      <dimension concept=""time:year""/>
      <metric concept=""population""/>
      <table ref=""countries_gender_slice_table""/>
    </slice>

  </slices>

  <tables>
    <table id=""countries_table"">
      <column id=""country"" type=""string""/>
      <column id=""name"" type=""string""/>
      <column id=""latitude"" type=""float""/>
      <column id=""longitude"" type=""float""/>
      <data>
        <file format=""csv"" encoding=""utf-8"">countries.csv</file>
      </data>
    </table>

    <table id=""countries_slice_table"">
      <column id=""country"" type=""string""/>
      <column id=""year"" type=""date"" format=""yyyy""/>
      <column id=""population"" type=""integer""/>
      <data>
        <file format=""csv"" encoding=""utf-8"">country_slice.csv</file>
      </data>
    </table>

    <table id=""states_table"">
      <column id=""state"" type=""string""/>
      <column id=""name"" type=""string""/>
      <column id=""country"" type=""string"">
        <value>US</value>
      </column>
      <column id=""latitude"" type=""float""/>
      <column id=""longitude"" type=""float""/>
      <data>
        <file format=""csv"" encoding=""utf-8"">states.csv</file>
      </data>
    </table>

    <table id=""states_slice_table"">
      <column id=""state"" type=""string""/>
      <column id=""year"" type=""date"" format=""yyyy""/>
      <column id=""population"" type=""integer""/>
      <column id=""unemployment_rate"" type=""float""/>
      <data>
        <file format=""csv"" encoding=""utf-8"">state_slice.csv</file>
      </data>
    </table>

    <table id=""genders_table"">
      <column id=""gender"" type=""string""/>
      <column id=""name"" type=""string""/>
      <data>
        <file format=""csv"" encoding=""utf-8"">genders.csv</file>
      </data>
    </table>

    <table id=""countries_gender_slice_table"">
      <column id=""country"" type=""string""/>
      <column id=""gender"" type=""string""/>
      <column id=""year"" type=""date"" format=""yyyy""/>
      <column id=""population"" type=""integer""/>
      <data>
        <file format=""csv"" encoding=""utf-8"">gender_country_slice.csv</file>
      </data>
    </table>
  </tables>

</dspl>
}}}
"	1305763609000000	1339774517000000
1145	enhancement		timmcnamara	ckan-backlog	new		Support the Handle System	"The Handle System is an initiative to provide persistent references for resources. That is, it's basically a proxy system for preventing link rot.

Its documentation is here: http://www.handle.net/. Servers running CKAN could host a ""Local Handle Service"", which redirects a hash of a resource to an actual URL.

Some suggested use cases:

 - Researcher would like to cite where data came from
 - Agencies would like to have a way to prevent vendor
   lock-in from CKAN if they decide to move to another
   platform"	1305764775000000	1339774502000000
1165	enhancement		nils.toedtmann	ckan-future	new		Add multi-site support to ckan	"Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3). 

That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's ""<VirtualHost>"", or tracd). Depending on the ""Host:"" header in the HTTP1.1 request, it would choose which local ckan ini file to obey.

I see two ways to constitute the map hostname-to-ini-file map:
 * ckan reads a set of ini files, and each ini file declares which servers names it is responsible for
 * In a global ini file, there are directives mapping servernames to ini files.

In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc."	1306413667000000	1339774466000000
1167	task		nils.toedtmann		closed	fixed	Create a standard CKAN system image for Amazon EC2 (AMI)	"We should create a public AMI with CKAN pre-installed and configured such that users can easily create their own EC2 machine with a running CKAN to play with.

There are three phases:
 1. [nils] Deploy an empty EC2 instance to become the CKAN image master instance
 2. Install a CKAN and give it a standard configuration.
 3. [nils] Create a AMI from the CKAN image master instance and publish it.

I am happy to do first and last. Who is installing and configuring CKAN?

Unfortunately AMIs are specific to region, architecture and storage type. We cannot maintain too many images, so a number of choices have to be made:
 * Which distribution/version? '''Ubuntu 10.04 LTS'''
 * Which architecture/instance-type? I suggest 64-bit/t1.micro
 * Which region? I suggest us-east-1 and maybe eu-west-1
 * Which storage type? '''EBS''' (way easier to make an AMI from than instance-store)
 * Install CKAN from deb packages via mercurial/virtualenv? I assume the latter because the AMI is targeted to developers?


----
[http://alestic.com/2009/06/ec2-ami-bundle 1]
[http://uec-images.ubuntu.com/releases/10.04/release/ 2]"	1306423016000000	1311178516000000
1177	defect		fccoelho		closed	fixed	routes.util.GenerationException	" Hello,

I have done a recent update to mu ckan install, and I am now getting this error for all my packages:
{{{
URL: http://ckan.emap.fgv.br/package/dengue-net
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
          
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
              
               response = self.app(environ, start_response)
              
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<         
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
              
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
          
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']   
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
              
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                  
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
          
           def _inspect_call(self, func):
>>  return func(**args)
Module ?:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__(""logging"").getLogger(""proxy_cache"")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
      
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.package:208 in read
<<          #render the package
               PackageSaver().render_package(c.pkg)
               return render('package/read.html')
      
           def comments(self, id):
>>  return render('package/read.html')
Module ckan.lib.base:74 in render
<<     
           return cached_template(template_name, render_template, cache_key=cache_key,
                                  cache_type=cache_type, cache_expire=cache_expire)
                                  #, ns_options=('method'), method=method)
>>  cache_type=cache_type, cache_expire=cache_expire)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:63 in render_template
<<              stream = item.filter(stream)
              
               return literal(stream.render(method=method, encoding=None))
          
           if 'Pragma' in response.headers:
>>  return literal(stream.render(method=method, encoding=None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
      
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out=out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:339 in __call__
<<          for filter_ in self.filters:
                   stream = filter_(stream)
               for kind, data, pos in stream:
                   cached = cache_get((kind, data))
                   if cached is not None:
>>  for kind, data, pos in stream:
Module genshi.output:670 in __call__
<<          _gen_prefix = _gen_prefix().next
      
               for kind, data, pos in stream:
                   output = cache_get((kind, data))
                   if output is not None:
>>  for kind, data, pos in stream:
Module genshi.output:771 in __call__
<<          push_text = textbuf.append
               pop_text = textbuf.pop
               for kind, data, pos in chain(stream, [(None, None, None)]):
      
                   if kind is TEXT:
>>  for kind, data, pos in chain(stream, [(None, None, None)]):
Module genshi.output:586 in __call__
<<      def __call__(self, stream):
               prev = (None, None, None)
               for ev in stream:
                   if prev[0] is START:
                       if ev[0] is END:
>>  for ev in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
      
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.markup:378 in _match
<<                      for event in self._match(self._flatten(template, ctxt,
                                                                  **vars),
                                                    ctxt, start=idx + 1, **vars):
                               yield event
>>  ctxt, start=idx + 1, **vars):
Module genshi.template.markup:327 in _match
<<                      break
      
               for event in stream:
      
                   # We (currently) only care about start and end events for matching
>>  for event in stream:
Module genshi.template.base:565 in _flatten
<<                  elif kind is EXPR:
                           result = _eval_expr(data, ctxt, vars)
                           if result is not None:
                               # First check for a string, otherwise the iterable test
>>  result = _eval_expr(data, ctxt, vars)
Module genshi.template.base:277 in _eval_expr
<<      if vars:
               ctxt.push(vars)
           retval = expr.evaluate(ctxt)
           if vars:
               ctxt.pop()
>>  retval = expr.evaluate(ctxt)
Module genshi.template.eval:178 in evaluate
<<          __traceback_hide__ = 'before_and_this'
               _globals = self._globals(data)
               return eval(self.code, _globals, {'__data__': data})
>>  return eval(self.code, _globals, {'__data__': data})
Module ?:16 in <Expression u""h.subnav_link(c, h.icon('comments') + _('Comments & Questions'), controller='package', action='comments', id=c.pkg.name)"">
<<        </li>
             <li py:if=""g.has_commenting"">
                 ${h.subnav_link(c, h.icon('comments') + _('Comments &amp; Questions'), controller='package', action='comments', id=c.pkg.name)}
             </li>
             <li>${h.subnav_link(c, h.icon('page_white_stack') + _('History'), controller='package', action='history', id=c.pkg.name)}</li>
>>  ${h.subnav_link(c, h.icon('comments') + _('Comments &amp; Questions'), controller='package', action='comments', id=c.pkg.name)}
Module ckan.lib.helpers:126 in subnav_link
<<      return link_to(
               text,
               url_for(action=action, **kwargs),
               class_=('active' if c.action == action else '')
           )
>>  url_for(action=action, **kwargs),
Module routes.util:280 in url_for
<<          raise GenerationException(
                   ""url_for could not generate URL. Called with args: %s %s"" % \
                   (args, kargs))
           return url
>>  (args, kargs))
GenerationException: url_for could not generate URL. Called with args: () {'action': 'comments', 'controller': 'package', 'id': 'dengue-net'}
CGI Variables
AUTH_TYPE 	'cookie'
DOCUMENT_ROOT 	'/etc/apache2/htdocs'
GATEWAY_INTERFACE 	'CGI/1.1'
HTTP_ACCEPT 	'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
HTTP_ACCEPT_CHARSET 	'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING 	'gzip, deflate'
HTTP_ACCEPT_LANGUAGE 	'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3'
HTTP_CONNECTION 	'keep-alive'
HTTP_COOKIE 	'__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode""; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode""; ckan_user=""flavio""; ckan_display_name=""Fl\xc3\xa1vio Code\xc3\xa7o Coelho""; ckan_apikey=""c139718d-918f-4a2b-b219-e33cb05cbe23""; __utmc=4669863'
HTTP_HOST 	'ckan.emap.fgv.br'
HTTP_KEEP_ALIVE 	'115'
HTTP_REFERER 	'http://ckan.emap.fgv.br/'
HTTP_USER_AGENT 	'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1'
PATH_INFO 	'/package/dengue-net'
PATH_TRANSLATED 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/package/dengue-net'
REMOTE_ADDR 	'10.250.48.110'
REMOTE_PORT 	'47624'
REMOTE_USER 	u'flavio'
REMOTE_USER_DATA 	'userid_type:unicode'
REMOTE_USER_TOKENS 	['']
REQUEST_METHOD 	'GET'
REQUEST_URI 	'/package/dengue-net'
SCRIPT_FILENAME 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py'
SERVER_ADDR 	'10.252.2.60'
SERVER_ADMIN 	'[no address given]'
SERVER_NAME 	'ckan.emap.fgv.br'
SERVER_PORT 	'80'
SERVER_PROTOCOL 	'HTTP/1.1'
SERVER_SIGNATURE 	'<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n'
SERVER_SOFTWARE 	'Apache/2.2.16 (Debian)'
WSGI Variables
application 	<beaker.middleware.CacheMiddleware object at 0x7f051c802690>
beaker.cache 	<beaker.cache.CacheManager object at 0x7f051c802750>
beaker.get_session 	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f051c8026d0>>
beaker.session 	{'locale': u'pt_BR', '_accessed_time': 1307358102.6266389, '_creation_time': 1302605470.0245121}
mod_wsgi.application_group 	'dck093|'
mod_wsgi.callable_object 	'application'
mod_wsgi.handler_script 	''
mod_wsgi.input_chunked 	'0'
mod_wsgi.listener_host 	''
mod_wsgi.listener_port 	'80'
mod_wsgi.process_group 	'ckan'
mod_wsgi.request_handler 	'wsgi-script'
mod_wsgi.script_reloading 	'1'
mod_wsgi.version 	(3, 3)
paste.cookies 	(<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio'>, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode""; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"";... __utmc=4669863')
paste.registry 	<paste.registry.Registry object at 0x7f051d7e5710>
paste.throw_errors 	True
pylons.action_method 	<bound method PackageController.read of <ckan.controllers.package.PackageController object at 0x7f051d7e5b10>>
pylons.controller 	<ckan.controllers.package.PackageController object at 0x7f051d7e5b10>
pylons.environ_config 	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons 	<pylons.util.PylonsContext object at 0x7f051d7e5a50>
pylons.routes_dict 	{'action': u'read', 'controller': u'package', 'id': u'dengue-net'}
repoze.who.identity 	<repoze.who identity (hidden, dict-like) at 139659946193344>
repoze.who.logger 	<logging.Logger instance at 0x7f051c801128>
repoze.who.plugins 	{'openid': <OpenIdIdentificationPlugin 139659931643152>, 'friendlyform': <FriendlyFormPlugin 139659929725584>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f051c9e7610>, 'auth_tkt': <AuthTktCookiePlugin 139659931643280>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f051c9dcfd0>}
routes.route 	<routes.route.Route object at 0x7f051c69fd50>
routes.url 	<routes.util.URLGenerator object at 0x7f051d7e57d0>
webob._parsed_cookies 	({'ckan': 'd4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06', '__utmz': '4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'auth_tkt': 'c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode', 'ckan_display_name': 'Fl\xc3\xa1vio Code\xc3\xa7o Coelho', 'ckan_apikey': 'c139718d-918f-4a2b-b219-e33cb05cbe23', 'ckan_user': 'flavio', '__utma': '4669863.1260802706.1295364409.1306935693.1306960483.86', '__utmc': '4669863', 'SESS61faf7f7d2406929b2f9eb4cbfff17d2': 'a3306f5d1194cf0d8a51dbf7b42c1259'}, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode""< wbr>; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!... __utmc=4669863')
webob._parsed_query_vars 	(GET([]), '')
webob.adhoc_attrs 	{'language': 'en-us'}
wsgi process 	'Multi process AND threads (?)'
wsgi.file_wrapper 	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f051cc1ad50>
wsgi.version 	(1, 1)
wsgiorg.routing_args 	(<routes.util.URLGenerator object at 0x7f051d7e57d0>, {'action': u'read', 'controller': u'package', 'id': u'dengue-net'})
}}}"	1307358586000000	1307374897000000
1178	defect		fccoelho		closed	fixed	Local storage issue with ckanext-storage	"I have followed the instruction for installation a configuration of ckanext-storage, but I am getting this error when I try to access the /storage/upload url:

URL: http://ckan.emap.fgv.br/storage/upload
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
          
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
              
               response = self.app(environ, start_response)
              
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:124 in __call__
<<                  return ['%s' % paste.registry.restorer.get_request_id(environ)]
              
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
              
>>  controller = self.resolve(environ, start_response)
Module pylons.wsgiapp:263 in resolve
<<          if self.log_debug:
                   log.debug(""Resolved URL to controller: %r"", controller)
               return self.find_controller(controller)
          
           def find_controller(self, controller):
>>  return self.find_controller(controller)
Module ckan.config.environment:39 in find_controller
<<          # Check to see if its a dotted name
               if '.' in controller or ':' in controller:
                   mycontroller = pkg_resources.EntryPoint.parse('x=%s' % controller).load(False)
                   self.controller_classes[controller] = mycontroller
                   return mycontroller
>>  mycontroller = pkg_resources.EntryPoint.parse('x=%s' % controller).load(False)
Module pkg_resources:1954 in load
<<      def load(self, require=True, env=None, installer=None):
               if require: self.require(env, installer)
               entry = __import__(self.module_name, globals(),globals(), ['__name__'])
               for attr in self.attrs:
                   try:
>>  entry = __import__(self.module_name, globals(),globals(), ['__name__'])
Module ?:87 in <module>
<<  class StorageAPIController(BaseController):
           ofs = get_ofs()
          
>>  class StorageAPIController(BaseController):
Module ?:88 in StorageAPIController
<<  class StorageAPIController(BaseController):
           ofs = get_ofs()
          
           @jsonpify
>>  ofs = get_ofs()
Module ?:69 in get_ofs
<<              continue
               kw[k[4:]] = v
           ofs = get_impl(storage_backend)(**kw)
           return ofs
>>  ofs = get_impl(storage_backend)(**kw)
Module ofs.local.pairtreestore:26 in __init__
<<          self.hashing_type = hashing_type
               self.shorty_length = shorty_length
               self._open_store()
          
           def _open_store(self):
>>  self._open_store()
Module ofs.local.pairtreestore:30 in _open_store
<<      def _open_store(self):
               if self.hashing_type:
                   self._store = PairtreeStorageClient(self.uri_base, self.storage_dir, shorty_length=self.shorty_length, hashing_type=self.hashing_type)
               else:
                   self._store = PairtreeStorageClient(self.uri_base, self.storage_dir, shorty_length=shorty_length)
>>  self._store = PairtreeStorageClient(self.uri_base, self.storage_dir, shorty_length=self.shorty_length, hashing_type=self.hashing_type)
Module pairtree.pairtree_client:91 in __init__
<<          self._decode = re.compile(r""\^(..)"", re.U)
              
               self._init_store()
      
           def __char2hex(self, m):
>>  self._init_store()
Module pairtree.pairtree_client:264 in _init_store
<<                      self.uri_base = prefix
                   else:
                       raise NotAPairtreeStoreException
      
               if not os.path.isdir(self.store_dir):
>>  raise NotAPairtreeStoreException
NotAPairtreeStoreException:
CGI Variables
AUTH_TYPE 	'cookie'
DOCUMENT_ROOT 	'/etc/apache2/htdocs'
GATEWAY_INTERFACE 	'CGI/1.1'
HTTP_ACCEPT 	'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
HTTP_ACCEPT_CHARSET 	'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING 	'gzip, deflate'
HTTP_ACCEPT_LANGUAGE 	'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3'
HTTP_CONNECTION 	'keep-alive'
HTTP_COOKIE 	'__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode""; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode""; ckan_user=""flavio""; ckan_display_name=""Fl\xc3\xa1vio Code\xc3\xa7o Coelho""; ckan_apikey=""c139718d-918f-4a2b-b219-e33cb05cbe23""; __utmc=4669863; hide_welcome_message=1'
HTTP_HOST 	'ckan.emap.fgv.br'
HTTP_KEEP_ALIVE 	'115'
HTTP_USER_AGENT 	'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1'
PATH_INFO 	'/storage/upload'
PATH_TRANSLATED 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/storage/upload'
REMOTE_ADDR 	'10.250.48.110'
REMOTE_PORT 	'57426'
REMOTE_USER 	u'flavio'
REMOTE_USER_DATA 	'userid_type:unicode'
REMOTE_USER_TOKENS 	['']
REQUEST_METHOD 	'GET'
REQUEST_URI 	'/storage/upload'
SCRIPT_FILENAME 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py'
SERVER_ADDR 	'10.252.2.60'
SERVER_ADMIN 	'[no address given]'
SERVER_NAME 	'ckan.emap.fgv.br'
SERVER_PORT 	'80'
SERVER_PROTOCOL 	'HTTP/1.1'
SERVER_SIGNATURE 	'<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n'
SERVER_SOFTWARE 	'Apache/2.2.16 (Debian)'
WSGI Variables
application 	<beaker.middleware.CacheMiddleware object at 0x7fc2999a6cd0>
beaker.cache 	<beaker.cache.CacheManager object at 0x7fc2999a6d90>
beaker.get_session 	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7fc2999a6d10>>
beaker.session 	{'locale': u'pt_BR', '_accessed_time': 1307373458.4790759, '_creation_time': 1302605470.0245121}
mod_wsgi.application_group 	'dck093|'
mod_wsgi.callable_object 	'application'
mod_wsgi.handler_script 	''
mod_wsgi.input_chunked 	'0'
mod_wsgi.listener_host 	''
mod_wsgi.listener_port 	'80'
mod_wsgi.process_group 	'ckan'
mod_wsgi.request_handler 	'wsgi-script'
mod_wsgi.script_reloading 	'1'
mod_wsgi.version 	(3, 3)
paste.cookies 	(<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio' hide_welcome_message='1'>, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid _type:unicode""; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221fla...lcome_message=1')
paste.registry 	<paste.registry.Registry object at 0x7fc299de4f50>
paste.throw_errors 	True
pylons.environ_config 	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons 	<pylons.util.PylonsContext object at 0x7fc299e27d50>
pylons.routes_dict 	{'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'}
repoze.who.identity 	<repoze.who identity (hidden, dict-like) at 140473773131872>
repoze.who.logger 	<logging.Logger instance at 0x7fc2999a3b48>
repoze.who.plugins 	{'openid': <OpenIdIdentificationPlugin 140473779397008>, 'friendlyform': <FriendlyFormPlugin 140473777417040>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7fc299b93a50>, 'auth_tkt': <AuthTktCookiePlugin 140473779397136>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7fc299b934d0>}
routes.route 	<routes.route.Route object at 0x7fc299864090>
routes.url 	<routes.util.URLGenerator object at 0x7fc299e27ed0>
webob._parsed_query_vars 	(GET([]), '')
webob.adhoc_attrs 	{'language': 'en-us'}
wsgi process 	'Multi process AND threads (?)'
wsgi.file_wrapper 	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fc299dae4e0>
wsgi.version 	(1, 1)
wsgiorg.routing_args 	(<routes.util.URLGenerator object at 0x7fc299e27ed0>, {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'})
Reply
	
Reply to all
	
Forward
		
	
Reply to all
paste@localhost to fccoelho
	
show details 10:17 (2 hours ago)
	
- Show quoted text -
CGI Variables
AUTH_TYPE 	'cookie'
DOCUMENT_ROOT 	'/etc/apache2/htdocs'
GATEWAY_INTERFACE 	'CGI/1.1'
HTTP_ACCEPT 	'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
HTTP_ACCEPT_CHARSET 	'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING 	'gzip, deflate'
HTTP_ACCEPT_LANGUAGE 	'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3'
HTTP_CONNECTION 	'keep-alive'
HTTP_COOKIE 	'__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode""; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode""; ckan_user=""flavio""; ckan_display_name=""Fl\xc3\xa1vio Code\xc3\xa7o Coelho""; ckan_apikey=""c139718d-918f-4a2b-b219-e33cb05cbe23""; __utmc=4669863; hide_welcome_message=1'
HTTP_HOST 	'ckan.emap.fgv.br'
HTTP_KEEP_ALIVE 	'115'
HTTP_USER_AGENT 	'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1'
PATH_INFO 	'/storage/upload'
PATH_TRANSLATED 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/storage/upload'
REMOTE_ADDR 	'10.250.48.110'
REMOTE_PORT 	'57429'
REMOTE_USER 	u'flavio'
REMOTE_USER_DATA 	'userid_type:unicode'
REMOTE_USER_TOKENS 	['']
REQUEST_METHOD 	'GET'
REQUEST_URI 	'/storage/upload'
SCRIPT_FILENAME 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py'
SERVER_ADDR 	'10.252.2.60'
SERVER_ADMIN 	'[no address given]'
SERVER_NAME 	'ckan.emap.fgv.br'
SERVER_PORT 	'80'
SERVER_PROTOCOL 	'HTTP/1.1'
SERVER_SIGNATURE 	'<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n'
SERVER_SOFTWARE 	'Apache/2.2.16 (Debian)'
WSGI Variables
application 	<beaker.middleware.CacheMiddleware object at 0x7fc2999a6cd0>
beaker.cache 	<beaker.cache.CacheManager object at 0x7fc2999a6d90>
beaker.get_session 	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7fc2999a6d10>>
beaker.session 	{'locale': u'pt_BR', '_accessed_time': 1307373466.085686, '_creation_time': 1302605470.0245121}
mod_wsgi.application_group 	'dck093|'
mod_wsgi.callable_object 	'application'
mod_wsgi.handler_script 	''
mod_wsgi.input_chunked 	'0'
mod_wsgi.listener_host 	''
mod_wsgi.listener_port 	'80'
mod_wsgi.process_group 	'ckan'
mod_wsgi.request_handler 	'wsgi-script'
mod_wsgi.script_reloading 	'1'
mod_wsgi.version 	(3, 3)
paste.cookies 	(<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio' hide_welcome_message='1'>, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid _type:unicode""; auth_tkt=""c6c3c3e0b3fe7bb765356659992760b24da43221fla...lcome_message=1')
paste.registry 	<paste.registry.Registry object at 0x7fc29a9da850>
paste.throw_errors 	True
pylons.environ_config 	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons 	<pylons.util.PylonsContext object at 0x7fc29a9ef950>
pylons.routes_dict 	{'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'}
repoze.who.identity 	<repoze.who identity (hidden, dict-like) at 140473790084352>
repoze.who.logger 	<logging.Logger instance at 0x7fc2999a3b48>
repoze.who.plugins 	{'openid': <OpenIdIdentificationPlugin 140473779397008>, 'friendlyform': <FriendlyFormPlugin 140473777417040>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7fc299b93a50>, 'auth_tkt': <AuthTktCookiePlugin 140473779397136>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7fc299b934d0>}
routes.route 	<routes.route.Route object at 0x7fc299864090>
routes.url 	<routes.util.URLGenerator object at 0x7fc29a9efcd0>
webob._parsed_query_vars 	(GET([]), '')
webob.adhoc_attrs 	{'language': 'en-us'}
wsgi process 	'Multi process AND threads (?)'
wsgi.file_wrapper 	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fc29a9d7d50>
wsgi.version 	(1, 1)
wsgiorg.routing_args 	(<routes.util.URLGenerator object at 0x7fc29a9efcd0>, {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'})"	1307373810000000	1308045351000000
1179	enhancement		timmcnamara	ckan-backlog	new		Support tag aliases	"A small number of tags are near-duplicates of each other.

Perhaps we could support word stemming from NLTK and/or manual tag aliases:

    statistics
    statistik
    ...
    survey
    surveying
    surveys"	1307429221000000	1339774332000000
1182	defect		timmcnamara	ckan-backlog	new		"Comments from deleted packages appear in ""Recent Comments"" feed"	"When a package has been deleted, say for spam moderation, comments still appear in the recent comments section.

This is a problem because non-admin users will be shown a warning that they're not authorised to view the package if they click on the link.

At CKAN.net currently, this affects the most recent comment."	1307658251000000	1339774319000000
1184	enhancement		timmcnamara	ckan-backlog	new		Support Wuala as CKAN storage option	"Most of CKANs storage options are tied to the USA. This brings concerns of data security for some organisations who may wish to adopt the system. [http://wua.la Wuala] is a distributed file system that stores data in a peer-to-peer manner. The company behind it, LaCie sells storage for a fee. However, they also enable clients to have 'free' storage space when machines act as a storage node. 

In order to be a storage node, a machine needs to be online for more than 14% of the time - roughly 4h per day. Most CKAN servers are likely to have a far greater uptime than this.

Supporting Wuala would go some way to enabling CKAN to be used in a secure manner. That is, CKAN could be promoted for organisational use where there is lots of data to be stored and large geographic distances to be managed. There is a [http://www.wuala.com/Wuala%20API/Sample%20Clients/Python/wualaAPI/ Python client available] and a fairly [http://www.youtube.com/watch?v=3xKZ4KGkQY8 long Google Tech Talk] that overviews the system."	1308034751000000	1339774306000000
1185	defect		timmcnamara	ckan-backlog	new		Administrators can't delete packages from web UI	"Administrators have ""View"", ""Edit"" and ""History"" tabs. However, I can't see a way to delete a package from the web UI.

Version: CKAN.net as of today"	1308111818000000	1339774289000000
1186	enhancement		dread		closed	fixed	Password reset facility	You can register a user with password and (optional) email address. But if you forget the password you can't then log in again. We need a password reset facility that sends and email with a new password.	1308142766000000	1310556519000000
1188	enhancement		nickstenning	ckan-backlog	new		Allow diffing against initial (blank) package version	"Currently the history page only allows diffing between different versions of a package, but there doesn't appear to be any easy way to see the changes introduced by the first version of a package.

I'm requesting the ability to diff against a ""blank slate"" initial state of a project, so I can see the content of the first project commit.

Not sure if this is a vdm feature, so I'm putting this ticket in against ckan."	1308153160000000	1339774275000000
1189	enhancement		timmcnamara		closed	invalid	Spideroak support in CKAN Storage	"[http://spideroak.com Spideroak] is a bulk storage platform by a company that releases quite a lot of [https://spideroak.com/code free software], has an extensive [https://spideroak.com/diy/ developer API] with pricing of $10/100GB/month.

The data store is optimised for bulk data storage and retrieval. This is the kind of use case that CKAN packages require. We don't need low latency. We are after low cost, high reliability solution."	1308187104000000	1323173227000000
1194	defect		dread		closed	fixed	"""Welcome back"" message for newly registered user"	"1. Register a new user (/user/register)
2. It redirects you to user page and displays ""Welcome back"" even though you are a new user."	1308563392000000	1317053688000000
1197	enhancement		timmcnamara	ckan-backlog	closed	wontfix	Add JavaScript guide for CKAN	"A new library, [http://blog.optimizely.com/introducing-guidersjs-an-open-source-guider-e guiders.js], has been open sourced that seems to be great at unobtrusively introducing new users to features of websites.

The library drives Optimizely's website. The !GitHub repo is here: https://github.com/jeff-optimizely/Guiders-JS

Some examples:

[[Image(http://posterous.com/getfile/files.posterous.com/temp-2011-06-22/BnlgHDvhjpFfqbwvprsIaklqrpdgJyebfuhxajbDhmvnFgzdzzCJdvluJFki/Screen_shot_2011-06-22_at_6.42.21_PM.png.scaled500.png)]]

[[Image(http://s3.amazonaws.com/files.posterous.com/temp-2011-06-22/CpvlcinkrxhtnAHfGtEHhxyGkraqlFmCjcnztFuugagBGxmBjCdwuquHpvEB/promo_gmail.png?AWSAccessKeyId=AKIAJFZAE65UYRT34AOQ&Expires=1308801019&Signature=DJWDgj7fmyq7K3ve177H7DPwKdw%3D)]]"	1308801032000000	1340020357000000
1198	enhancement		dread	ckan-backlog	new		Publisher hierarchy	"'Publisher' entities in the model. They are hierarchical. 

'User-Publisher' connections with one or more roles (e.g. drafter, moderator). 

Authorization settings can control who can set what values in a 'published by' type field.

Publishers and User-Publishers available to read in the API.

Future tickets will provide:
 * API to write Publishers and User-Publishers
 * UI to edit Publishers and User-Publishers

(This feature deprecates authorization groups)"	1308820592000000	1339774200000000
1200	enhancement		pudo		closed	duplicate	Port PDEU theme to CKAN.net	"PDEU has a newer layout which could be adapted to also be used on ckan.net. For this, things should be cleaned up, copied to the ckanext-ckan.net repo as needed and re-colored the core CKAN color scheme. 

Maybe we could have this coincide with a CKAN rename? "	1308824017000000	1311180218000000
1201	enhancement		kindly	ckan-backlog	new		seperate out logic in atom feeds to logic layer.	Simplify the logic in the atom feed an make all feeds use logic layer to return lists.	1308928892000000	1310124297000000
1204	enhancement		timmcnamara		closed	invalid	renaming packages seems to result in a server error	"When renaming a package, and doing other changes at the same time an error is reported. However, the changes seem to go through fine.

ENV:

 - datacatalogs.org as of 29 June 2011"	1309303816000000	1311179980000000
1209	defect		kindly		closed	fixed	sort out rendering of formalchemy package preview	"Got broken due to the moderated edits changes #1141.

Need to make sure preview works."	1309817240000000	1310571715000000
1215	defect		dread		closed	fixed	Edit Resources table - removing all rows prevents adding them again	"I'm not sure if this was reported earlier, but I have found a minor bug in the ""Add new package"" form. As far as I can see this bug exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). This is a problem with adding new resources in ""Resources"" section. To reproduce the bug do the following:

 1. Go to ""Add new package"", for instance http://ckan.net/package/new
 2. Scroll down to ""Resources"" section.
 3. Click on ""Remove this row"" link (red circle).
 4. Say ""Yes"" to ""Are you sure you wish to remove this row?""
 5. Now there should be no rows. Only the header ""URL*   Format  Description""
 6. Click on ""Add row to table""
 7. All you get now is another row with table header ""URL*        Format Description"". You will not be able to add any resources now. Solution is to reload the page.

The simplest solution to this bug will be to hide the ""Remove this
row"" link if there is only one row."	1310133224000000	1310385485000000
1219	defect		timmcnamara		closed	fixed	CSS issues on IE7	"As reported on [http://lists.okfn.org/pipermail/ckan-dev/2011-July/001043.html ckan-dev]:

> items in the footer of CKAN (""Packages"", ""Groups & Tags"", ""About"", ""Language"", etc.) are shown vertically instead of horizontally in IE7. This works fine in later browsers like IE8, IE9, FF4, and latest Opera and Chrome.

> This seems to exist in all recent CKAN versions up to 1.4.1a."	1310423688000000	1310740534000000
1226	defect		dread		closed	worksforme	Genshi stream exception	"IndexError exception in genshi.filters.i18n keeps popping up daily.

e.g.
{{{
...
Module ckan.controllers.authorization_group:28 in index
<<              items_per_page=20
               )
               return render('authorization_group/index.html')
       
           def read(self, id):
>>  return render('authorization_group/index.html')
Module ckan.lib.base:74 in render
<<      
           return cached_template(template_name, render_template, cache_key=cache_key, 
                                  cache_type=cache_type, cache_expire=cache_expire)
                                  #, ns_options=('method'), method=method)
>>  cache_type=cache_type, cache_expire=cache_expire)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:63 in render_template
<<              stream = item.filter(stream)
               
               return literal(stream.render(method=method, encoding=None))
           
           if 'Pragma' in response.headers:
>>  return literal(stream.render(method=method, encoding=None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
       
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out=out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:339 in __call__
<<          for filter_ in self.filters:
                   stream = filter_(stream)
               for kind, data, pos in stream:
                   cached = cache_get((kind, data))
                   if cached is not None:
>>  for kind, data, pos in stream:
Module genshi.output:670 in __call__
<<          _gen_prefix = _gen_prefix().next
       
               for kind, data, pos in stream:
                   output = cache_get((kind, data))
                   if output is not None:
>>  for kind, data, pos in stream:
Module genshi.output:771 in __call__
<<          push_text = textbuf.append
               pop_text = textbuf.pop
               for kind, data, pos in chain(stream, [(None, None, None)]):
       
                   if kind is TEXT:
>>  for kind, data, pos in chain(stream, [(None, None, None)]):
Module genshi.output:586 in __call__
<<      def __call__(self, stream):
               prev = (None, None, None)
               for ev in stream:
                   if prev[0] is START:
                       if ev[0] is END:
>>  for ev in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """"""
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module ckanext.googleanalytics.plugin:62 in download_adder
<<              (downloaded %s times)</span>'''
                   count = None
                   for mark, (kind, data, pos) in stream:
                       if mark and kind == START:
                           href = data[1].get('href')
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1175 in __call__
<<          """"""
               callable_value = hasattr(self.value, '__call__')
               for mark, (kind, data, pos) in stream:
                   if mark is ENTER:
                       if callable_value:
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """"""
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
       
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.markup:378 in _match
<<                      for event in self._match(self._flatten(template, ctxt,
                                                                  **vars),
                                                    ctxt, start=idx + 1, **vars):
                               yield event
>>  ctxt, start=idx + 1, **vars):
Module genshi.template.markup:327 in _match
<<                      break
       
               for event in stream:
       
                   # We (currently) only care about start and end events for matching
>>  for event in stream:
Module genshi.template.base:545 in _flatten
<<          while 1:
                   for kind, data, pos in stream:
       
                       if kind is START and data[1]:
>>  for kind, data, pos in stream:
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range
}}}"	1311008861000000	1314029434000000
1227	enhancement		timmcnamara	ckan-backlog	new		Display packages' tags in search results	"In when displaying search results, it would be useful to also display the tags of a package. Sometimes it's difficult to infer the scope of what the package does from the title and the first sentence of the description. Tags are quite concise way to display rich information. 

ENV=datacatalos.org, with CKAN 1.4.2a"	1311034262000000	1339774147000000
1230	enhancement		kindly	ckan-sprint-2011-10-28	closed	fixed	Make sure plugins tables work in tests.	Dgu tests failed as harvesting table are not created at the correct point.	1311100811000000	1311154142000000
1232	enhancement		thejimmyg	ckan-backlog	new		[super] Interface improvements	"Child tickets:

 * #1194 ""Welcome back"" message for newly registered user
 * #1202 Links to datapkg utility don't lead to info about it
 * #925 Change the search box icon to remove the down arrow
 * #923 Search box doesn't work in leaderboard page in stats extension
 * #1034 Flash message cached
 * #737 Markdown syntax summary page
 * #811 Extra field editing form layout breaks when there are long field names
"	1311178296000000	1315948536000000
1233	enhancement		thejimmyg	ckan-backlog	new		[super] Improve wiki-style functionality for history	"At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas:

 * #191 Searching by modification date
 * #193 Searching by time-related field
 * #301 Package discussion pages
 * #1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages
 * #1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page
 * Other improvements as per my word doc."	1311179392000000	1315948668000000
1235	enhancement		thejimmyg	ckan-future	new		[super] Search Improvements	"Child tickets:

 * #234 UI Review - Autocomplete package names & tags in search
 * #193	Searching by time-related field
 * #191	Searching by modification date
 * #905 Unable to search with accented characters in package names
 * #906 Ability to search without accents for accented words
 * #924 Search box has no search button

Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two."	1311182641000000	1311182641000000
1242	enhancement		timmcnamara		closed	fixed	thedatahub.org's login page	"The right hand side of [http://thedatahub.org/user/login thedatahub.org login screen] is a little confusing: 

{{{
    Not a member?

    Join CKAN to contribute packages under your own name.

     - Register
     - Reset your password
     - Privacy Policy
}}}

Suggested improvements to make:

 - replace CKAN with thedatahub.org
 - in general, edit the copy to be more inviting
 - remove ""Reset your password"" from that bubble
 - move the privacy policy link to somewhere less prominent "	1311569218000000	1323173292000000
1243	enhancement		timmcnamara		closed	invalid	"[thedatahub.org] ""Upload Data"" should provide a nicer"	"If someone who is not logged in clicks on ""Upload Data"", they are presented with

{{{
    Not authorized to upload files.
}}}

This is in emboldened red text. That is, the system is telling a user that they're doing something illegitimate, even though they're actually following an invitation from the main menu.

I propose two options for resolving this:

 1. only display the ""Upload Data"" tab if someone is authorised to upload data
 2. if someone isn't logged int, provide a nice welcome message thanking them for wanting to upload data, but we only allow registered users to do so

ENV: thedatahub.org, CKAN 1.4.2b"	1311577466000000	1328000405000000
1244	enhancement		dread		assigned		Notes field carriage-returns converted to CRLF	"When you edit a package in the web form, if the notes field had {{{\n}}} as the End Of Line symbol, it gets lost when you preview or save the package, and the notes field is displayed all on one line.

This can be seen when editing annakarenina (as created by 'paster create-test-data'). The diff shows for example:
{{{
- Some test notes
+ Some test notes
?                +
}}}
but it would more clearly be shown as:
{{{
- Some test notes\n
+ Some test notes
?                ++
}}}

This is a significant problem with DGU, since a lot comes in via the API.

It's not clear what we should do about it. We could standardise on \n or \r\n when the form submission comes in. Do different browsers on different platforms do different things with EOLs?

== Analysis ==

Displaying the package: the Markdown processor respects both EOLs when displaying the field, putting each line in a <p> tag.

Creating the package edit form: placed into <textfield>. 

Browser displaying package edit form: <textfield> displays \n and \r\n as EOL. But \n\n gets compressed to one EOL. But on submission, both are returned as \r\n.

Receiving the edited package: Somewhere along the line the EOL gets converted to \n\n.
"	1311689456000000	1340191253000000
1245	task		rgrp	ckan-v1.5	closed	fixed	Reorganise ckan.org pages	"Here is Rufus' plan for ckan.org reorganisation from http://ckan.okfnpad.org/documentation (22/7/11) (ticket made by dread):

{{{
/solutions
  /overview <--- current 'about' page (the software)
  /customers <-- say ""in addition to our clients...""
  /features <-- needs most work (pictures etc)
/pricing/  <-- simplest thing possible ... talk with Ira (cover pricing and support)
/developers <--- links to http://docs.ckan.org/, getting the software, the wiki, the mailing list /contact
/blog
}}}

Tasks: 
 * add tagline in header - ""the data portal software""
 * make and add glossy PDF
 * change ""data hub"" to ""data portal"" passim (if only using one  term)
 * add info@ckan.org 
 * image banner on home page - redo screenshots without menus
"	1311861708000000	1317422640000000
1257	enhancement		dread	ckan-backlog	new		Anti-Spam tools	"We are getting more and more spam on ckan.net and we need to improve our strategy of combating it. It is bad because google ranks who we link to (which we do want for legitimate links), and our front page contains the latest package edits, so spam is immediately very visible.

Spam:
 * Package creation
 * Package edit
 * User creation
 * Group creation

Systems to consider:
 * Automatic
   * captcha
   * bayesian scoring
 * Sysadmin
   * a tool to quickly analyse and remove packages, edits and users
 * Helpful users
   * button to click indicating spam, leading to any/all of: 
     * add TODO item (which emails package admin and sysadmins)
     * quarantine the package/user so it doesn't show up on the front page and a warning is shown when you view it

General thoughts:
 * We should learn from the wikipedia tools, as they are the experts
 * This is a serious problem that is only going to get worse and is a threat to the whole site."	1312283565000000	1339774129000000
1260	enhancement		pudo	ckan-backlog	new		Remove duplicate functions from _util.html	There seems to be both a list view for dictized and non dictized data structures for all entities in _util.html at the moment. Probably in the back of someone's mind already, but cleanup here would be nice. 	1312366652000000	1313401499000000
1261	defect		pudo	ckan-backlog	new		Investigate dots in extras search	It seems that searching for extras_foo:value works with solr, but extras_foo.bar:value doesn't. No theory on why. 	1312366768000000	1312366768000000
1262	enhancement		pudo	ckan-backlog	new		"Enforce ""create-user"" permission"	This does not seem to have any implications at the moment, it should lock down registration and remove all related links. 	1312375296000000	1323090112000000
1272	enhancement		amercader	ckan-sprint-2011-10-10	closed	fixed	Store spatial extents provided in the package form	"Consolidate the extra `spatial` as the field for providing information about the geographic extent of the package. This will integrate automatically with the spatial search and give developers a single point to develop geospatial features.
"	1313411787000000	1317381996000000
1273	requirement		amercader	ckan-backlog	new		Create docs for API v3		1313412083000000	1313412083000000
1278	enhancement		amercader	ckan-backlog	new		Refactor authorized_query calls	"There are some functions that still use the Auhtorizer().authorized_query method:

{{{
./ckan/controllers/authorization_group.py:24:        query = ckan.authz.Authorizer().authorized_query(c.user, model.AuthorizationGroup)
./ckan/lib/base.py:237:        groups = ckan.authz.Authorizer.authorized_query(c.user, model.Group, 
./ckan/lib/search/sql.py:55:        q = authz.Authorizer().authorized_query(username, model.Group)
./ckan/lib/search/sql.py:118:        q = authz.Authorizer().authorized_query(self.options.get('username'), model.Package)
./ckan/logic/action/get.py:154:    query = Authorizer().authorized_query(user, model.Group, model.Action.EDIT)

./ckan/tests/test_authz.py:158:        q = self.authorizer.authorized_query(self.notadmin.name, model.Package)
./ckan/tests/test_authz.py:353:        q = self.authorizer.authorized_query(self.notmember.name, model.Package)
./ckan/tests/test_authz.py:357:        q = self.authorizer.authorized_query(self.member.name, model.Package)
./ckan/tests/functional/test_authorization_group.py:44:        group_count = Authorizer.authorized_query(u'russianfan', model.AuthorizationGroup).count()
}}}"	1313415177000000	1313415177000000
1280	defect		kindly	ckan-backlog	closed	fixed	fix sqlalchemy so that it works with postgres 9.0	After the ckan migrate process, sqlalcehmy reflect does not work and causes an error when reflecting indexes.  A bug report needs to be put into sqlalchemy.	1313513090000000	1328786670000000
1285	enhancement		dread	ckan-future	assigned		Errors cause emails	"Currently a sysadmin gets an email when an exception is not caught. But there are occasions when we DO want to catch an exception so we can fail nicely for the user, but the sysadmin STILL gets an email to know to fix something.
e.g. if there is an exception when search indexes a package. You want to catch the exception and still run any other notify calls.
"	1314116944000000	1338206151000000
1286	enhancement		rgrp	ckan-backlog	new		Remove remaining formalchemy stuff	"Stuff I've spotted:

 * forms/*
 * template/group/edit_form.html
 * template/package/edit_form.html

This can go once new DGU form is in."	1314116996000000	1342436420000000
1288	defect		dread	ckan-backlog	new		Package edit/creation can't include 'relationships' field	"When you create or edit a package (via the API), you aren't able to specify the relationships it has. (If you do you get 409 {{{ {""__junk"": [""The input field __junk was not expected.""]} }}})

The normal way to create relationships is via /api/rest/relationships/ and this works. But when you GET a package, the dictionary lists all relationship details. So this bug creates a problem for editing a package that has relationships - you want to GET it, make any edits and then PUT it back. The work-around is to delete the 'relationships' key from the dict before you PUT it back.

== Options ==

Ideally, CKAN would read the 'relationships' key and make the necessary changes. This is a chunk of work.

Another good option is to allow an unchanged 'relationships' value, but barf it is edited. This is also a chunk of work.

A bad option would be to just ignore the 'relationships' value, since users will get frustrated changing this value and wonder why it never saves, not understanding it is different to all the rest, without error message.

A final option is to get rid of relationships altogether."	1314203695000000	1339774098000000
1289	CREP		dread	ckan-backlog	closed	wontfix	Remove 'relationships'	"== Abstract ==

Package Relationships have not taken off in the 18 months we've had them in the API. There are some issues with them and we need to spend more time improving them or consider getting rid of them.

== The Problem ==

Original use cases are expressed here: #253 Here are comments about how we could handle these specific examples better:
1. groups of packages - maybe better with a custom tag?
2. fragment resources - soon to be covered by 'kind' resource field #957
3&5. derived resource - better to have some sort of resource relationship perhaps?
4. linked resource - again better to have some sort of resource relationship perhaps?

Outstanding issues needing serious effort to fix:
 * #256 Editing them in Web UI (not done yet)
 * #1288 Package edit/creation can't include 'relationships' field

== Specification ==

Remove relationships from model, API, tests, Web UI. Data migration to remove from db.

== Why do it this way == 

Getting frustrated having problems with the code, when it's not used much. Often asked about what it's for, but rarely used. Seems an overly complicated design.

== Backwards Compatibility ==

n/a

== Implementation plan ==

=== Deliverables ===

See Specification

=== Risks and mitigations ===

Risk: a customer suddenly wants this, and the new ways to relate resources are not in place yet.

Mitigation: discuss this decision thoroughly to make sure we are confident the use cases are not important. Discuss with team, ckan-discuss and specifically the LOD people who have some related packages on thedatahub.org.

=== Participants ===

David Read

=== Progress ===

Not yet."	1314206502000000	1317315211000000
1314	enhancement		dread	ckan-backlog	assigned		ckanclient search - generator improvements	"Apparently the search generator always makes two requests, even if you don't want to see the search results, which might be slow. Can this be optimised?

Maybe we should also provide a second search function that doesn't use the generator - the original simple search function (that leaves the user to deal with limit & offset)."	1315395410000000	1340191233000000
1317	defect		dread	ckan-backlog	assigned		password reset - improve user search	"In password reset, it gets confused if you have two similar users. This is because with the string the user provides, it searches several fields, not just name but also fullname and email address, allowing you to search for these. But only name is unique.

Specific problem:
Ira searches for ""Irina"" then it finds both:
{{{<User name=irina fullname=Irina email=irina.bolychevsky@okfn.org ]}}}
and
{{{<User name=shevski fullname=Ira email=>}}}
(I think)

Maybe need to choose which field it searches?"	1315415539000000	1340191221000000
1322	enhancement		dread		assigned		Action API improvements	"Focusing on improving Action API as the v3 API:

 * have an optional parameter of the data_dict called ""options"".  Options would contain items that would get passed into the context. e.g. {{{{""options"" : {""ref_package_by"": ""id""}}. }}}
 * instead of using API version to change the way packages are referenced, use the ref_package_by.
 * All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'.
 * Action API is v3 of api, replacement for v1 & v2. Default for most urls is still v1, but if url is /api/action then default to v3.

Next steps:
 * Add search API (package, resource, 
 * Add Util API
 * Clarify JSONP still applies
 * Add doc strings, clarifying parameters"	1315474749000000	1340191088000000
1326	enhancement		thejimmyg	ckan-backlog	new		Write a set of auth plugin functions to integrate with Druapl	Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor.	1315821084000000	1315821084000000
1327	enhancement		rgrp	ckan-v1.6	closed	duplicate	[super] Dataset Archiving	"Split out of #852. Automated archiving of datasets (related to QA).

''' Automated archiving using worker process '''

 * #890 - Timed actions in ckanext-queue
 * #891 - Resource download worker daemon
 * #892 - Make stored data available in WUI"	1315821490000000	1320662446000000
1328	defect		minspamboks@…		assigned		Unicode & paster commands	"A possible bug in CKAN when I tried deleting users using ""paster --plugin=ckan user delete"" command.

To reproduce the bug do the following:

1. Create a user with an ID (which in my case was a user's full name)
that contains non-unicode caracters like Norwegian ""æ"", ""ø"", or ""å"".

2. Make sure that you can see something like the example below:
(pyenv) rm@mycomputer:$ paster --plugin=ckan user
Users:
name=Rustæm

3. Then try deleting the user with following command:
(pyenv) rm@mycomputer:$ paster --plugin=ckan user delete ""Rustæm""

You should now get a python encoding error. I know that this is quite
rare case, but in our case it caused some trouble. Could you guys have
a look at this bug?

CKAN ver. 1.3.3."	1315823110000000	1340191065000000
1330	enhancement		rgrp	ckan-backlog	closed	invalid	Deprecate / Remove test_authz.py	"test_authz.py appears to test in great detail some very specific additional authz (related to total site lock-down it seems -- introduced I think for hri project).

I think there are simpler ways to get total site lockdown (use external auth!) and this test is slow and delicate (e.g. depends on specific words in templates). Suggest removing. If we don't remove we should at least refactor tests for access to certain pages to use a proper method of testing (e.g. agreed html comments in each page) rather than being depending on the presence of absence of specific wording."	1315899129000000	1327060201000000
1332	defect		dread		closed	fixed	i18n IndexError exceptions	"We get this i18n error for the URL http://no.ckan.net/authorizationgroup (which runs 1.4.2 with Norwegian translation). I'm not sure why it occurs. Fixing it is low importance though, because this site doesn't use Authorization Group feature

Usually caused by bots. The user does not appear to be logged in.

{{{
WebApp Error: <type 'exceptions.IndexError'>: list index out of range
...
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range
}}}"	1315906429000000	1315999104000000
1352	enhancement		amercader	ckan-backlog	new		Use logic functions instead of as_dict when indexing entities	"The current search implementation uses the output of the the as_dict method of the domain Package object to update the index

https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48

It also uses package_to_api1 in the SynchronousSearch plugin:

https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93

This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model).

The search should use the logic function to get the package properties:

{{{
get_action('package_show')(context,data_dict)
}}}

"	1316615397000000	1339774086000000
1353	defect		nickstenning	ckan-v1.5	closed	fixed	No UI to remove resources	I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org.	1316729765000000	1317075904000000
1355	defect		amercader	ckan-backlog	new		Package extras property does not include the newly created ones	"The extras in the package object sent to the extensions after editing (https://bitbucket.org/okfn/ckan/src/01efd5649c10/ckan/logic/action/update.py#cl-226) do not include the newly added.


"	1317034126000000	1339774056000000
1366	defect		dread	ckan-future	assigned		Search inside extra fields	"SOLR search doesn't support searching for part of an extra field, but it does for other fields.

i.e. title=""One Two Three"" matches q=one AND q=title:one
and geographic_coverage=""England Scotland"" matches q=England BUT NOT q=geographic_coverage:England

This problem emerged when we went to SOLR in #1275 (CKAN 1.5a). Tests were skipped.

This is could be a problem for DGU and maybe elsewhere.
"	1317290992000000	1338206707000000
1373	defect		shevski	ckan-sprint-2011-11-07	closed	fixed	home page view does not react to logging in / out	"Either: thedatahub.org does not display 'add a dataset' top navigation link or the 'create dataset' main central link to logged in user - instead showing 'sign up' link in the central box. It does however display the 'my account' and 'logout' link in the top right correctly.

Navigating to another part of the site (e.g. search or about or my account) does bring the 'add a dataset' navigation link back and the functionality works.

Works fine on test.ckan.net so only on thedatahub.

Not obviously a caching problem, since I did try clearing cache.

OR: it displays as if you're logged in even when you log out. The links doesn't work and prompt you to log in though."	1317829479000000	1320174240000000
1388	defect		dread	ckan-backlog	closed	fixed	etags caching on home page	"Needs to update on:
 1. language - It's broken on thedatahub.org if you change to German and then click on the CKAN icon top-left to go to thedatahub.org again.
AND
 2. whether signed in or not - #1373
BUT not latest revisions (which is what it was)

Or get rid of etag caching on this page and others?


"	1318429443000000	1320174229000000
1395	defect		seanh		closed	fixed	"""Import Error: cannot import name UnicodeMultiDict"" when installing ckan from source"	"At the ''paster db init'' command when installing ckan from source I get the error ''""Import Error: cannot import name UnicodeMultiDict""'' (happens with both ckan 1.4.2 and today's latest bleeding edge code, on Ubuntu 10.04.3).

UnicodeMultiDict has been removed in a recent version of python-webob, and the ''pip install ... lucid_missing.txt'' causes a too-new version of python-webob to be installed into ckan's virtualenv (the new webob gets installed as a dependency of formalchemy).

I manually did ''pip uninstall webob'' and then ran ''paster db init'' again and it worked."	1318520183000000	1320857823000000
1406	enhancement		zephod	ckan-backlog	new		Re-enable RSS subscriptions	"RSS 'subscribe' buttons appeared in many places on the site but were not very helpful. They took (confused) users pointed to the raw feed code, and Google Reader could not understand the feed. Safari, however, could interpret it correctly.

Their presentation needs to be clear and consistent; the RSS feed really needs testing in a variety of readers; and we need to decide exactly which items should get a feed. (Package updates? Groups?)"	1318861327000000	1320930088000000
1414	enhancement		shevski	ckan-backlog	new		track user log-ins on thedatahub.org	"Set up tracking for user logins so that we have stats about how many active users of thedatahub exist
want to be able to see who logged in the the last x months"	1319454782000000	1319454782000000
1419	enhancement		dread	ckan-sprint-2011-11-07	closed	invalid	Can't log in via OpenID	"I couldn't log into theDataHub with OpenID today. I tried both Google ID and MyOpenID. Both times the login on the remote auth server went fine, but when it returns you to theDataHub you get error ""Login failed. Bad username or password."""	1319543013000000	1319796164000000
1423	enhancement		markbrough	ckan-backlog	new		Edit resources suggestions	" * Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: http://thedatahub.org/dataset/edit/iati-registry
 * Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in."	1319641906000000	1338203678000000
1424	enhancement		dread	ckan-backlog	new		Openness notice should be clearer	"ckan-discuss discussion suggests changes to the 'openness' indicator
( http://lists.okfn.org/pipermail/ckan-discuss/2011-October/001786.html )

Dataset view page:
 * If there is an explicit but non-OKD compliant license, such as CC-BY-NC, then this should be stated explicitly, perhaps: “This dataset is Not Open. License: Creative Commons Attribution Noncommerical. This is not an open license as it does not meet the Open Knowledge Definition.”
 * If the license is marked as “Other::License Not Specified”, then this should be stated explicitly, perhaps:
 “This dataset is Not Open. It is published without an explicit license, the publisher reserves all rights to the dataset.”
 * 3. If the license field was left empty by the contributor of the Data Hub record, then again this should be stated explicitly, perhaps: “This dataset is Not Open. The license of this dataset is unknown or unspecified. Start an enquiry on IsItOpenData »
 * There is a bug so that non-open licenses doesn't have an openness notice.
 * If downloadable resources are not available, this should not affect 'openness' - check this has been removed."	1319648089000000	1319648089000000
1430	defect		amercader	ckan-sprint-2011-11-07	closed	fixed	Documents get mixed between SOLR cores	"On some occasions (apparently random), the documents indexed in a specific SOLR core get mixed with different site_ids.

E.g: We look for all documents in the testing.iatiregistry.org core, faceted by site_id. We would expect all documents to have site_id = iati_testing, but some of them have site_id = iatiregistry.org

http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=*:*&fq=+state:active&facet=true&facet.field=site_id

{{{
<lst name=""facet_fields"">
<lst name=""site_id"">
<int name=""iati_testing"">265</int>
<int name=""iatiregistry.org"">255</int>
</lst>
</lst>
}}}

If we compare one of the records which disappeared from the ""iati_testing"" site_id in both the production and testing SOLR cores
of the server, the records are exactly the same, including the indexed_ts property:

http://okfn-solr.fry-it.com:8080/solr/iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id

http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id

Note that the response from the URLs shown may vary, as the testing site could have been reindexed."	1320068076000000	1324033923000000
1432	enhancement		rgrp	ckan-backlog	new		[super] Data processing system for CKAN and Webstore	"Super ticket: #1190

A data processing system which utilizes the  Webstore. One could
get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)"	1320142747000000	1339774041000000
1435	enhancement		rgrp	ckan-v1.6	closed	wontfix	Switch to continuous.io for buildbot (?)		1320145831000000	1323283538000000
1438	enhancement		dread	ckan-future	new		Action API - parameter discovery/checking	"Many actions in the Action API require parameters. What params are needed should be listed and checked. Because currently, if you get them wrong you simply get a useless 500 error.

Currently they are listed in the docs, extracted from the code manually.

So you could GET /action/api/package_list to receive not only the help text, but a list of arguments.

And if you send an extra or missing argument then an intelligent error message can be returned.

= implementation =

How about some sort of decorator on the action function:
{{{
@logic_params(id, offset, limit)
def get_package_list(context, data_dict):
    ...
}}}
This would do the param checking, and is there a way to extract these params from the function? Or do a registration of the logic function?

I'd certainly like to keep the list of the list of params for the function with the function, for ease of reading the code.

Another good thing would be to pass in the params named as themselves, rather than having them contained in the data_dict."	1320161890000000	1338205050000000
1439	enhancement		dread	ckan-backlog	new		Action API discoverablility	"A good service API needs to be discoverable, so you are not always having to refer to the documentation html.

Maybe /api/action should return a list of actions available? (Currently this returns a 404.) 
 * It would be nice to sort these into get/create/update/delete.
 * #1438 Parameters for each of the actions must be discoverable too

/api/action/{action_name} should also return the help text / parameters allowable. (Currently this returns 400 error)"	1320161970000000	1325474974000000
1457	defect		jilly.mathews	ckan-future	new		Bug with DataNL instance	" ""when logging into http://register.data.overheid.nl/ with OpenID, the /user/me page gives a 404. CKAN version 1.3.4.""

n the manual it says an API key kan be created via http://test.ckan.net/user/me /. However, when I try the corresponding  http://register.data.overheid.nl/user/me, I get a 404 error (not found).	"	1320925041000000	1320925041000000
1460	enhancement		rgrp		assigned		Improve extensions documentation	"Current extensions documentation needs some work: http://docs.ckan.org/en/latest/plugins.html

 * Queue extension section may now be out of date (?)
 * Think about how it integrates with https://github.com/okfn/ckanext-example (especially tutorial and example extension)
 * Document all plugin points (auto-extract from CKAN source??)"	1321114523000000	1340191001000000
1461	defect		pudo		closed	fixed	CkanClient doesn't submit auth headers for GET requests	e.g. package_register_get.	1321354037000000	1321359503000000
1463	defect		thejimmyg	ckan-sprint-2011-11-21	closed	wontfix	QA extension no-longer works with packaged CKAN 1.5	"The extension needs upgrading:

    No module named webstore.database

The other option is to delay this functionality and merge with the publisher dashboards after the group refactor.

John, David Raznick, David Read, Ira, what do you think?"	1321375614000000	1321874259000000
1464	enhancement		thejimmyg	ckan-v1.6	closed	wontfix	Replace RabbitMQ with Celeryd to support running multiple instances	"The current harvesting implementation can only have one instance per server.

We could either:

 * Accept this
 * Change the RabbitMQ code to support multiple instances
 * Switch the entire harvesting to use Celery
 * Wait for James's redis based live feedback based system"	1321375709000000	1328529042000000
1465	enhancement		thejimmyg	ckan-v1.7	closed	fixed	Upgrade the harvester to support publishers properly	"At the moment most of the harvesting functionality is only available to sysadmins. This is fine for most cases but going forward we will need to grant each publisher access to their own sources, harvest objects and harvest object errors. This will probably be done based on the groups refactor.

One more pressing concern is giving DGU publishers the ability to make calls to /harvest/object/id. We want them to be able to see documents that faild validation and also to see *all* errors in each document."	1321375923000000	1338203858000000
1466	enhancement		thejimmyg	ckan-backlog	new		Need to support https login for multiple instances as part of the CKAN package install		1321375978000000	1328529062000000
1474	enhancement		kindly	ckan-sprint-2011-11-21	closed	fixed	fix up navl tests	navl tests are being skipped unskip them!	1321825892000000	1321826753000000
1487	enhancement		kindly	ckan-sprint-2011-12-05	closed	fixed	Fix group ordering on homepage	ordering on homepage by name instead of group count	1322094280000000	1324474147000000
1489	enhancement		dread	ckan-backlog	assigned		Updating example theme/extension	"ckanext-example needs updating for CKAN 1.5:
 * theme changes
 * new forms

About: 'ckanext-exampletheme' was created in Spring 2011 as an example CKAN extension that showed how to customise the look & operation of CKAN. This moved to github and renamed 'ckanext-example'. "	1322137920000000	1324292384000000
1509	defect		dread		assigned		Mis-dated old revisions of datasets	"e.g. http://thedatahub.org/dataset/osm%402011-07-12T12%3A16%3A47.590358 gives:
{{{
This is the current revision of this dataset, as edited at 2011-07-12 12:16.
}}}
but it should say the date 2011-07-12 (which is the data being displayed).

The problem is looking at PackageRevision, when it might be a tag or extra."	1323090398000000	1340190814000000
1514	defect		dread	ckan-v1.6	closed	duplicate	Modifying user name loses connection with revisions	"If you edit your user name, the number of revisions you made returns to 0. This is because in the Revision object, the user's name is stored, rather than the user's ID.

rgrp:  We can reconnect that pretty easily (and have a
longer term solution that involves not using the usernames but the
userids in in the Revision objects so we don't hvae this problem in
future!)"	1323100659000000	1323279018000000
1520	task		dread	ckan-sprint-2011-12-19	closed	fixed	Disable name changing	Because of #1514 we should just disable name changing, until #1514 is done.	1323169663000000	1323280999000000
1532	defect		dread	ckan-sprint-2011-12-19	closed	fixed	Registration with OpenID has misleading error message	"The log-in page says ""Login using Open ID"" and gives instructions for signing up. YET this is only available to users who've already added openid to their account. If you have not done this and then sign-in via OpenID (which is successful from the OpenID end) then you are told ""Login failed. Bad username or password."" in a flash message.

Proposed solution (i don't know if this is possible):
 * When you log-in for the first time via OpenID, it doesn't actually log you in in CKAN. It just sends you to the 'Create User' page with the OpenID field pre-filled, and puts up a flash message ""This OpenID account is not yet registered on thedatahub. Please complete your details."". This allows you to complete the registration and logs you in, and allows you to log-in directly with OpenID in the future.

Alternatives:
 * Just change the error message to be 'You need to register in CKAN first. Quote your OpenID in the registration form to use it in future.'
 * Remove OpenID altogether"	1323276392000000	1323956027000000
1534	enhancement		rgrp	ckan-backlog	new		Change revisions to record userid rather than username	"The use of username is problematic because username's can change.

 * Change all revision creation code to use user id (simplest is to change c.author field in lib/base.py (?))
  * (?) Add a field ipaddr for ip address of anonymous users? (or just keep putting this in author field on Revision and then acception that those won't match when we do a look up against user table)
 * Change user view page to look up against user id rather than name
 * Perform migration on existing Revision objects
   * Match should probably be against both openid and username when searching Revisions' author field (especially true on CKAN where some people have already changed their username from being their openid)"	1323278790000000	1338205050000000
1535	enhancement		dread	ckan-backlog	new		Plump for auth header of: X-CKAN-API-KEY	"When using the API, the apikey needs to be supplied in a header called 'Authorization'. Because some proxys / deployments use this header for other things, a configurable header was provided as an alternative, with default ""X-CKAN-API-KEY"".

Rufus suggests having *one* way for this.
a) making this not configurable any more
b) making X-CKAN-API-KEY the default

(keep Authorization allowed, but not documented, for backwards compatibility)"	1323279082000000	1339774019000000
1542	enhancement		dread	ckan-backlog	new		Buttons to purge spam datasets and groups	"A sysadmin should be able to easily examine a suspect group or package, determine if it was created by a spammer (as opposed to being a legitimate object that has been graffitied by a spammer) and purge it.

The existing two-stage revision delete is currently unreliable and perhaps too laborious.

Olav and Richard have needs along this line."	1323364930000000	1339774000000000
1544	task		dread	ckan-backlog	new		delete old git branches	"We have 150 odd branches ({{{git branch -a}}}) - most of them old - we should prune them. At very least, branches that have been merged in should be deleted. Look at old branches that haven't been merged in and wonder why.

May be of some use:
{{{
git branch --merged
git branch --no-merged
}}}"	1323702610000000	1323702610000000
1545	enhancement		amercader	ckan-sprint-2012-01-09	closed	wontfix	Remove external asset dependencies	"CKAN is pulling a number of resources from external locations. This causes problems when connectivity is limited and you have to work locally. Maybe some of them cold be moved to CKAN source to avoid external requests.

Quick search:
{{{

./ckan/templates/layout_base.html:            <img src=""http://assets.okfn.org/images/logo/okf_logo_white_and_green_tiny.png"" id=""footer-okf-logo"" />
./ckan/templates/layout_base.html:            <a href=""http://opendefinition.org/""><img alt=""This Content and Data is Open"" src=""http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png"" style=""border: none ; margin-bottom: -4px;""/></a>
./ckan/templates/package/resource_read.html:                <img src=""http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png"" alt=""[Open Data]"" />
./ckan/templates/package/read.html:          <img src=""http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png"" alt=""[Open Data]"" /></a>
./ckan/templates/_util.html:                    <img src=""http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png"" alt=""[Open Data]"" />
./ckan/templates/_util.html:                  <img src=""http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png"" alt=""[Open Data]"" />
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      this.$dialog.html('<h2>Loading results...</h2><img src=""http://assets.okfn.org/images/icons/ajaxload-circle.gif"" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:          self.setMessage('Uploading file ... <img src=""http://assets.okfn.org/images/icons/ajaxload-circle.gif"" class=""spinner"" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      self.setMessage('Checking upload permissions ... <img src=""http://assets.okfn.org/images/icons/ajaxload-circle.gif"" class=""spinner"" />');
Binary file ./ckan/lib/app_globals.pyc matches
./ckan/lib/app_globals.py:                                  'http://assets.okfn.org/p/ckan/img/ckan.ico')
./ckan/config/deployment.ini_tmpl:ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico
}}}"	1323702635000000	1325260051000000
1549	enhancement		ross	ckan-backlog	closed	wontfix	[super] Short link tool	"It would be great to have a CKAN extension that allowed users (or CKAN itself) to generate short links to other URIs (both internal and external). Once created, shortlinks made by CKAN should be changeable. This would allow uploaded content to be moved without the user's link changing at all. The tool itself might also be of use as a general link-shortener to users other than the CKAN system itself.

Another useful feature would be for this to also collect some simple analytics such as the referrer and client IP for future reference. I'm not yet sure what we would do with the analytics other than some sort of popularity metric.

Questions:
  - Core, or Extension, or Self-hosted?
"	1324036998000000	1325474219000000
1550	enhancement		ross	ckan-backlog	assigned		Allow simple auth via the API	It should be possible to pass userid/username and api key and obtain a response from CKAN for external services that use CKAN auth. Those services shouldn't be talking to the DB directly.	1324049369000000	1346670055000000
1578	enhancement		rgrp	ckan-backlog	new		[super] Re-enable and refactor ratings		1324322443000000	1325473015000000
1585	enhancement		dread		closed	fixed	Security fix	(details embargoed until 31/1/2012)	1324473465000000	1340633128000000
1596	enhancement		dread	ckan-future	new		Refactor authz roles	"Suggestions from rgrp:

 * Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
 * Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
 * Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

(This ticket is split off from #1065)"	1324549888000000	1338205019000000
1597	enhancement		dread	ckan-sprint-2012-01-23	closed	fixed	Tag search - filter by group	"I want to browse tags, but filtered for a particular group. Currently our tag API doesn't allow for filtering by group. 

This is important for improving groups as communities within a site #1521. It would be easy to do this by adding an option to filter by a group. BUT are there any other use cases that would warrant a more complete faceted tag search?

--

BTW I can currently draw a tag cloud for a group - I can get the top tags used in a group like this:

curl http://thedatahub.org/api/action/package_search -d
'{""q"":""groups:country-ca"", ""facet.field"":""tags"", ""rows"":""0""}'

but it only contains the top 20 tags."	1324550492000000	1326821156000000
1598	enhancement		rgrp	ckan-backlog	new		Reinstate Ratings	"Ratings were disabled approximately a year ago because:

 * Unclear purpose and UX. What did ratings tell you? How useful were they?
 * Spamming (esp by bots: you could submit an anonymous rating via a GET request which caused problems)

Both problems are solvable and it would be nice to have this feature reinstated.

 * Purpose: can make this more purposable by limiting to logged in users (or at least distinguishing logged in from non-logged in users)
  * Even better we could allow ratings to be made public (I'm interested in what someone else I respect finds important)
 * Spamming: limit to logged in users and / or use AJAX over an API to submit ..."	1325177524000000	1325474818000000
1604	enhancement		dread	ckan-backlog	new		Get ckanext-moderatededits working with CKAN 1.5+ templates	"ckanext-moderatededits requires an old and possibly development version of CKAN. It would be good to update it for later CKAN versions.

According to the README, you need CKAN from branch feature-1141-moderated-edits-ajax but the changelog suggests this branch went into version 1.4.2. So it possibly works with 1.4.2 and 1.4.3(.1). But CKAN 1.5 has revamped templates, so the genshi stream filters definitely don't work.

BTW history_ajax/read_ajax calls have been deprecated in CKAN since 1.5.2a and will need fixing up to use the Action API too as part of this."	1325352429000000	1325352429000000
1606	enhancement		dread	ckan-backlog	new		metadata license config option	"Add a config option to choose the metadata licence. Set default to Open Database License.

Currently the dataset edit form says ""Important: By submitting content, you agree to release your contributions under the Open Database License."" This is hard-coded, but not suitable for when DGU uses the CKAN form - they use the OGL."	1325501130000000	1339773981000000
1615	enhancement		thejimmyg	ckan-v1.6	closed	worksforme	CKAN Should work behind a proxy server	"This would allow deployment via Nginx or Apache using proxy to Paster, uWSGI. At the moment CKAN isn't aware of the proxy's IP address so when you perform an action which does a redirect (such as adding a package), CKAN redirects you to the *internal IP* not the external *proxy IP*.

We would like this work to facilitate testing within VMs as part of our new build infrastructure.

It would also be nice if CKAN worked when mounted at a path other than /. That could be dealt with in another ticket because it isn't a problem at the moment."	1325687841000000	1328888870000000
1628	defect		dread	ckan-sprint-2012-02-06	closed	wontfix	get ckanext-dgu working with ckan 1.5.1	johnglover said: I can confirm that even with the mapping fix, the ckanext-dgu dgu_form plugin template does not work properly with 1.5.1, so should probably not be installed at present. The edit page is ugly but should work (eg: http://dgu-os.okfn.org/dataset/edit/abandoned-vehicles), but the 'add a dataset' page is broken (eg: http://dgu-os.okfn.org/dataset/new)	1326214762000000	1328526967000000
1642	defect		pudo	ckan-backlog	new		Extra link generators generate garbled HTML	"I had a package descriptions with URLs that contain ""group:foo"". This produces garbled output as the system tries to generate two sets of links: the outer link and an inner link. 

Need to fix the parser. 

Text: 


Webdienst basierende Bereitstellung von Geobasisdaten der Freien und Hansestadt Hamburg. Folgende Geobasisdaten werden als WebMapTileService (WMT-S) für die Dauer des Wettbewerbs netzbasiert unter der Creative Commons Lizenz zur Verfügung gestellt: Digitale Orthophotos 40 cm Auflösung (Layer: apps4d_DOP40), Digitale Stadtkarte (Layer: apps4d_DISK), Digitale Regionalkarte (Layer: apps4d_DIRK), Digitale Karte 1:5000 (Layer: apps4d_DK5).

Metadateneinträge zu den Daten im PortalU:

* [apps4d_DOP40](http://www.portalu.de/trefferanzeige?docuuid=D6082285-6CAC-49C5-B7ED-B5A72BC17DEC&plugid=/kug-group:kug-iplug-udk-db_hh)
* [apps4d_DISK](http://www.portalu.de/trefferanzeige?docuuid=D2DA3359-3DC9-4CA4-AE9B-7C63D53B220F&plugid=/kug-group:kug-iplug-udk-db_hh)
* [apps4d_DIRK](http://www.portalu.de/trefferanzeige?docuuid=B7DDD599-F677-4C80-BED5-F11D82D5D1DD&plugid=/kug-group:kug-iplug-udk-db_hh)
* [apps4d_DK5](http://www.portalu.de/trefferanzeige?docuuid=2AE6D23E-48A5-4D85-BC0A-160737E0C8D2&plugid=/kug-group:kug-iplug-udk-db_hh)


One fix is quoting the URLs"	1326382171000000	1339773967000000
1643	enhancement		shevski	ckan-backlog	new		Add fixed tags to thedatahub for better browsing	Similar to publicdata.eu, want to have themed areas such as finance, environment, census, etc and country tags	1326393293000000	1326393293000000
1644	enhancement		shevski	ckan-backlog	new		Order default dataset page by most downloaded resources on thedatahub	"Instead of alphabetically as we do currently, alternatively by most viewed datasets

for http://thedatahub.org/dataset"	1326393542000000	1326393542000000
1647	enhancement		shevski	ckan-backlog	new		add links to ckan discuss & dev to thedatahub	In the footer as well as more clearly & directly on the About page	1326673852000000	1326707383000000
1648	enhancement		shevski	ckan-backlog	closed	fixed	Clarify that additional info = extra fields + add guidance	"Super ticket: #1506

Need to decide which term to use and then have the same for editing as well as viewing a dataset.

In creating/editing a dataset, want more explanation about adding extra fields (probably as a tooltip or similar).. i.e. that this let's you add extra custom metadata such as 'location: uk' which is then searchable etc"	1326674843000000	1330632344000000
1656	enhancement		ross		closed	duplicate	Configuration for reverse proxying	"Provide configuration for reverse proxying that will correctly handle the mapping of a URL to a sub-folder (using X-SCRIPT-NAME)

* Analysis of the best solution [1d]

* Implement reverse proxying in tandem with #1653  [2d]

* Document and store the configuration files. [1d]
"	1326711575000000	1326711659000000
1661	defect		dread	ckan-backlog	assigned		Wrong Routes version installed by CKAN package	"Jaakko Louhio reported that the wrong Routes version got installed during CKAN's package install.

He is using Ubuntu 10.04 and he believes it install Routes 1.12.3 instead of what we use which is 1.11."	1326718061000000	1339773949000000
1662	defect		dread		closed	wontfix	OpenID not compatible with mounting CKAN at non-root URL	"Mounting CKAN at a non-root URL was made to work properly here: #1659

Unfortunately OpenID doesn't play nicely and would require some work to get working."	1326730366000000	1326730414000000
1668	defect		dread	ckan-backlog	new		repoze version discrepency	"There's a discrepency in repoze.who versions between the source and package installs:

 * repoze.who - package 1.0.18 vs source 1.0.19
 * repoze.who-friendlyform - package 1.0b3 vs source 1.0.8

We get a test failure [1] with the 1.0b3 version (from the ubuntu 10.04 python-repoze.who-plugins package). But we've not noticed any problems on s057 instances (br, no, ie etc) which have the package versions of repoze.who.

The reason the package install uses the earlier packaged versions rather than the ones we'd like is that repoze uses all sorts of horrendous import hacks, making it too difficult to put into our 'ckan-conflict' source package.

James suggests we 'do something horrible like dynamically patch repoze on CKAN import'.

[1] http://buildbot.okfn.org/builders/builder-ckan/builds/1371/steps/shell/logs/stdio
ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode"	1326801746000000	1326801746000000
1677	enhancement		amercader	ckan-v1.6	closed	duplicate	Make synchronous search the default behaviour	"Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default.
We could keep a config flag to deactivate it, but synchronous search should be the default behaviour."	1326807604000000	1326807655000000
1679	enhancement		dread	ckan-backlog	new		Default roles problem	"The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py).

The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different. 

We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?"	1326823042000000	1339773923000000
1684	enhancement		ross	ckan-backlog	assigned		Remove all config from ckanext-archiver	"ckanext-archiver currently has a settings file (and a default) and it should be passed in all relevant information from the context.

Remove all settings (ARCHIVE_DIR and MAX_CONTENT_LENGTH and others) and pass them in from CKAN."	1326983821000000	1346670037000000
1692	enhancement		rgrp	ckan-v1.7	closed	wontfix	Add image attribute to Dataset and Group	"Add image attribute for Dataset and Group which would be shown in prominent place.

For dataset this could be a relevant image and for Group this would be more of a logo.

I think this has very high value from a UX point of view.

== Issues ==

 * Do we need to support uploading and storing the image (or do users store elsewhere)
  * Could use something like transloadit service
  * Use our own storage facility with a reserved directory
 * Do we need to do resizing etc?
  * Maybe. Perhaps can prefer this in reasonably small size and then just size down in css for thumbnails etc.
"	1327108761000000	1338204395000000
1697	enhancement		rgrp	ckan-backlog	new		A Configurable list of states for a Dataset	"Currently have 'active' and 'deleted' suggest also:

 * 'draft'
 * 'hidden'

(Do we need both). Also write out workflows related to these."	1327400630000000	1338204189000000
1717	enhancement		shevski	ckan-backlog	new		[super] Search UX improvements	"1. Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search)

2. Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page).

3. Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only.  Or y'know, a better way of doing it.

4. More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu

5. Blue search button should be displayed in line with the search bar, not underneath

6. Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name"	1327603981000000	1330088539000000
1740	enhancement		seanh	ckan-future	new		Get rid of `from module import ...`	"It's really bad to do {{{from module import *}}} and CKAN has a lot of them. I suggest a three-pronged approach:

1. Don't add any more of them.

2. When you're programming if you see an easy opportunity to remove one then do so.

3. At some point we should task someone to go through the code and remove them all (which is what this ticket is for), but this will be a big job and may break things.

We should also get rid of most or all of the {{{from module import foo}}} and {{{from module import foo, bar}}} statements.

I think the right thing to do is just {{{import module}}} and then use {{{module.foo}}} in your code. But if you find yourself doing {{{module.foo.bar}}} then you may have a code smell.

See: http://docs.python.org/dev/howto/doanddont.html"	1328094369000000	1328094884000000
1745	enhancement		rgrp	ckan-v1.9	new		Dataset search UX improvements as of Jan 2012	"Changes to make search both more exploratory and more satisfying to use

 * Search query build - #1603
  * Ability to add new facet fields ""live""
  * That is add fields which then contain faceted options (a bit like data.hri.fi)
 * (??) Autocomplete  / drop down on search (i.e. search while you type)
  * Dubious about value / cost ratio here

Probably would involve to pure JS and HTML implementation.

== Implementation ==

Probably require

 *  API changes to expose solr style API directly #1737
"	1328224941000000	1340033358000000
1746	enhancement		seanh	ckan-backlog	closed	wontfix	Activity streams pagination	Currently user, package and group activity streams only return the most recent 15 activities, even though all activities are kept in the db. Do we want to add pagination - to both the API and the HTML pages - to support retrieving older activities?	1328446488000000	1355141062000000
1747	enhancement		seanh	ckan-backlog	new		Expire old activities	Currently the activity streams database tables just get longer and longer over time. Do we want to eventually delete the oldest activities, to keep the length of the table within limits?	1328446589000000	1339773859000000
1749	enhancement		seanh	ckan-future	assigned		Allow creating activity details through API	Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts.	1328465406000000	1338213933000000
1750	enhancement		seanh	ckan-backlog	new		Move ckan/lib/activity.py into the model	Move ckan/lib/activity.py moved to into the model - say ckan/model/activity_extension.py, because it's so tightly knit with the model code, whereas most of the lib code is used in the controllers.	1328465888000000	1339773840000000
1787	enhancement		dread	ckan-future	new		[super] Improve RESTful API	" * Lists of entities should be full URLs, rather than just the names
 * Discoverability - /api/v3/rest should list the entity types that can be listed

This could be v3 of the RESTful interface."	1328702082000000	1328702082000000
1789	enhancement		seanh	ckan-backlog	new		Implement a tag_update() logic action function	"So users can rename a tag and/or move it between vocabularies using the API.

Currently we have create_tag() and delete_tag(), but if you were to 'update' a tag by deleting it and then recreating it all the datasets that had that tag will have lost it and you'll have to re-add it to them all.

What should happen to datasets that have the tag, if the tag gets moved between vocabularies? All the datasets just keep the tag with the new vocabulary? This will become a problem if/when we support 'radio button'-style vocabularies (where each dataset can only have one tag from the vocabulary)."	1328805413000000	1339773812000000
1790	enhancement		dread	ckan-future	new		Click to delete tags, rather than have all existing tags in the tag text box	"From Pablo:

Editing the tags field is clumsy when there are too many tags. Could
show existing effectively as tags (like delicious), then allow clicks
to delete. New tags added via text box."	1328888674000000	1328888674000000
1798	defect		dread		closed	fixed	API search in non-q fields has exception for unicode characters	"You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q.

For example:
{{{
http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93
}}}
This ""N-dash"" (Unicode character 2013) causes this exception:
{{{
Module ckan.controllers.api:460 in search
<<                      if ver in u'12':
                               # Otherwise, put all unrecognised ones into the q parameter
                               params = convert_legacy_parameters_to_solr(params)
                           query = query_for(model.Package)
                           results = query.run(params)
>>  params = convert_legacy_parameters_to_solr(params)
Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr
<<      for search_key in non_solr_params:
               value_obj = legacy_params[search_key]
               value = str(value_obj).replace('+', ' ')
               if search_key == 'all_fields':
                   if value:
>>  value = str(value_obj).replace('+', ' ')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128)
}}}

This problem affects CKAN 1.5 and 1.5.1 only."	1329395420000000	1329395560000000
1800	refactor		seanh	ckan-future	new		Tidy up *_list() and *_search() functions in ckan/logic/action/get.py	"For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed.

Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality."	1329405129000000	1338204886000000
1801	enhancement		dread	ckan-backlog	closed	fixed	No links to password reset	You can reset your password (#1186) but you have to know the URI (/user/reset) - there is no link to this page!	1329405290000000	1343144718000000
1805	enhancement		toby		closed	fixed	Error pages do not translate	"eg 404 page

create a 404 error via a bad url - translation links go to /document/error not the actual bad url"	1329489216000000	1329760795000000
1823	enhancement		rgrp	ckan-backlog	new		Spring clean bin directory	Huge number of accumulated (and likely unnecessary) scripts in /bin directory.	1329773331000000	1338203554000000
1824	enhancement		seanh	ckan-future	new		Add vocabulary pages	"For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page.

We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar."	1329845089000000	1338204958000000
1827	enhancement		dread	ckan-backlog	new		'Register' link should be hidden if you not allowed to register	"{{{
I have just deny visitors the create-user permission:
sudo -u ckanstd paster --plugin=ckan roles deny reader create-user -c /etc/ckan/std/std.ini
sudo -u ckanstd paster --plugin=ckan roles deny anon_editor create-user -c /etc/ckan/std/std.ini

and after restarting, the register link is *not* hidden, but now when you access the register page, it shows you this message ""Unauthorized to create a user"" (when not logged in). But anyway that is an improvement.
}}}"	1329924939000000	1339773730000000
1831	enhancement		ross	ckan-backlog	assigned		Login with email address	"Users should be able to log in using either their username, or their email address, both of which are unique.

Will require a change to UsernamePasswordAuthenticator in ckan.lib.authenticator.py and possible a useful User.by_email in the user model if it doesn't already exist."	1330073906000000	1346670504000000
1832	enhancement		dread	ckan-backlog	assigned		dataset purge API	"Purging datasets (deleting them fully, not just changing the state to 'deleted') is important for users testing dataset creation over the API on a test CKAN instance. 

Without this, they need to resort to more difficult methods such as:
 * cleaning and reloading the database
 * setting the test datasets to state 'deleted' and also appending a suffix '_00' and incrementing the number until there is no clash of names.

Requested for NHSIC.

== Implementation ==

 * This could slot into the Action API. 
 * Of course we would need to ensure the user's had been given the specific right to purge.
 * I suggest we log the full details of the dataset being purged."	1330077745000000	1339773706000000
2197	defect		zydio	ckan-backlog	assigned		Storage Metadata API: add/update not working with local file storage (Pairtree)	"If OFS is configured with Pairtree to use a local file storage, all POST requests to add/update metadata ( /api/storage/metadata/{label} ) will fail.

This is due to the use of [https://github.com/okfn/ofs/blob/master/ofs/remote/botostore.py BotoOFS] specific ''private'' methods in [https://github.com/okfn/ckan/blob/master/ckan/controllers/storage.py StorageAPIController.set_metadata()], eg: self.ofs.'''_require_bucket'''(bucket), self.ofs.'''_get_key'''(b, label), self.ofs.'''_update_key_metadata'''(k, metadata) ... those methods can't be found in [https://github.com/okfn/ofs/blob/master/ofs/local/pairtreestore.py POTFS] and this causes errors.
The API should use only [https://github.com/okfn/ofs/blob/master/ofs/base.py OFSInterface] methods, or should conditionally make calls based off the actual type of ''self.ofs''.

''PS: I did set ""ckan"" as ""Component"" in the ticket because storage has been integrated back into the core in CKAN 1.6''"	1330421377000000	1346662128000000
2198	enhancement		zydio		closed	fixed	API documentation is missing Storage Metadata API info	Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on [https://github.com/okfn/ckanext-storage/blob/master/README.rst Storate Metadata API].	1330421743000000	1339771453000000
2201	enhancement		rgrp	ckan-v1.7	closed	duplicate	Add citation info to Dataset and Resource page	"Suggest:

%title. %author. Retrieved %date. %site_title.

For resource: %title = %dataset_title. %resource_name."	1330765080000000	1330765400000000
2202	enhancement		rgrp	ckan-future	reopened		Display page view count on dataset and resource pages	Just like we display download counts we should display view counts.	1330765455000000	1338204929000000
2230	refactor		seanh	ckan-v1.7	closed	fixed	Tidy up of search facets code duplication	"Because of a clash between two development branches there is some duplication of code to do with code facets (note: at the time of writing the code duplication exists only on the feature-1821-multilingual-extension branch, but this will be merged into master at some point):

The package_search() function is adding the search facets to the search results twice with two different data structures, with keys ""facets"" and ""new_facets"". It should be reduced to just the new facets (with the key changed to ""facets"").

Also the group and package controllers are adding both facets and new_facets to the context, should be new_facets only (but renamed to facets).

The facet_items() function in helpers.py should be removed, it uses the old facets structure and shouldn't be needed anymore with the new facets structure.

In facets.html, facet_sidebar() should be removed as it uses the old facets structure and facet_div() implements the same functionality but uses the new facets.

In facets.html, facet_list_items() will have to be updated to not use the facet_items() helper and to use the new facets structure instead.

Anywhere that ""new_facets"" appears it will have to be changed to ""facets"" (e.g. in the ckanext/multilingual/plugin.py.

This is the merge commit that introduced the duplication: https://github.com/okfn/ckan/commit/1153aa876f54c22289e460aeececea22d1d4d51d

This is the earlier commit where the search facets were refactored: https://github.com/okfn/ckan/commit/3970e52008b75933fda1be1d488bed2578d98c9c"	1331744085000000	1338204707000000
2234	enhancement		seanh	ckan-future	assigned		Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN templates	"You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

- Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds

- Feed items should probably be cached somewhere


Suggested 

- The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.

- For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference

- Caching is pretty crucial and should probably obey the ttl of the feed.

"	1331902755000000	1346669567000000
2235	enhancement		rgrp	ckan-future	new		Group drop down on dataset edit should use chosen and sort groups by name		1331907357000000	1338204726000000
2247	enhancement		zephod	ckan-backlog	new		Resource preview glitch in some browsers	"From Ira:
Preview for google spreadsheets are not displaying correctly for me in Firefox v.10.0.02, fine in Chrome. 
http://i.imgur.com/KJaqz.png"	1332246614000000	1332246614000000
2258	enhancement		rgrp	ckan-future	new		Customizable contributor agreement	" * Customize text at bottom of forms
 * Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...)"	1332751549000000	1338204747000000
2265	enhancement		dread	ckan-future	new		'More Like This' for a dataset	"When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar.

SOLR has a feature for finding documents similar to a particular document, called 'More Like This'.

We would like this for DGU."	1332865220000000	1339771350000000
2283	enhancement		rgrp	ckan-v1.8	closed	wontfix	[super] Data Previewer / Viewer v3	" * Basic geo map
 * Facets
 * Filtering
 * Pivot table support (?)
 * Export data (?)
 * Save data query and share
 * More and better graphing
 * Embedding! - /resource/../embed

User stories and analysis: http://ckan.okfnpad.org/feature-2283-data-viewer-v3"	1334138341000000	1340623843000000
2284	enhancement		ross	ckan-backlog	assigned		Local queries/views/viz in Related stuff.	"From #2204

not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ...

### Comments

In the simplest form, right now all I need to store is store the url to the resource plus hashbang config for recline. That would be super easy ...
"	1334218700000000	1346669513000000
2287	enhancement		seanh	ckan-future	new		Consistent datetime and time period strings localization	"Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support.

For example, on the user index page right now there are time period strings like ""less than 1 month"" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly).

ISO date format everywhere might be a good idea.

Need to consider local timezone issues.

Python standard library and Babel may have useful helpers for this.

Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page?"	1334254123000000	1339411445000000
2288	defect		seanh	ckan-v1.7	closed	wontfix	Fix i18n of strings on auth group add/edit pages	"There are a couple of strings on the authorization group add/edit form that are not being translated:

""Name: Please enter a value"" (error message when you try to add an auth group with no name)

""User name""

""Save"" button

The form seems to be built using FormBuilder. See ckan/forms/authorizationgroup.py, builder.set_displayed_fields()"	1334255003000000	1339411611000000
2309	enhancement		ross	ckan-v1.8	closed	wontfix	Organizations review with SH	Review organizations code with Sean and explain the key parts.	1334591679000000	1340623966000000
2316	enhancement		kindly	ckan-sprint-2012-04-30	closed	fixed	add resource status api to see the state of celely tasks	"add resource_status_show action function to see the state of celery tasks.

Need to get information from both out task status table and the celery_taskmeta table."	1335016037000000	1338205426000000
2317	enhancement		rgrp	ckan-v1.7	closed	wontfix	Corrections to dataset creation form for v1.7	" * Add home page url
 * Remove group option (seen it confuse several users) - perhaps replace with tags (?)
 * Check form looks ok in portugese (seems to be a problem with css)
"	1335022061000000	1340033433000000
2319	enhancement		rgrp	ckan-v1.8	closed	wontfix	Better auto-complete for groups on dataset edit page	Use jquery chosen?	1335211353000000	1340624083000000
2320	enhancement		rgrp	ckan-v1.9	new		Introduce new state: archived	Some datasets become obsolete: e.g. http://datahub.io/dataset/uncdb Do not want to delete them but want them marked as archived or deprecated or something.	1335211949000000	1340624119000000
2321	enhancement		rgrp	ckan-v1.9	new		Use resource names/titles in URLs	"Would make them much nicer but requires:

 * Ensure uniqueness (or deal with ambiguity)
 * (?) Slug/names versus titles
  * Aside: what is currently called name should probably be title
"	1335236412000000	1340624174000000
2324	enhancement		amercader	ckan-v1.9	new		Link API docs to relevant version not just latest	"The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version.

Also, remove the 'API' link (to /api/1) as pointless and misleading"	1335351532000000	1345199411000000
2328	enhancement		rgrp	ckan-v1.9	new		Link to Catalog API for each Dataset, Group and Tag	At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page?	1335442501000000	1340624331000000
2329	enhancement		rgrp	ckan 2.0	assigned		Add back in RSS/Atom links on relevant pages	E.g. on dataset page, on revision page, on user page and on search results.	1335442540000000	1346664271000000
2333	enhancement		rgrp	ckan-v1.9	new		Introduce new resource type datastore	"This is a resource that is *just* in the datastore. At the moment there are hacky ways to do this (create resource with meaningless URL etc) but we should do this properly:

 * New Option: DataStore Table in add a resource form
 * URL field is disabled (or points at /api/data or something rather pointless)
 * Download button is disabled
   * Better would be to find a way to serialize an ES table or stream it ...
"	1335651360000000	1340624473000000
2348	enhancement		ross	ckan-future	assigned		Java client library for CKAN	"Start a Java library for interacting with the CKAN Action API.

Currently requires support for Groups, Tags, Resources,  and Search.  

An example app would be very useful."	1335879980000000	1346670024000000
2378	enhancement		ross	ckan-future	assigned		Extract metadata directly from resources that contain it.	"Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata.  For example some jpeg files might contain Exif ( http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards' 

[ ] Where the metadata is available it  should be added as extra fields on the resource that is uploaded.  

  [ ] HTML?

  [ ] Image formats (exif/gif etc)

  [ ] PDF

[ ] Eventually consider ...

    [ ] MS Office documents


 

"	1336651088000000	1346669985000000
2395	defect		dread	ckan-backlog	new		paster db clean/init don't work when spatial extension enabled	"If you have a spatial enabled database then if you don't disable the spatial extension in the CKAN config temporarily then you get errors when you run paster db clean and paster db init.

Can't you just modify the clean and init functions to run without extensions enabled? 

The wider problem is that extensions do their own inits every time you do load_environment, which seems crazy and inefficient to me, since this occurs every time a request comes into CKAN. But that is another problem/ticket."	1337159793000000	1339771313000000
2400	defect		amercader	ckan-sprint-2012-05-29	closed	worksforme	Password reset fails	"One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed

{{{
""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)""
}}}



"	1337188614000000	1338205947000000
2404	enhancement		rgrp	ckan-future	assigned		Auto-screenshot for related items	"As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere 

Implementation: (?) Use phantomjs or something like http://snapito.com/
Investigate for 1.8, will be needed for TDH Oct."	1337583347000000	1346669526000000
2479	enhancement		markw	ckan-v1.9	new		Meanings of Author and Maintainer fields are unclear	"CKAN's default schema has fields for Author ('The name of the main contact, for enquiries about this particular dataset') and 'Maintainer ('another important contact person'). The distinction is not clear. Also the fields are often left blank.

The roles that seems most important are those of the original owner/publisher of the data, and the person maintaining the CKAN record/copy of it. So I suggest:

(1) Rename the fields 'Owner' and 'Maintainer'.

(2) Change the explanatory text for the relevant fields:

Owner: 'The person or organisation who create/collect/publish the data in this dataset.'
Owner e-mail: 'E-mail address for enquiries to the Owner named above.'
Maintainer: 'The person maintaining this dataset on [name of CKAN instance], if different from the above.'
Maintainer e-mail: 'E-mail address for enquiries to the Maintainer named above.'

(3) When a logged-in user creates a new dataset, the main form should have a checkbox, checked by default, marked 'I am the maintainer of this dataset'. If checked, the Maintainer name and e-mail fields are populated from the user's profile.
"	1338452898000000	1339771115000000
2480	enhancement		markw	ckan-v1.9	new		Better message when dataset has no resources	"If a dataset has no resources the resources list currently says
'(none)'.

Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'"	1338453093000000	1339771086000000
2483	enhancement		markw	ckan-v1.9	new		Non-local resources should not have Download links	"At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally.

This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table.

Suggested fix:

 * Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource)
 * Disable the Download button unless the resource is stored locally.
"	1338468734000000	1339771043000000
2486	defect		seanh	ckan-backlog	new		Should be able to use . in dataset names		1338655583000000	1339770978000000
2495	enhancement		rgrp	ckan-v1.8	closed	duplicate	Stats page has lost styling ...	http://datahub.io/stats	1339056932000000	1340624905000000
2508	enhancement		seanh	ckan-backlog	new		Make it possible to run CKAN tests for each language	"Mistakes in translated strings can cause CKAN to crash or otherwise not work, but it's not practical to manually test every page and function of CKAN in every language that we have new translations for before a CKAN release. It'd be great if the tests could automatically be run for each language.

This is probably a big job, we would have to get the tests to respect a language setting in the ini file, check for any individual test cases that specify the language (e.g. in the URL), and also fix test cases that look for specific English words in HTML output, etc.

In the meantime, a good stop-gap solution might be a script that tests for common mistakes in the po files."	1339411335000000	1339770771000000
2513	enhancement		ross	ckan-backlog	assigned		Dataproxy should not default to utf8	"Unless explicitly told by the source web server the dataproxy should not assume that the content it has can be encoded as UTF-8.  Even though the chars from 128 - 255 overlap an attempt to decode some byte array as utf8 will fail whenever a latin1 char whose bitpattern has the MSB set.

This will mean that the UTF8Recoder can be more rigid in its acceptance of data, Postel aside."	1339575820000000	1346669646000000
2529	enhancement		rgrp	ckan-v1.9	new		DataHub (or CKAN) widgets	"Simple widgets in pure JS. For example:

 * Count of datasets in a group (could generalise to a query but not sure how useful that is ...)
 * Embeddable list of top X (5) datasets for a given query
 * Embeddable list of *my* datasets

Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query).

Or: we have this attached to areas of site where relevant.

Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page).

Cf. http://okfnlabs.org/ckanjs/"	1339750049000000	1340624917000000
2531	enhancement		rgrp	ckan-backlog	new		New state option: archived / deprecated	"Deleted means things will get purged at some point.

Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated.

@richard cyganiak ..."	1339750787000000	1339770649000000
2535	enhancement		rgrp		assigned		SSL certificate for DataHub + https by default	"DataHub is increasingly used and we should ensure it uses ssl as part of general security.


See also #1446 (Need to support https login for multiple instances as part of the CKAN package install)"	1339758027000000	1346662082000000
2543	enhancement		icmurray	ckan-v1.9	new		facet.sort is not available in the package_search action	"Not all solr facet parameters are available through the pcakage_search action.  In particular, `facet.sort` has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort"	1340013335000000	1340633091000000
2544	enhancement		icmurray	ckan-future	closed	duplicate	facet.sort is not available in the package_search action	"Not all solr facet parameters are available through the pcakage_search action.  In particular, `facet.sort` has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort"	1340013421000000	1340015580000000
2546	requirement		ross	ckan-backlog	assigned		ODS Managing homepage content	"
== Requirements ==

Require the ability for users to control some level of content that is visible on the home page of their ODS installation.  This may be through RSS/Atom feeds (see #2234) or another mechanism but should result in admins being able to change blocks of text on their homepage.

This should not be configuration, but accessible through WUI.


== Interface ==

None

== User Stories ==

 * As a system administrator I want to have control over content displayed on the front page beyond featured/popular items.  
 
 * As a system administrator I don't want to manage content through having to write an extension.

 
== Tasks ==

[ ] Analysis



== Estimates ==


"	1340016842000000	1346663437000000
2552	enhancement		ross	ckan-future	assigned		Controlling access to features	"== Requirements ==

To provide a freemium service it is necessary to be able to provide differing __levels__ of functionality based on the '''type''' of user (see #2550). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks.

Initial implementation should focus on limiting access to datastore disk space.

== Interface ==

These changes are currently only for the data hub and should be kept as much as possible within the data hub extension.


== User Stories ==

 * As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited).  

 * As a system administrator I don't expect to need to manage the levels of users '''or''' the features that this applies to.
 

== Tasks ==

[ ] Clarification of requirements/analysis

[ ] Tests

[ ] Code

    [ ] Model

    [ ] API
  
    [ ] UI

[ ] Documentation


== Estimates ==
"	1340018770000000	1346669544000000
2554	enhancement		ross	ckan-backlog	assigned		Research Virtuoso cartridges	"Look into writing a cartridge for importing CKAN data into a Virtuoso quadstore

http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger#How Does It Work?"	1340026645000000	1346670433000000
2573	enhancement		icmurray	ckan-future	new		package_search does not allow solr's per-field facet parameters	"Solr allows its facet parameters to be specified on a per-field basis, eg. `facet.limit` applies to ''all'' facet fields, but solr allows it to be overriden for a specific field, eg. `facet.tags.limit`.

We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept.  See `ckan.lib.search.query.VALID_SOLR_PARAMETERS`."	1340112439000000	1340633101000000
2582	enhancement		rgrp	ckan-v1.9	new		Do not hide notes / readme on dataset pages	"Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?)

Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket."	1340312340000000	1340625009000000
2585	enhancement		seanh	ckan-v1.9	new		Escape solr control characters in search queries, add advanced search screen	"Suggestion from David Read:

We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for ""Spend over £25,000 - NHS Leeds"" then it would not come up with the dataset with that exact name. It was excluding datasets with the word ""NHS"" due to the dash/minus sign. It works fine if you escape the minus sign: ""Spend over £25,000 \- NHS Leeds"".

So in data.gov.uk I've added escaping of such control characters in
our plugin and this useful routine:

http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/

Perhaps you would consider providing this in CKAN core in future?

I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?"	1340360773000000	1340625078000000
2602	defect		seanh		closed	fixed	Users can set a dataset's organization to None, and then they cannot edit it	With the organizations and organizations_dataset plugins enabled login and create a new dataset, adding it to a group that you are a member of. Edit the dataset and set the group to none. Now you can no longer edit your own dataset. (You can also specify None for the organization when first creating the dataset, same problem.)	1340722892000000	1340805904000000
2607	defect		seanh	ckan-backlog	assigned		'Upload a file' appears on resource form when storage not enabled	"if the user tries to upload a file they will get ""Failed to get credentials for storage upload. Upload cannot proceed""

Maybe add a test for it this time, this bug has appeared and reappeared before"	1340803808000000	1346663383000000
2608	refactor		seanh	ckan-v1.8	closed	wontfix	Uploading a file to a resource hangs	I'm not seeing any javascript or python error, the upload just hangs.	1340806788000000	1343125400000000
2622	defect		seanh		new		Login fails in Opera 12	"Try to login to CKAN using Opera 12, get ""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"""	1340902602000000	1340902602000000
2625	enhancement		seanh	ckan-v1.9	new		Add i18n strings from non-core but supported extensions to ckan.pot file	Have to decide which non-core extensions are going to be supported first.	1341236903000000	1341236903000000
2635	enhancement		dread		new		Non-destructive SOLR reindex	"You can't run the search-index reindex on a live server because it will give us bad results for 2 to 3 hours while it runs. Can there be an option that doesn't delete the entire index at the start? 

Instead it could just delete any items that don't exist any more, then delete them and regenerate them one by one. So the total number of datasets doesn't change much."	1341829394000000	1341829394000000
2654	enhancement		ross	ckan 2.0	assigned		UI support for ordering groups on group_read page	"The group_index page has no support in WUI for ordering the groups displayed. Should allow sorting by name

Add support for this for datahub now, and discuss for new 1.9 UI"	1341943891000000	1346662156000000
2673	enhancement		rgrp		new		simplify set of options for resources	Far too many resource options. Lets restrict back to data file and API. Visualizations etc can either get linked in description or in the Related items.	1342300559000000	1342300559000000
2683	enhancement		seanh		new		Add no-cache header to _tracking API call's response to make sure it doesn't get cached		1342446577000000	1342446577000000
2686	defect		shevski	ckan-backlog	assigned		enabling datastore & data API breaks recline	"First I noticed that the gold prices dataset preview was not displaying & has data API enabled
Secondly I tried enabling datastore for http://datahub.io/dataset/adur_district_spending/resource/281dffa6-ea9b-4446-be41-05dced06591f and after I saved the preview no longer worked. Unticking the datastore & data api checkbox brought it back

Is this a known issue?"	1342516011000000	1346663300000000
2688	enhancement		ross		new		Allow ordering of groups in WUI	Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name.  It would be better if it could be sortable by name (or reversed) or by package_count (or reversed)	1342520875000000	1342520875000000
2719	defect		dread		new		Feeds controller does not catch NotAuthorized exception	Results in bad user experience and WebApp errors emailed out. Seen in 1.7.1	1342872863000000	1342872863000000
2731	enhancement		markw		new		Some sites permanently 'down for maintenance'	"A large number of XXX.ckan.net sites give the following message:

""This Site is Down for Maintenance
We apologize for the inconvenience.
~ The Open Knowledge Foundation sysadmins.""

The message is unhelpful and patently false - the sites do not exist. Some of them were supposed to have been redirected to a relevant group at thedatahub.org in this ticket (now closed):

http://trac.okfn.org/ticket/933

However, the redirection only seems to have worked in one case, http://si.ckan.net. 

The problem still affects the following sites - the first 4 of which have supposedly been merged:

 * http://lt.ckan.net
 * http://hu.ckan.net
 * http://be.ckan.net
 * http://gr.ckan.net

 * http://nz.ckan.net
 * http://bg.ckan.net
 * http://fi.ckan.net
 * http://katalogdanych.centrumcyfrowe.pl
 * http://pl.ckan.net
 * http://southampton.ckan.net
 * http://sl.ckan.net

Please sort this out by redirecting, removing the sites, giving a more helpful (and accurate) failure message, etc, as appropriate."	1343045168000000	1343051608000000
2732	enhancement		ross	ckan-backlog	assigned		New file upload functionality	"We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is http://ckan.okfnpad.org/uploads  

'''Simplifying uploads'''

Currently uploads are too painful/difficult/fiddly to use and/or configure.  We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces.

We want to fix:

 * File uploads themselves
 * Storage of uploaded files
 * Notification of the upload to other components
 
 
'''File uploads'''

Things file upload should do:

 * Allow sysadmin to disable
 * Allow auth'ed users to upload
 * Store whatever they send on disk, and store DB entry linking the file to the person
 * When creating the resource, the user should be able to choose from all of the files
   they have uploaded but not yet associated with a resource. This will allow for bulk
   upload and then a delayed association.  Whenver a user creates a resource they
   either upload a file now, or see previously uploaded files.

{{{
Can we do the upload asynchronously and then associate the 
uploaded key with the resource before the save ? What happens 
if the user tries to submit before asymc upload finishes ? Should 
we delay them?
}}}

The upload workflow should look like...

 1. File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data.  
 2. 
     a. When called via ajax then the ID of the newly created file should be returned,
     b. When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param.
 3. The resource save should check whether it has a file id and in that case updates the file object to point to the resource.

This should enable:

 * Separate file upload into a user's temporary store, either individually or as a batch.
 * Creating resources and simply choosing from previously uploaded, unassigned files
 * Adding files/data to a resource after the fact.

   

'''File storage'''

File storage should be local to the CKAN install, and not a remote service.  Any archiving to remove storage providers should be outside of the main request.

File storage should:

 * allow moving data, a sysadmin should be able to move the storage root and change configuration and  have the system continue running (i.e. don't store absolute paths).
 * provide maintainability, it should be easy to determine which old files are not associated with resources  and thus can be cleaned up.
 * allow for collection of information (i.e. estimate of storate space used)
 * check whether there is enough space and handling the conequences cleanly
 * ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage.
 * Have a configurable maximum accepted blob size during upload.
 * Should store what meta-data was provided with the upload, such as mimetype.

Somewhere in the DB we should store ...

||'''Column'''||'''Notes'''||
||id||An identifier||
||owner||The owning user, who uploaded the file||
||path||The path (from the 'storage root') to the file||
||size||The size in bytes of the file on disk||
||mimetype||The mimetype of the file, as provided by the uploader||
||upload_date||When the data was uploaded||
||resource||The ID of the resource it belongs to. A unidirectional relationship.||
||archived_url||The URL where this file has been archived||

Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files.


'''Notifications'''

We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified.  The primary use case for this is to notify the component that will translate/upload certain formats to the data store.

We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file).

"	1343058789000000	1346663270000000
2745	defect		amercader	ckan-v1.9	new		Password reset returns an exception if the key parameter is missing	"Instead of showing a notice, the password reset page throws an exception if the key parameter is missing:

{{{
Module ckan.controllers.user:329 in perform_reset
         c.reset_key = request.params.get('key')
               if not mailer.verify_reset_link(user_obj, c.reset_key):
                   h.flash_error(_('Invalid reset key. Please try again.'))
                   abort(403)
 if not mailer.verify_reset_link(user_obj, c.reset_key):
Module ckan.lib.mailer:100 in verify_reset_link
     if not user.reset_key or len(user.reset_key) < 5:
               return False
           return key.strip() == user.reset_key
 return key.strip() == user.reset_key
AttributeError: 'NoneType' object has no attribute 'strip'
}}}

Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8

instead of:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8

"	1343145931000000	1343145931000000
2748	enhancement		shevski	demo phase 5	new		add 'add new resource' button to sidebar	"When editing a resource you see the current and any other existing resources in sidebar

(e.g. see http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True)

Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page"	1343212878000000	1344503744000000
2761	enhancement		seanh	ckan-v1.8	new		Document all the errors you can get when setting up filestore, and how to fix them	"Add it to a 'Troubleshooting' section on the filestore page: http://docs.ckan.org/en/ckan-1.7.1/filestore.html

For the error messages and their solutions, see various threads on ckan-dev"	1343302566000000	1343302566000000
2762	defect		seanh	ckan-v1.8	new		test_related.py crashes	"/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError (cannot import name assert_regexp_matches)"	1343303753000000	1343303753000000
2763	defect		seanh	ckan-v1.8.1	new		Multilingual tests failing	test_multilingual_plugin.TestDatasetTermTranslation.test_dataset_index_translation, test_multilingual_plugin.TestDatasetTermTranslation.test_group_read_translation both failing for me on master	1343303819000000	1350303864000000
2764	enhancement		rgrp		closed	duplicate	Simplify filestore	"Definitely do:

 * Local file filestore: Remove pairtree (and OFS) and do something very simple
 * 

Options:

 1. Local file only. Allow for uploaders to GS / S3 in the background 
   * Advantages: simpler, if local upload can do progress bars etc etc
   * Disadvantages: strain for web-app (upload a 2GB file what happens)
    * This is probably fixable ...
 2. Stick with how we are  

=== New filestore without pairtree ===

Each uploaded file is located on disk at:

{uuid}/{filename}

Alternative:

yyyy/mm/dd/{uuid}/{filename}

Need the uuid to avoid collisions.

Metadata: Store no metadata."	1343303928000000	1343320440000000
2777	enhancement		icmurray		new		bug: user attributes		1343726363000000	1343726363000000
2783	enhancement		shevski		closed	wontfix	Way to generate homepage backgrounds for demos/ckan hosted	"Really we want this for CKAN Hosted & for creating demos.

Currently we're using http://leaflet.cloudmade.com/ - but by finding the area & taking screenshots. Would be better to interate with the app so that:
a) this happens automatically and 
b) correct attribution will be visible on maps"	1343812931000000	1343818702000000
2786	enhancement		shevski	demo phase 5	new		target blank HTML downloads	e.g. if I click on download here: http://s031.okserver.org:2375/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b can it take me to a new tab instead of navigating away from ckan?	1343819814000000	1343819814000000
2795	enhancement		toby	demo phase 5	new		Check validation of HTML, CSS, JS	Ensure that we are being standards compliant	1343903128000000	1343903128000000
2810	enhancement		kindly	ckan-future	new		heroku ckan support	Get ckan working on heroku	1344364858000000	1344364858000000
2811	defect		seanh	ckan-v1.8	closed	fixed	Author doesn't show on dataset read page	Edit a dataset and add an author, dataset read page still says 'author not given'. If you fill out both the author and author email fields then it works.	1344415861000000	1349869110000000
2817	defect		danieljohnlewis	demo phase 5	closed	worksforme	Item Type Reverts on Invalid URL	"Problem: When creating a related item, if you put in a invalid URL it loses the choice of Item Type (e.g. Visualisation, Application) and reverts to the first time (API).
Expected: It remembers which one was chosen"	1344504076000000	1344544901000000
2819	defect		danieljohnlewis	demo phase 5	closed	worksforme	Related Item Summary of Descriptions Markdown Issue	"Problem: When creating a related item it mentions that you can use markdown in the description, however markdown doesn't render in apps page nor in related items page.
Expected: Make shortened descriptions (on apps page & related items page) render markdown
"	1344504321000000	1344545479000000
2820	defect		danieljohnlewis	demo phase 5	new		English Language: Visualization -> Visualisation	"Problem: In the English version (which has a UK flag, indicating British English), the word ""Visualization"" is used. For an example see the ""Filter by type"" drop down on the /apps page.
Expected: This should be ""Visualisation"" in British English. Any instances of ""Visualize"" should be changed to ""Visualise"" too.
"	1344504455000000	1344504455000000
2821	enhancement		danieljohnlewis	demo phase 5	new		Featured Items on Filter	"Problem: On /apps page in the Filter Results box there is a ""Only show featured items"" checkbox, on selection it comes up with 0 solutions.
Expected: Presumably an admin can create ""featured items"" so that they can be randomly selected on front page (is this correct)? If there are no ""featured items"" in the whole database can this check box be hidden?
Bug is: no UI or obvious way to create featured items. Also the
checkbox looks un-styled"	1344504504000000	1344505492000000
2824	defect		danieljohnlewis	demo phase 4	closed	fixed	Multiple of the same Flag Issues	"There is some bizarre language flag issues.
1) Log in to http://s031.okserver.org:2375/ (I'm logged in as danieljohnlewis)
2) Visit the apps page: http://s031.okserver.org:2375/apps
3) Have a look at the languages drop down, flags look fine (although Deutsche does have an EU flag rather than the German flag.. but that's irrelevant)
4) Change filters to, say, type: Application and sort: Newest, and apply. Web address becomes: http://s031.okserver.org:2375/apps?page=1&type=application&sort=created_desc
5) Look at the languages drop down: Flags are largely incorrect, with more than one country having the same flag.

Please see the attached screenshot.

Note: this was originally tested with Google Chrome on Linux (Ubuntu)."	1344505215000000	1344532640000000
2825	defect		danieljohnlewis	demo phase 5	closed	fixed	Loss of URL encoding on language change on apps page	"Problem: On /apps page if you start in one language, filter the data, and then change languages it loses the correct format for the HTTP Get Parameters (and therefore loses the filters). Example, start in English language, change filter settings to to Application and Newest, change language to German.
Expected: Retain correct format for HTTP Get Parameters so that the filter settings are retained. Solution will involve not turning ""="" into ""%3D"" in the URL, i.e. turn off URL encoding."	1344505317000000	1344547172000000
2827	defect		danieljohnlewis	demo phase 4	closed	fixed	View Item - RSS Feed Issues	"Problem: Server Error (HTTP Code 500) when trying to see information related to an RSS feed... example: http://s031.okserver.org:2375/en/dataset/testerisation/resource/974df29e-77bb-4b5e-982c-cb7a295c1923
Expected: A page that looks a bit more like this: http://s031.okserver.org:2375/dataset/epims-lod2/resource/97531bd8-e75c-4efa-bfa2-bb51d0ba90d0"	1344505590000000	1344549139000000
2831	enhancement		aron.carroll	ckan 2.0	new		Create a limited subset of markdown that's supported	"Allowing people to use the full range of markdown results in extremely messy output across the site. I'd suggest limiting support to only a subset of common use cases.

 * Allow all inline elements, this allows bold, italic, code and links.
 * Allow lists.

Disallow

 * Horizontal Rules
 * Headings
 * Block quote and code (this may turn out to be useful and so could be included)

This way you get markdowns paragraph handling and a few inline styles without breaking the entire layout of the page.

Here's the full syntax if anyone is interested http://daringfireball.net/projects/markdown/syntax"	1344512467000000	1344512467000000
2832	enhancement		shevski	demo phase 4	closed	duplicate	can't add a dataset to more than one group	"add to groups is a drop down menu where you can only choose one
needs a new UI & logic allowing user to add new groups & potentially remove from other lists"	1344521472000000	1344542984000000
2833	enhancement		aron.carroll	demo phase 5	new		Load module templates before calling .initialize()	"I think this would be a nice feature for remote loading templates if the options.template value ends in "".html"".

{{{
ckan.module('my-module', {
  options: {
    template: 'my-template.html'
  },
  initialize: function () {
    this.template // This is the loaded template.
  }
});
}}}"	1344531939000000	1344531939000000
2834	enhancement		aron.carroll	demo phase 4	new		Defer all publish calls until all modules have been initialised	"Currently if a module uses sandbox.publish() in its initilaize() function then any modules initialised afterwards will miss the event.

We should queue all calls to .publish() in the pubsub module in an array until all modules have loaded then iterate over and execute them."	1344532093000000	1344856692000000
2835	enhancement		aron.carroll	demo phase 5	new		Client module needs a template loading method	"{{{
Client#getTemplate(name, params, success, error);
}}}


Where params, success and error are optional arguments. test/index.html already has an implementation called loadFixture()."	1344532233000000	1344532233000000
2838	enhancement		seanh	ckan-v1.9	new		Context variables accepted by action functions need to be documented	"I was doing this:

{{{
context = {'model': base.model, 'session': base.model.Session,
                    'user': toolkit.c.user or toolkit.c.author,
                    'extras_as_string': True}
group_dict = logic.get_action('group_show')(context,
                    {'id': group_id})
}}}

in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass {{{'extras_as_string': True}}} in the context. I don't think this or other context variables are documented anywhere."	1344630911000000	1344630911000000
2839	enhancement		seanh	ckan-v1.8	closed	fixed	Specifying sort order breaks search results on group page	Create a group, create some datasets and add them to the group, then view the group's page with a sort-order specified in the URL, e.g. {{{/group/foo?q=&sort=title_string+asc}}}, no datasets are displayed even though there are datasets in the group that match the search query. Remove the sort order from the URL and the datasets will appear.	1344787799000000	1344854522000000
2842	enhancement		seanh	ckan-v1.8	closed	fixed	Allow sort ordering of dataset listings on group pages		1344855695000000	1344855852000000
2844	enhancement		rgrp		new		SQL-only (no solr) version of CKAN	" * ~~Search needs to run of local DB (again)~~
 * paster db clean attemps to connect to SOLR (still works as does db first but then excepts which is not nice UX)"	1344859168000000	1345454527000000
2846	defect		seanh	ckan-v1.8.1	new		Organizations allows you to set an organization as the parent organization of itself	that doesn't make any sense	1344860536000000	1350303844000000
2850	enhancement		aron.carroll	demo phase 4	new		Add JSON support for IE7	"The JavaScript just isn't going to work without it. Look into either the newer JSON3:

http://bestiejs.github.com/json3/

or the old faithful:

https://github.com/douglascrockford/JSON-js/"	1344940692000000	1344940692000000
2851	enhancement		ross		new		Preview of PDF tries to connect to datastore	"e.g. http://datahub.io/dataset/ccc-gistemp/resource/80ebdbd6-d91b-4fef-9db4-d3dfbd7e868e

CKAN attempts to render the PDF as a datastore tabular date even though the mimetype of the PDF is correctly specified."	1345032766000000	1345032766000000
2854	enhancement		shevski	demo phase 5	new		no way to see which dataset an app is related to on /apps page	e.g. from http://s031.okserver.org:2375/apps?page=1 you cannot edit a related item as an admin or view what dataset the item is related to	1345048728000000	1345048728000000
2867	enhancement		ross		new		1.8 url_for error	"From: <thedatahub.org@s077.okserver.org>
Date: Thu, Aug 16, 2012 at 10:30 AM
Subject: WebApp Error: <class 'routes.util.GenerationException'>: url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6
To: ckan-sysadmin@okfn.org, kindly@gmail.com


URL: http://ckan.net/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:155 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
       
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
<<                                               r'\1', oldpath)
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug(""Calling controller class with WSGI interface"")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:239 in __call__
<<          try:
                   res = WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  res = WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """"""Hide the traceback for everything above this method""""""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.storage:190 in file
<<              return fapp(request.environ, self.start_response)
               else:
                   h.redirect_to(file_url)
>>  h.redirect_to(file_url)
Module ckan.lib.helpers:57 in redirect_to
<<      if are_there_flash_messages():
               kw['__no_cache__'] = True
           return _redirect_to(url_for(*args, **kw))
       
       def url(*args, **kw):
>>  return _redirect_to(url_for(*args, **kw))
Module ckan.lib.helpers:79 in url_for
<<          # fix ver to include the slash
               kw['ver'] = '/%s' % ver
           my_url = _routes_default_url_for(*args, **kw)
           kw['__ckan_no_root'] = no_root
           return _add_i18n_to_url(my_url, locale=locale, **kw)
>>  my_url = _routes_default_url_for(*args, **kw)
Module routes.util:265 in url_for
<<      if not isinstance(url, str) and url is not None:
               raise GenerationException(""url_for can only return a string, got ""
                               ""unicode instead: %s"" % url)
           if url is None:
               raise GenerationException(
>>  ""unicode instead: %s"" % url)
GenerationException: url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6
CGI Variables
CKAN_CURRENT_URL	'/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca'
CKAN_LANG	'en'
CKAN_LANG_IS_DEFAULT	True
CONTENT_TYPE	'; charset=utf-8'
DOCUMENT_ROOT	'/etc/apache2/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_CHARSET	'utf-8;q=0.7,iso-8859-1;q=0.2,*;q=0.1'
HTTP_CONNECTION	'close'
HTTP_HOST	'ckan.net'
HTTP_USER_AGENT	'Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)'
PATH_INFO	'/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca'
PATH_TRANSLATED	'/etc/ckan/datahub/apache.wsgi/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca'
REMOTE_ADDR	'127.0.0.1'
REMOTE_PORT	'37236'
REQUEST_METHOD	'GET'
REQUEST_URI	'/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca'
SCRIPT_FILENAME	'/etc/ckan/datahub/apache.wsgi'
SERVER_ADDR	'127.0.0.1'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.0'
SERVER_SIGNATURE	'<address>Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.22 (Ubuntu)'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x7f5312a51650>
beaker.cache	<beaker.cache.CacheManager object at 0x7f5312a51790>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f5312a516d0>>
beaker.session	{'_accessed_time': 1345109433.273328, '_creation_time': 1345109433.273328}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.handler_script	''
mod_wsgi.input_chunked	'0'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'8080'
mod_wsgi.process_group	'datahub'
mod_wsgi.request_handler	'wsgi-script'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(3, 3)
paste.cookies	(<SimpleCookie: >, '')
paste.registry	<paste.registry.Registry object at 0x7f5303b05610>
paste.throw_errors	True
pylons.action_method	<bound method StorageController.file of <ckan.controllers.storage.StorageController object at 0x7f530891fa10>>
pylons.controller	<ckan.controllers.storage.StorageController object at 0x7f530891fa10>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x7f52f864f850>
pylons.routes_dict	{'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'}
repoze.who.logger	<logging.Logger object at 0x7f53127b2b50>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 139994769488336>, 'friendlyform': <FriendlyFormPlugin 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>}
routes.route	<routes.route.Route object at 0x7f531296e690>
routes.url	<routes.util.URLGenerator object at 0x7f52f82cd450>
webob._parsed_query_vars	(GET([]), '')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52f5870cd8>
wsgi.version	(1, 1)
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0x7f52f82cd450>, {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'})
"	1345114669000000	1345128279000000
2869	defect		shevski	demo phase 4	new		user profile not linked on disqus until page refresh	"Reported by Daniel:

Problem: DISQUS comments don't appear with profile details (inc. username).
How to reproduce:
1) Go to a dataset, e.g. I went to http://s031.okserver.org:2375/dataset/ff
2) I logged into DISQUS using the Google Authentication, e.g. in my case that is okfn.daniel.lewis@gmail.com
3) I made a comment and submitted
4) It comes up with the comment, but no linked profile until the page gets refreshed"	1345114714000000	1345114714000000
2871	defect		shevski	demo phase 3	closed	fixed	user profile about text does not show up	"tested with my http://s031.okserver.org:2375/user/shevski and daniel's account. I have added about text and yet see '
Irina Bolychevsky

You have not provided a biography.'

The about text is saved (and viewable on the edit form itself) but does not show on profile
"	1345114953000000	1345121991000000
2872	defect		seanh	ckan-v1.8	closed	fixed	Non-open licenses are marked as open	Set a dataset's license to 'Other (Not Open)', the dataset will have an Open Data icon next to it 	1345124135000000	1350304179000000
2875	enhancement		toby	demo phase 4	closed	fixed	Fanstatic css issues	"Fanstatic does not support 

@import ...

relative imports for images etc


need to get a fix for these asap

"	1345208199000000	1345214469000000
2881	defect		seanh	ckan-v1.8.1	new		Remove or improve 'Domain Model' docs page	"http://docs.ckan.org/en/ckan-1.8/domain-model.html

Currently not very useful. Also these pages:

http://docs.ckan.org/en/latest/domain-model-dataset.html

http://docs.ckan.org/en/latest/domain-model-resource.html

http://docs.ckan.org/en/latest/domain-model-task-status.html

This just seems very incomplete to me (surely there is more to the domain model the dataset, resource and task status?) and very likely to not get updated as changes are mode to the code. Would anyone seriously refer to this rather than just looking at the source code?

If we want to have something along these lines in the docs, I suggest adding docstrings to packages, modules, classes and methods and then using autodoc to pull them into sphinx."	1345539917000000	1345626056000000
2882	defect		seanh	ckan-v1.8.1	new		Remove User Stories Overview from docs?	"Why is this here? Is it up to date? Is anyone keeping it up to date? Does this belong in some sort of CKAN planning or roadmap document rather than in the documentation?

http://docs.ckan.org/en/latest/user-stories.html"	1345540723000000	1345540723000000
2883	enhancement		seanh	ckan-v1.8.1	new		Add high level dev overview to 'For CKAN Developers' section of docs	Here's a draft: https://gist.github.com/3414107	1345546355000000	1345629890000000
2884	defect		seanh	ckan-v1.8.1	new		Lock icon shown next to unknown licenses	Set the license of a dataset to blank (the first option) or to 'License Not Specified' or 'Other (Non-Commercial)' and a lock icon appears next to the license on the dataset read page. Shouldn't this be a question mark or something?	1345564873000000	1345564873000000
2885	enhancement		seanh	ckan-v1.9	new		Labels (editor, admin) not translated in authorization pages for datasets or groups		1345625351000000	1345625524000000
2886	enhancement		ross	ckan 2.0	new		Configurable related items	"Related items (Apps & Ideas) have a collection of types, which are currently fixed.  Some of these types are ambiguous, such as ideas in that they may not have a link - and if they do it is likely to be to a blog post - another existing type.

Whilst the URL is required, and we think this should stay required, we should also allow users to change the types found in Apps & Ideas to a shortened list that suits their requirements."	1345625718000000	1345625718000000
2887	defect		seanh	ckan-v1.8.1	new		"""Welcome to CKAN!"" on front page is untranslatable"	"ckan/templates/home/index.html contains:

    <h1 class=""page_heading"">Welcome to ${g.site_title}!</h1>

which means that just ""Welcome to"" ends up in the ckan.po files for
translation. This makes the phrase untranslatable in languages that have a different word ordering than English where the site title needs to be somewhere other than at the end of the sentence."	1345627079000000	1345627233000000
2888	enhancement		kindly		new		Datapreview in Iframe	Try and put datapreview in Iframe.	1345745461000000	1345745461000000
2889	enhancement		seanh	ckan-v1.9	new		Support JSON values in extras, when returning JSON from API	"For example, edit the warandpeace dataset and add an extra with key ""comments"" and value ""[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]"", now visit API URLs such as:

/api/search/dataset?sort=metadata_modified+desc&all_fields=1&limit=10

/api/rest/dataset/warandpeace

/api/action/package_show?id=warandpeace

The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects."	1345811737000000	1345811737000000
2890	enhancement		seanh	ckan-v1.8.1	new		Collect data previews and data store docs in one chapter	"Currently there is this page:

http://docs.ckan.org/en/latest/data-viewer.html

which covers Recline Data Explorer and other kinds of data preview in CKAN. It is under the Publishing Datasets section in the documentation. I had to to a search for 'recline' to find it.

Separately there is this page: http://docs.ckan.org/en/ckan-1.7.1/datastore.html which covers datastore, datastorer, and the data api.

I suggest collecting this together in one chapter called 'Data Previews'. If I understand it right the general gist would be:

CKAN has builtin previews of data resources on resource pages, enabled by default.

Images, Google Documents, and web page resources will be loaded into embedded iframes for preview.

Text-like files will be displayed raw.

CSV or Excel files uploaded to CKAN will be previewed using Recline Data Explorer.

Additionally, you can enable CKAN's DataStore, requires you to install ElasticSearch and nginx and put datastore.enabled=1 in your ini file. Lets you use the Data API to query data.

Does having DataStore enabled mean you get preview of more types of resources? Any resource that's available via the Data API will be previewed using Recline, 

You can install ckanext-datastorer, and then CSV and Excel files _linked to_ as CKAN resources will be previewed using Recline also. Requires celeryd."	1346149236000000	1346175867000000
2891	defect		seanh	ckan-v1.8.1	new		Update or remove DataStore google presentation	"Currently this page:

http://docs.ckan.org/en/ckan-1.8/datastore.html

has an embedded Google docs presentation about data store which contains some outdated information, e.g. links to datastoreclient which was merged into ckanclient.

Suggest moving the google presentation into the documentation proper, so it works with grep etc."	1346149423000000	1346154177000000
2892	defect		seanh	ckan-v1.8.1	new		IDatasetForm's form_to_db_schema() breaks API?	"It looks like when an IDatasetForm plugin with a form_to_db_schema() method is active, then the posted data dict is validated against the schema even if the user is posting to an API endpoint like /api/action/package_update etc. This breaks the API, for example 'name' and 'title' are optional keys for package_update (you can supply only the 'id'), but the schema validation will fail if either are not present in the posted dict.

It may be we need different schema for package create and package update, currently IDatasetForm has just the one form_to_db_schema() method."	1346154125000000	1346154163000000
2893	defect		seanh	ckan-v1.8.1	new		Tags missing from form_to_db_package_schema()	"With an IDatasetForm plugin active with a form_to_db_package_schema() method that returns a schema based on ckan/logic/schema.py:form_to_db_package_schema() I'm getting validation errors like this:

[ckan.controllers.api] Validation error: ""{'__type': 'Validation Error', '__junk': [u'The input field __junk was not expected.']}""

They seem to be caused when a data dict with key 'tags' is passed to e.g. the package_update API, for example in ckan/tests/functional/api/model/test_vocabulary.py there are various tests that get this error when they call package_update (if you run the tests with a suitable IDatasetForm plugin active).

In ckan/logic/schema.py:form_to_db_package_schema() 'tags' is for some reason deliberately removed from the schema."	1346155864000000	1346155864000000
2894	defect		seanh	ckan-v1.8.1	new		Empty lists get stripped from package dicst in package_show()	"If there's an IDatasetForm plugin active with a form_to_db_schema(), then packages with e.g. no extras get the 'extras' key entirely stripped by the validation in package_show, which causes some tests to crash, e.g. in ckan/tests/functional/api/test_activity.py (if the tests are run with a suitable IDatasetForm plugin active).

If a package has no extras the dict returned by package_show should still have an 'extras' key with an empty list as the value.

Suspect this also effects packages with no tags, no groups, etc."	1346157634000000	1346157634000000
2895	enhancement		rgrp		new		QA pages (e.g. broken resource links) are not paginated	And as a result timeout on e.g. the datahub ...	1346175485000000	1346175485000000
2896	defect		seanh	ckan-v1.8.1	new		resource_create crashes on ValidationErrors with KeyError: 'resources'	"ckan/logic/action/create.py:resource_create():

    try:
        pkg_dict = _get_action('package_update')(context, pkg_dict)
    except ValidationError, e:
        errors = e.error_dict['resources'][-1]
        raise ValidationError(errors)

This assumes that package_update always returns an error dict with 'resources' in it. This assumption doesn't hold."	1346257754000000	1346257754000000
2897	defect		seanh	ckan-v1.8.1	new		db_to_form_package_schema() crashes historical dataset page	"If there's an IDatasetForm plugin active with a db_to_form_schema() method based on db_to_form_package_schema(), then viewing historical versions of datasets (e.g. http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191) crashes.

Reason is that revison_id and revision_timestamp need to be added to db_to_form_package_schema(), something like:

        schema.update({'revision_id': [validators.ignore_missing, unicode]})
        schema.update({'revision_timestamp':
            [validators.ignore_missing, unicode]})
"	1346330495000000	1346330495000000
2898	defect		johnmartin	demo phase 5	new		Looses data entered on step 1 of create dataset after login	"To reproduce:

1. Enter details for new dataset without being logged in

2. Hit login page for just before step 2

3. Once logged in you end up back at the empty add dataset form (instead of expected 2nd step of dataset edit)"	1346332094000000	1346332120000000
2899	enhancement		johnmartin	demo phase 5	new		Step 2 > Save & add another bug	"Goto:

1. Add dataset

2. Get to step 2

3. Click ""Save & add another"" without adding any data

4. Ends on step 3

Should: fail to validate and remain on step 2"	1346332519000000	1346332519000000
2900	defect		seanh	ckan-v1.8.1	new		snippets/package_list.html template crashes on dataset that has no 'notes'	"Sometimes a dataset dict comes through package_show() with 'notes': None and then the 'notes' key gets stripped from the dict when validation happens (validation only happens if there's an IDatasetForm plugin with a db_to_form_schema() method) and then package_list.html template crashes, e.g. on user profile pages.

Fix in the template? Or in the schema?

The template also crashes on packages with no tags (empty list gets stripped from dict) and probably other empty fields as well"	1346335207000000	1346335788000000
2901	enhancement		johnmartin	demo phase 5	new		Language Dropdown bug in footer with IE7	"The language dropdown in Internet Explorer 7 behaves oddly. Essentially I think the problem is with the way that the dropdown decides to go into ""drop-up mode"" (e.g there aint enough screen space below the initial state so it has to drop up)."	1346754674000000	1346754674000000
2902	enhancement		ross	ckan 2.0	new		genshi is used in Group controller	"The Group controller uses the genshi module where it may be better for it to use Markup() in the template.

See https://github.com/okfn/ckan/pull/117/files#r1524863"	1346782230000000	1346782230000000
2904	enhancement		johnmartin	demo phase 5	new		Show more formats/groups/tags shouldn't display if there aren't more	On the search results page when there are no more formats, groups or tags to display it should not display the show more links.	1346854444000000	1346854444000000
2905	enhancement		johnmartin	demo phase 5	new		Add dataset URL key behaviour	"On step 1 for adding a dataset can we add a ajax ping that checks if the URL that is generated by the JS is taken? Behaviour should be as follows:

1. User types title in box (as current)

2. URL key is regexed by JS (as current)

3. The JS does a ajax call to check if the generated URL key is valid. The ajax callback returns a JSON dump which contains what the URL key should be (e.g. if 'test' was taken it returns 'test-1')

4. If the URL key returned in the ajax is different than the one generated by the JS then:

4a. If the user hasn't clicked the edit button then: it automatically updates the URL key field with the URL key supplied in the ajax

4b. If the user has clicked the edit button then: a form error next to the input appears saying that the URL key is taken and within that error is suggests the URL key returned by the ajax as alternative (the user can click said URL key within the error to automatically choose that one)

The URL key isn't reserved by the ajax and upon form submission the usual validation should still apply."	1346855283000000	1346855283000000
2906	defect		dominik	demo phase 5	new		Wrong flags in language dropdown	"The flags in the dropdown on details pages are not right. 

You can see the error on http://s031.okserver.org:2375/dataset/malawi-aid-projects/resource/b717c20e-2006-4ad4-82d2-59b57ebc1ab0."	1346861144000000	1346862889000000
2908	enhancement		johnmartin	demo phase 5	new		Dataset related with image should retain aspect ratio	At the moment if you have an related item with an image associated to a dataset that appears in the side column it's resized but doesn't keep it's original aspect ratio.	1346931162000000	1346931162000000
2909	enhancement		johnmartin	demo phase 5	new		User profile headers are a bit weird	They seem to be not exactly well thought out. Needs some thought on better display of the information that they contain.	1346931348000000	1346931348000000
2910	enhancement		johnmartin	demo phase 5	new		User listings are not very compelling	They are currently are just a small image / name. They should be: Larger image, Full name (username), description, edits/dataset	1346931587000000	1346931587000000
2911	enhancement		kindly		new		Internal documentation of Organization Groups	A summary/user story doc of how organizations and groups are expected to work.	1346941384000000	1346941384000000
2912	defect		seanh	ckan-v1.8.1	new		ckanext/organizations: Neither public or private initially selected when adding dataset	When adding a dataset with the organizations extension on, neither the Public or the Private radio button is initially selected. Public should be selected.	1347271076000000	1350303897000000
2913	defect		seanh	ckan 2.0	new		IGroupForm has package_form() method	Should be group_form() I think	1347374739000000	1350303779000000
2914	defect		seanh	ckan-v1.8.1	new		Add form_to_db_schema_options() to IDatasetForm and IGroupForm	This optional method is supported when creating or updating groups and packages and is used in DefaultGroupForm and DefaultPackageForm, but seems to be missing from the interfaces. Should be added with docstring.	1347445795000000	1350303928000000
2915	refactor		seanh	ckan 2.0	new		Refactor form_to_db_schema_options()	"Having two methods form_to_db_schema(self) and form_to_db_schema_options(self, options) seems unnecessary, why not just have form_to_db_schema(self, options=None)?

Fixing this might break existing extensions although I don't think any are using form_to_db_schema_options() so it should be okay.

The same thing goes for db_to_form_schema_options().

Also why are we passing a dict 'options' that always contains the same three keys 'api', 'context' and 'type'? Why not three params api=None, contenxt=None, type=None?"	1347446269000000	1347447588000000
2919	defect		seanh	ckan-v1.8	new		Remove IController from docs	Still in docs but seems gone from code	1347461253000000	1347461253000000
2920	defect		seanh	ckan 2.0	new		Genshi text template message extractor no longer needed?	"In setup.py, do we still need:

    ('templates/**.txt', 'genshi', {
        'template_class': 'genshi.template:TextTemplate'
    }),

in message_extractors? Doesn't like there are any txt files in there
anymore."	1347530249000000	1347530249000000
2921	defect		seanh	ckan 2.0	new		Add docstring to top of lib/extract.py file	"I think it couldn't hurt for this module to have a docstring at the top
of the file explaining what the module is for. I know from setup.py that
it's there to provide the extract_ckan() string extractor function for
Babel, but I think looking at the file on its own it's not so obvious.

Also a couple of other small fixes:

jinja2_cleaner looks like a helper function only meant to be used by
extract_ckan(), might make things clearer if it was called
_jinja2_cleaner(), just so it doesn't look like a public function the
module wants to export.

jinja_extensions seems to be a module-level variable that is only used
in one function, could be moved into the function, unless you think
there might be more functions that want it in future."	1347530407000000	1347530407000000
2922	defect		seanh	ckan 2.0	new		Better docstring for CKANInternationalizationExtension	"I'm unsure about what's going on here. As I understand it, when we run
`python setup.py extract_messages` it's going to use the extract_ckan()
function from ckan/lib/extract.py to process the HTML files. For the
HTML files that are jinja2 templates, extract_ckan() will call
jinja2_cleaner() which will call regularise_html() on the strings.  So
the strings are regularised when they are extracted from the source
files.

But then we have the parse() method of CkanInternationalizationExtension
also calling regularise_html(). I don't get what's happening here. Why
do the strings need to be regularised twice?

My guess is that CkanInternationalizationExtension is used when the
strings are extracted from the templates at runtime, and they need to be
regularised at this time in order to match them against the regularised
strings in the mo files to find the translations to output?

Maybe CkanInternationalizationExtension needs a better docstring saying
what it does?"	1347530507000000	1347530507000000
2923	defect		seanh	ckan 2.0	new		Change regularise -> regularize	The function is called regularise_html(), can't remember what file it's in.	1347530582000000	1347530582000000
2924	defect		seanh	ckan 2.0	new		Better docs for trans js command, and add to release process	"Add a better docstring to for trans js command explaining what it does and why, and how to use it.

Also add this command to the CKAN Release Process as it's needed there"	1347530671000000	1347530671000000
2925	defect		seanh	ckan 2.0	new		Remove trans mangle paster command?	"- Is `trans mangle` really necessary? If you upload a pot file to
  Transifex, it can generate a po file for you with 100% strings
  translated into a fictional pseudo language where everything is really
  long strings of unicode characters. I found this worked well for
  coverage testing, and also tests handling of unicode and long strings
  all over the place."	1347530768000000	1347530768000000
2926	enhancement		seanh	ckan 2.0	new		I*Form tests for per-type templates	"Finish the tests for the per-package-type custom template support in IPackageForm, IGroupForm and IOrganizationForm (i.e. the new_template(), read_template() etc. methods).

The new way that template inheritance works in jinja looks like it should allow this to be tested nicely without duplicating entire templates in the example extensions."	1347552081000000	1347552081000000
2927	enhancement		seanh	ckan 2.0	new		Test new package/group/organization type URLs support	IPackageForm, IGroupForm and IOrganizationForm allow you to define e.g. new package types with new top-level URLs, but the example extensions don't have tests for this yet.	1347552153000000	1347552153000000
2928	enhancement		seanh	ckan 2.0	new		Run CKAN tests with example_i*form extensions enabled	Before releasing CKAN 2.0 we need to run all the CKAN tests with a modified test-core.ini with the example_idatasetform, example_igroupform and example_iorganizationform plugins enabled. If any tests fail, fix the bugs. This needs to be done for each release so add it to the release process.	1347552334000000	1347552334000000
2929	enhancement		seanh	ckan 2.0	new		Remove is_fallback() from I*Form extension interfaces?	"If a plugin wants to take over the default package type for example, it can return ['dataset'] from its package_types() method. I don't see how there can be a package type that is not the default 'package' and is not returned by the package_types() method of a plugin, because the URLs for such a package type would not be mapped. Am I missing something or is is_fallback() not needed?

Same for IGroupForm and IOrganizationForm"	1347552597000000	1347552597000000
2930	defect		seanh	ckan-v1.8.1	closed	fixed	convert_from_extras() returns qupted strings from API	"Use an IGroupForm plugin to add a custom metadata field to groups using convert_to_extras() and convert_from_extras(), when calling group show the value comes back quoted, e.g. '""my_value""'

Should add tests to example_igroupform and others that setting and getting the custom fields works through the action API."	1347639864000000	1361988592000000
2931	enhancement		seanh	ckan 2.0	new		Better docstring for app_globals.py	"''' The application's Globals object ''' is not very informative.
"	1347891194000000	1347891378000000
2932	enhancement		seanh	ckan 2.0	new		Add docstring to system_info.py	What is the system_info table for? It looks like a way to override config file settings in the database, perhaps so that admins can use a web interface to change site title etc. Could do with a docstring explaining the intended purpose of the table.	1347893429000000	1347893429000000
2933	enhancement		seanh	ckan 2.0	new		Update theming docs	"There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs.

I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents."	1347895272000000	1347895272000000
2934	enhancement		Dominik		new		Webtests for CKAN	"The tests in CKAN only test single pages (functional tests) or single methods (unit tests). In order to make sure that the whole system still works as expected, we need tests that cover whole processes. 

An example user story to be tested would be:
""As a user with create permissions I want to be able to log in, create a new resource, save it and then make sure that it is in the system.""
or:
""As a user I want to use the search functionality and open a resource from the search page.""

These user stories require a different testing system with interaction and processes. "	1347966647000000	1348060201000000
2936	defect		seanh	ckan-v1.8	new		Updating a group via the API clears its packages	"If the group dict that you post to the API does not have any 'packages' key then all the group's packages get removed. I think it would be better if you could just update e.g. the group's description without having to also post the list of packages, and apparently this is how other update API actions work.

Might be worth checking all the update API actions for this behaviour, and making sure they're all consistent"	1348048066000000	1348048066000000
2937	defect		seanh	ckan-v1.8	closed	fixed	GroupController.history() missing extras_as_string	"GroupController's history() method doesn't pass 'extras_as_string': True in the context when it calls group_show. This means that if you have an IGroupForm plugin that is adding a custom metadata field and using convert_to/from_extras() then a field value of 'foo' will be returned as '""foo""'

Other GroupController and PackageController methods do pass 'extras_as_string'"	1348155730000000	1348238875000000
2943	enhancement		dominik		new		Chrome does not resize preview	Chrome does not resize iframe after a full refresh/ on first load	1349089686000000	1349090759000000
2944	enhancement		dominik		new		Recline preview does not work in Opera	Opera cannot show a recline preview. I shows 0 records.	1349090802000000	1349090802000000
2945	enhancement		dominik		new		Pdf preview does not load in IE	The pdf preview does not load in IE 9.	1349090869000000	1349090869000000
2946	enhancement		dominik		closed	duplicate	Pdf preview does not load in IE	The pdf preview does not load in IE 9.	1349090873000000	1349090992000000
2947	enhancement		dominik		new		Redirect to the resource page from /resource/{res-id}	"Use story:
As a user of the datasore I want to be able to put the resource id without the dataset name in a url and see the resource page.

Proposed redirect:

/resource/{res-id} -> dataset/foo-bar/resource/{res-id}"	1349097325000000	1349097325000000
2948	enhancement		dominik		new		Negative range breaks datastore backend	Recline allows negative ranges. The backend returns an error 500 for that which breaks the recline preview.	1349101854000000	1349101865000000
2950	requirement		icmurray	ckan 2.0	closed	fixed	paster command to minify javascript and css	"With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment.  One of these is:

 - the fanstatic_resources defined in `ckan/lib/fanstatic_resources.py` (bottom of module) should be prepared, and minified.

This ticket is to:

 - provide a paster command which when run will generate the minified javascript as css files.
 - stop the auto-minification of files when CKAN starts up.
 - remove reference to the minified files in the `.gitignore` file.
 - add instruction to the release process to run this command, and check-in the minified files to the repo.

## Background

Currently, minification works seamlessly without the need for any preparation when CKAN is started in a development setup.  But on a production site, the webserver will (almost certainly) not have write-access to the directories that will contain the minified files.  And so the minification will fail, and the site will end up serving the un-minified media, or even *old* minified media.

One way around this would be to allow webserver write access to the directory its serving out of.  But this is not generally considered good practice.

Another method would be to distribute the minified files with CKAN.  This ticket describes how to do this without causing a lot of noise in the commit history of the repo.

The auto-minifcation occurs when importing `ckan/lib/fanstatic_resources.py`.

## Changing the process slightly

The reason for moving away from auto-minifying files at start-up, to minifying files when running a paster command is:

 - we should be distributing the minified files when we make a release (as we do with translation files).  This makes it easier to install CKAN from source.

 - if we distribute the minified files, then they need to be checked-in to the repo.

 - if they are auto-minified, whenever small changes are made, then this will create a lot of noise in the repo.  It will be clearer to have the minified-files generated manually as part of the release process. (Or whever we make a deployment internally)."	1349118376000000	1349189654000000
2951	enhancement		icmurray	ckan 2.0	new		Paster command for building css from less	"With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is:

 - the less files should be compiled to css (main.css, not just main.debug.css)

This ticket is to provide a paster command which will compile the .less into the main.css file.  The idea of the paster command is that it will be run by developers, and they will check in the resulting .css files.  It will also provide a convenient mechanism for production scenarios with node.js installed on the production server, as they will be able to compile the .less there as well.

## Background

The .css files that need to be served are built using `less`.  The css files that are generated *are* checked-in to the repo; but they are a build artifact.  So the general workflow is:

 - commit changes to the less files
 - build the main.css file and commit

This paster command slots into the above workflow for convenience.

## Notes

 - doc/frontend-development.rst
 - bin/less

to run this paster command, `Node` will be required.  So that dependency should be checked.

## The paster command

This is the proposed behaviour of the paster command:

 - ensure `custom.less` does not exist
 - for each colour in{fuchsia,green,maroon,red}:
    - generate a `custom.less` file for $colour.  There's a paster command that does this already: `paster color <color name>.
    - generate the `css` from the `less` files.  ie - the equivelant of running `bin/less --production`.
      - this will generate a `main.css` file, which should be renamed to `$color.css`.

 - ensure `custom.less` does not exist
 - generate the `css` from the `less` files.  This will create a `main.css` with the default colour scheme.

 - call the paster command that minifies css and js files.  (This command does not exist yet, see #2950)"	1349118740000000	1349175030000000
2953	defect		dominik		closed	fixed	Server error in template directories	"If you go to: /{any template dir} for example /home or /related, a Server errror occurs. 

IOError: [Errno 21] Is a directory: u'/../venv/src/ckan/ckan/templates/home'"	1349252920000000	1349257893000000
2957	task		dominik		new		New datastore on postgres prior to 9.0	The ``GRANT SELECT ON ALL`` and ``ALTER DEFAULT PRIVILEGES`` are not available in Postgres prior to version 9.0. 	1349292214000000	1349292214000000
2958	defect		seanh	ckan-v1.8	new		Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8	"e.g. 2012_08(주요국가).xls

I tested in CKAN 2.0 and it seemed to work fine there."	1349342976000000	1349342976000000
2960	task		dominik	ckan-backlog	new		Private datasets are not private	At the moment everyone can read all resources. There is no support for hiding resources that are in private datasets.	1349694255000000	1349694255000000
2962	enhancement		dominik	ckan-backlog	new		Search across multiple ckan instances	"Could be done by:
- using the solr distributed search
  - difficult set up
- merging result sets from apis
  - make sure that results can be merged properly (score, facets, ...)"	1349736622000000	1349736622000000
2963	defect		amercader	ckan-v1.8	new		Timeout on tag pages with lots of datasets	"e.g. http://thedatahub.org/tag/lod

Tags with less datasets work fine (e.g. http://thedatahub.org/tag/railways)"	1350295999000000	1350295999000000
2964	defect		seanh	ckan 2.0	new		Last organization admin can remove herself	"If you are the only admin of an organization you can edit the organization's members and demote yourself, then the org has no admins and no one (except sysadmins maybe) can edit it.

Last admin should not be able to remove or demote herself.

Also applies to groups."	1350296058000000	1350296058000000
2965	defect		amercader	ckan-v1.8	new		Stats extension broken on 1.8	" * Graphs not showing (looks like a flot related file is missing)
 * Wrong groups counts (e.g. Data Explorer Examples show 1800 datasets when it onlu has 8)"	1350296141000000	1350296157000000
2966	enhancement		seanh	ckan 2.0	new		'Add' button text is wrong when editing organization members	e.g. if I just changed a member's capacity I am not adding anything 'save' is better maybe	1350296149000000	1350296149000000
2967	defect		seanh	ckan 2.0	new		Organization members edit page reloads after demoting self	Edit an organizations members page and demote yourself from organization admin, after saving the members edit page reloads even though you are no longer an admin and should no longer be able to access this page.	1350296227000000	1350296227000000
2968	defect		seanh	ckan 2.0	new		Anyone can access organization members page	The button will not show if you are not authorized but browse to /organization/members/foo and you can edit the members, it does stop you when you try to save your changes, but you shouldn't be able to get to the page at all	1350296355000000	1350297070000000
2969	defect		seanh	ckan 2.0	new		Group members page 500s	The group members page (e.g. /group/members/roger) seems broken, IDs instead of user names are shown for the members, and clicking on a member 500s	1350296454000000	1350296454000000
2970	defect		seanh	ckan 2.0	new		Organization and group member links use id not name	e.g. it the 'Members' button links to /organization/members/0a44... instead of /organization/members/foobar	1350296531000000	1350296531000000
2971	enhancement		seanh	ckan 2.0	new		"""Are ytou sure you want to delete this member?"" should say which member"	when deleting members from groups and orgs	1350296840000000	1350296840000000
2972	enhancement		seanh	ckan 2.0	new		Remove any imports of authz.py and delete file	it is no longer used	1350297442000000	1350297442000000
2973	enhancement		seanh	ckan 2.0	new		Move new_authz.py into logic/auth/__init__.py	"Makes sense to keep all auth stuff under logic/auth.

Also decide which functions from new_authz.pu should be marked private with _.

Also decide which functions from new_authz.py are really helpers for the auth functions in get.py, update.py and delete.py, and should maybe moved to another file, e.g. logic/auth/helpers.py"	1350297545000000	1350297545000000
2974	enhancement		seanh	ckan 2.0	new		General of all auth functions	"From Toby:

A general cleanup of all auth functions (in logic/auth) to check their fitness, error messages, etc."	1350297612000000	1350297612000000
2975	enhancement		seanh	ckan 2.0	new		Tests for auth functions and new actions	"From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions
"	1350297660000000	1350297660000000
2976	enhancement		seanh	ckan 2.0	new		Polish group and organization member pages	"From Toby: A little polish to the member pages and probably a little hardening of
the controller and logic actions"	1350297709000000	1350297709000000
2977	enhancement		seanh	ckan 2.0	new		Fix user autocomplete on group and organization member pages	"From Toby: @johnmartin the user autocomplete on
member add needs fixing it gets data but does not understand what to do
with it
"	1350297749000000	1350297749000000
2978	enhancement		seanh	ckan 2.0	new		Tests for permissions for organizations and groups	"From Toby: checking the logic around the who can do what from the user stories -
via some tests may be a good approach
"	1350297806000000	1350297806000000
2979	enhancement		seanh	ckan 2.0	new		Requesting membership to groups and organizations	"From Toby: The requesting membership user stories and implementation remain needed
"	1350297844000000	1350297844000000
2980	enhancement		seanh	ckan 2.0	closed	wontfix	Add new groups and organizations functionality into legacy templates	From Toby: add the functionality to legacy templates	1350297883000000	1350560252000000
2981	refactor		seanh	ckan 2.0	new		Remove config from jinja templates	"From Toby: remove the config from jinja2 templates get all such stuff via g. (before 2.0)
"	1350297960000000	1350297960000000
2982	refactor		seanh	ckan 2.0	new		Move functionality from controllers into template helpers	"From Toby: strip more functionality from controllers and add via template helpers
eg activity streams
"	1350298002000000	1350298002000000
2983	refactor		seanh	ckan 2.0	new		Refactor lib/base.py to remove circular import issues	"From Toby: c) refactor lib/base.py to remove the circular import issues (render
functions to lib/render.py - would fix much of that)

"	1350298044000000	1350298044000000
2984	refactor		seanh	ckan 2.0	new		Auto-populate context with user, model, session, etc.	"From Toby: make the context auto-populated with things like user/model/session if
not user supplied - especially for extensions (pre 2.0) maybe via a helper
or else a decorator on the action - helper seems nicer maybe done via
get_action()"	1350298101000000	1350298101000000
2985	refactor		seanh	ckan 2.0	new		Make lib/helpers.py more template-specific	"From Toby: make lib/helpers more template specific and less generally shared with
none template uses (some functions are shared but many shouldn't be) -
again help avoid circular imports
"	1350298188000000	1350298188000000
2986	refactor		seanh	ckan 2.0	new		Make lib/base.py more like it was originally intended	"From Toby: make lib/base.py be more like it was originally intended eg defines
stuff that is shared eg _ but maybe try to include stuff like
render/get_action if the circular import stuff can be solved - which should
be doable if hard and likely needs an extra file or two for the externals _
etc
"	1350298245000000	1350298245000000
2987	refactor		seanh	ckan 2.0	new		Remove all direct calls to logic action and auth functions	From Toby	1350298287000000	1350298287000000
2988	enhancement		seanh	ckan 2.0	new		UI functional tests for CKAN 2.0	Afaik the new CKAN 2.0 frontend has no functional tests	1350298935000000	1350298935000000
2989	enhancement		seanh	ckan 2.0	new		"""Add dataset to organization"" should auto-select the organization"	"'Add dataset to organization' button, when you get through to the third stage of the new dataset form the organization you came from is not selected.

I don't see any option to choose the group when adding or updating a dataset, but if I add a dataset via the ""Add dataset to group"" button on a group's page, then the dataset seems to get added to that group. (And I can also add/remove existing datasets by editing the group.) I wonder if organizations should work the same way, instead of having an Organization drop-down when creating or updating a dataset. The add dataset page needs to somehow indicate that you're adding a dataset to a certain group or organization though, doesn't currently."	1350299052000000	1350299052000000
2990	enhancement		seanh	ckan 2.0	new		Fix descriptions of groups and organizations on /groups and /organizations pages		1350299091000000	1350299091000000
2991	enhancement		seanh	ckan 2.0	new		Add blocks to header.html template	Currently if you want to customise the header.html template you have to copy the whole thing, it isn't broken up into blocks like other templates are.	1350385820000000	1350385820000000
2992	enhancement		johnmartin		new		Delete resource should send me back to edit dataset page		1350407956000000	1350407956000000
2993	defect		seanh	ckan 2.0	new		"""logged_in"" and ""visitor"" show in user list at /users"		1350466922000000	1350484826000000
2994	enhancement		seanh	ckan 2.0	new		Add dataset and user popovers throughout the site	"If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button.

These popovers should appear wherever datasets or users are listed, throughout the site."	1350483964000000	1350483964000000
2995	enhancement		seanh	ckan 2.0	new		Popovers for resources, groups, organizations, tags...	"If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button.

It would be nice to add similar popovers for other types of object in CKAN, e.g. resources, groups, organizations, tags..."	1350484041000000	1350484041000000
2996	enhancement		seanh	ckan 2.0	new		Fix behaviour of View Profile button in user popover when already on user's profile	"The user popovers that appear when you hover the mouse over a user in an activity stream include a View Profile button that takes you to that user's profile page. If you are already on that user's profile page then it simply reloads the same page, but it sends you to the Datasets tab if you were on another tab.

Possible solutions:

Don't show button if already on user's page?

Button just makes popover disappear, if already on user's page?

Button does reload the page, but reloads the same tab (datasets, followers, activity stream) that you were on.

Marking this low priority"	1350484238000000	1350484238000000
2997	enhancement		seanh	ckan 2.0	new		Add activity streams to dataset pages	Add an activity stream tab to dataset pages, like we have on user profile pages. Dataset activity streams are already implemented in the backend.	1350484306000000	1350484306000000
2998	enhancement		seanh	ckan 2.0	new		Add activity streams to group pages	Before we do this we should implement smarter group activity streams (i.e. what activities should appear in a group's activity stream?) and following of groups.	1350484394000000	1350484394000000
2999	enhancement		seanh	ckan 2.0	new		Add activity streams to organization pages	Before we do this we need to merge the organizations branch into master, we need to implement smarter activity streams for organizations, and we need to implement following of organizations.	1350484457000000	1350484457000000
3000	enhancement		seanh	ckan 2.0	new		Add number of followers to dataset pages	User profile pages now show some interesting stats about the user: num. datasets, num. edits and num. followers. Would be nice to add some similar info to dataset pages. At least num. followers can be done (already implemented in backend).	1350484560000000	1350484560000000
3001	defect		seanh	ckan 2.0	new		Multilingual plugin crashes CKAN on add dataset when some languages are default	"Enable the multilingual plugins:

ckan.plugins = stats synchronous_search multilingual_dataset multilingual_group multilingual_tag

and set your default language to one not supported by the multilingual plugin, e.g.

ckan.locale_default = cs_CZ

now run CKAN and try to add a dataset:

File '/home/seanh/Projects/ckan171/ckan/ckanext/multilingual/plugin.py', line 141 in before_index
  text_field_items['text_' + default_lang].extend(all_terms)
KeyError: 'text_cs_CZ'

It doesn't matter what language you are viewing the site in in your browser, the default language setting in the ini file determines whether it crashed or not.

A number of supported languages are defined at the top of ckanext/multilingual/plugin.py. I think if the default language is not one of these it crashes.

I think this affects all versions of CKAN since the multilingual plugin was added so at least 1.7, 1.8 and 2.0"	1350579048000000	1350579048000000
3002	defect		amercader	ckan-v1.8	new		API v1/2 'legacy' search parameters must be escaped before they are put into a Solr query string	"Just to track @tauberer patch on Github.
Would be nice to write a test for it. Probably going to 1.8.1"	1350639142000000	1350639142000000
3003	enhancement		amercader	ckan-v1.8	new		New pg databases should be created with UTF8 encoding rather than system default	"Just to track @tauberer patch on Github.
Probably going to 1.8.1"	1350639199000000	1350639199000000
3004	defect		seanh	ckan 2.0	closed	fixed	ImportError: No module named polib	This is happening whenever people try to run paster commands. polib should only be needed for the check-po-files command don't import otherwise.	1350907790000000	1361988802000000
3006	enhancement		seanh	ckan-v1.8.1	new		Update translations from Transifex	There are some new translations already on the CKAN 1.8 resource on Transifex already (e.g. 100% Norwegian translation I think). We should at least pull these from Transifex for 1.8.1, and may as well do a call for translations too.	1351506144000000	1351506144000000
3007	enhancement		seanh	ckan 2.0	new		Adding a dataset creates multiple activities	Adding a new dataset creates multiple activity stream activities, e.g. seanh created the dataset foo, seanh add the resource bar to the dataset foo, and seanh updated the dataset foo are all created when I add a new dataset. I wonder if these can be collapsed into a single activity.	1351516507000000	1351517019000000
3008	enhancement		seanh	ckan 2.0	closed	fixed	Add tests for dashboard activity stream auth	We need to test that only logged in users can see their own dashboard activity streams, and they can only see their own dashboard activity streams not those of other users.	1351600860000000	1355141103000000
3010	enhancement		seanh	ckan 2.0	new		Pin images don't appear in data explorer	"When minified files are in use (set debug = false in ini file) the pin images for points don't show up in the data explorer map view when viewing a resource with geolocation data.

set debug = true will fix the problem by using unminified files instead but also turns on a bunch of debug stuff you wouldn't want on a production site.

middleware.py, around line 73, is where the decision to turn minified files on or off based on the debug setting is made. This can be hacked on a production site to run with unminified files but otherwise run on production mode thereby getting around the issue with the pins. But that doesn't fix the underlying bug."	1351705506000000	1351705506000000
3014	defect		seanh	ckan 2.0	new		Crash when deleting a non-empty vocabulary	"From Knud Möller:

when I try to delete a non-empty tag vocabulary via the API (through HTTP), I get an internal server error. Checking
the logs, this turns out to be a consistency error raised by  sqlalchemy:

Error - <class 'sqlalchemy.exc.IntegrityError'>: (IntegrityError) update or delete on table ""vocabulary"" violates
foreign key constraint ""tag_vocabulary_id_fkey"" on table ""tag""
DETAIL:  Key (id)=(21421955-7560-467c-af30-9f790b73e6ae) is still referenced from table ""tag"".
 'DELETE FROM vocabulary WHERE vocabulary.id = %(id)s' {'id': u'21421955-7560-467c-af30-9f790b73e6ae'}
URL: http://33.33.33.10:5000/api/action/vocabulary_delete

The error makes sense, but I'm wondering if it would be useful to extend the API to also allow the deletion of        
non-empty vocabularies, possibly via a parameter (not sure what best practice in API design is). At the very least, it
would be cool if the error message coming back in the response had more information in it."	1352803808000000	1352803808000000
3019	defect		seanh	ckan 2.0	new		Cannot delete dataset extras	Deleting extras in the web interface is broken	1352918678000000	1352918678000000
3020	enhancement		seanh		new		Update CKAN coding standards		1353096711000000	1353096711000000
3024	enhancement		seanh		closed	fixed	Add activity stream tab to dataset read page		1353949986000000	1355141087000000
3030	enhancement		toby		new		clean up helper functions		1355834038000000	1355834038000000