The Group controller uses the genshi module where it may be better for it to use Markup() in the template.

See ​https://github.com/okfn/ckan/pull/117/files#r1524863

Having two methods form_to_db_schema(self) and form_to_db_schema_options(self, options) seems unnecessary, why not just have form_to_db_schema(self, options=None)?

Fixing this might break existing extensions although I don't think any are using form_to_db_schema_options() so it should be okay.

The same thing goes for db_to_form_schema_options().

Also why are we passing a dict 'options' that always contains the same three keys 'api', 'context' and 'type'? Why not three params api=None, contenxt=None, type=None?

Organization administrators can delete themselves from the user management pages. We should disable this on the front-end as well as the back-end.

In setup.py, do we still need:

('templates/.txt', 'genshi', {

'template_class': 'genshi.template:TextTemplate'

}),

in message_extractors? Doesn't like there are any txt files in there anymore.

I'm unsure about what's going on here. As I understand it, when we run python setup.py extract_messages it's going to use the extract_ckan() function from ckan/lib/extract.py to process the HTML files. For the HTML files that are jinja2 templates, extract_ckan() will call jinja2_cleaner() which will call regularise_html() on the strings. So the strings are regularised when they are extracted from the source files.

But then we have the parse() method of CkanInternationalizationExtension? also calling regularise_html(). I don't get what's happening here. Why do the strings need to be regularised twice?

My guess is that CkanInternationalizationExtension? is used when the strings are extracted from the templates at runtime, and they need to be regularised at this time in order to match them against the regularised strings in the mo files to find the translations to output?

Maybe CkanInternationalizationExtension? needs a better docstring saying what it does?

Add a better docstring to for trans js command explaining what it does and why, and how to use it.

Also add this command to the CKAN Release Process as it's needed there

Finish the tests for the per-package-type custom template support in IPackageForm, IGroupForm and IOrganizationForm (i.e. the new_template(), read_template() etc. methods).

The new way that template inheritance works in jinja looks like it should allow this to be tested nicely without duplicating entire templates in the example extensions.

Before releasing CKAN 2.0 we need to run all the CKAN tests with a modified test-core.ini with the example_idatasetform, example_igroupform and example_iorganizationform plugins enabled. If any tests fail, fix the bugs. This needs to be done for each release so add it to the release process.

The application's Globals object is not very informative.

There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs.

I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents.

We need a nice way to be able to get to editing the different resources that are associated to a dataset.

With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is:

• the fanstatic_resources defined in ckan/lib/fanstatic_resources.py (bottom of module) should be prepared, and minified.

This ticket is to:

• provide a paster command which when run will generate the minified javascript as css files.
• stop the auto-minification of files when CKAN starts up.
• remove reference to the minified files in the .gitignore file.
• add instruction to the release process to run this command, and check-in the minified files to the repo.

## Background

Currently, minification works seamlessly without the need for any preparation when CKAN is started in a development setup. But on a production site, the webserver will (almost certainly) not have write-access to the directories that will contain the minified files. And so the minification will fail, and the site will end up serving the un-minified media, or even *old* minified media.

One way around this would be to allow webserver write access to the directory its serving out of. But this is not generally considered good practice.

Another method would be to distribute the minified files with CKAN. This ticket describes how to do this without causing a lot of noise in the commit history of the repo.

The auto-minifcation occurs when importing ckan/lib/fanstatic_resources.py.

## Changing the process slightly

The reason for moving away from auto-minifying files at start-up, to minifying files when running a paster command is:

• we should be distributing the minified files when we make a release (as we do with translation files). This makes it easier to install CKAN from source.

• if we distribute the minified files, then they need to be checked-in to the repo.

• if they are auto-minified, whenever small changes are made, then this will create a lot of noise in the repo. It will be clearer to have the minified-files generated manually as part of the release process. (Or whever we make a deployment internally).

There are now javascript translations. There's a paster command, trans js. This is a ticket to ensure that using trans js is documented in the normal ckan translation workflow (which is followed after a feature freeze, prior to release). And to see if there's any need for further incorporation into the current translation process.

Repro:

• Create a new Group, named "test-group".
• Add a dataset to it.
• Verify the dataset belongs to the group by visiting the Group's read-page
• Update the Group through the action api (group_update), using the uid in the "id" field, and a new name in the "name" field.
• Visit the group's read-page. The list of datasets will be empty.

This was an issue when editing a Group through the web interface, which was fixed in [1]. However it only fixes the issue in the group controller.

[1] ​https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335

If you are the only admin of an organization you can edit the organization's members and demote yourself, then the org has no admins and no one (except sysadmins maybe) can edit it.

Last admin should not be able to remove or demote herself.

Also applies to groups.

Edit an organizations members page and demote yourself from organization admin, after saving the members edit page reloads even though you are no longer an admin and should no longer be able to access this page.

The group members page (e.g. /group/members/roger) seems broken, IDs instead of user names are shown for the members, and clicking on a member 500s

when deleting members from groups and orgs

Makes sense to keep all auth stuff under logic/auth.

Also decide which functions from new_authz.pu should be marked private with _.

Also decide which functions from new_authz.py are really helpers for the auth functions in get.py, update.py and delete.py, and should maybe moved to another file, e.g. logic/auth/helpers.py

From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions

From Toby: @johnmartin the user autocomplete on member add needs fixing it gets data but does not understand what to do with it

From Toby: The requesting membership user stories and implementation remain needed

From Toby: remove the config from jinja2 templates get all such stuff via g. (before 2.0)

From Toby: c) refactor lib/base.py to remove the circular import issues (render functions to lib/render.py - would fix much of that)

From Toby: make lib/helpers more template specific and less generally shared with none template uses (some functions are shared but many shouldn't be) - again help avoid circular imports

From Toby

'Add dataset to organization' button, when you get through to the third stage of the new dataset form the organization you came from is not selected.

I don't see any option to choose the group when adding or updating a dataset, but if I add a dataset via the "Add dataset to group" button on a group's page, then the dataset seems to get added to that group. (And I can also add/remove existing datasets by editing the group.) I wonder if organizations should work the same way, instead of having an Organization drop-down when creating or updating a dataset. The add dataset page needs to somehow indicate that you're adding a dataset to a certain group or organization though, doesn't currently.

Currently if you want to customise the header.html template you have to copy the whole thing, it isn't broken up into blocks like other templates are.

If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button.

These popovers should appear wherever datasets or users are listed, throughout the site.

The user popovers that appear when you hover the mouse over a user in an activity stream include a View Profile button that takes you to that user's profile page. If you are already on that user's profile page then it simply reloads the same page, but it sends you to the Datasets tab if you were on another tab.

Possible solutions:

Don't show button if already on user's page?

Button just makes popover disappear, if already on user's page?

Button does reload the page, but reloads the same tab (datasets, followers, activity stream) that you were on.

Marking this low priority

Before we do this we should implement smarter group activity streams (i.e. what activities should appear in a group's activity stream?) and following of groups.

User profile pages now show some interesting stats about the user: num. datasets, num. edits and num. followers. Would be nice to add some similar info to dataset pages. At least num. followers can be done (already implemented in backend).

This is happening whenever people try to run paster commands. polib should only be needed for the check-po-files command don't import otherwise.

Adding a new dataset creates multiple activity stream activities, e.g. seanh created the dataset foo, seanh add the resource bar to the dataset foo, and seanh updated the dataset foo are all created when I add a new dataset. I wonder if these can be collapsed into a single activity.

We want a "bubble" with a number in it to appear in the top-right of the site next to your user name when you're logged in, telling you how many new activity stream items you have in your dashboard activity stream.

There's a github issue for this here:

​https://github.com/okfn/ckanext-pdeu/issues/11

Fixes and updates to Recline / Datapreview for CKAN 2.0.

Essentially the default bootstrap dropdowns don't work properly without JavaScript? which is a little bit of an issue with JS not working.

Deleting extras in the web interface is broken

​https://github.com/okfn/ckanext-pdeu/issues/13

Have a news section (suggest as a sidebar item).

News section will link to latest 3/4 blog posts on CKAN from blog.okfn.org.

Details:

• Suggest pulling via rss or similar.
• Will want to cache this ...

Cost: 4h?

Overview

Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management.

Examples

1. There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial.

2. In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.)

3. OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it.

4. IPCC collection of data linked / mirrored. Not sure if there are useful relationships here?

5. Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM.

See more examples discussed here: ​http://trac.ckan.org/ticket/253

Implementation

This is split into four tickets:

• Model: ticket:254
• Read in WUI: ticket:255
• Edit in WUI: ticket:256
• API: ticket:257

No need for write access to be provided API for the moment.

This ticket also encompasses ticket:169 (Package derivations) and ticket:176 (Package dependencies).

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN

## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png

## Site tagline / description (used on front page)
ckan.site_description = 

## Used in creating some absolute urls (such as rss feeds, css files) and 
## dump filenames
ckan.site_url =

## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico


## An 'id' for the site (using, for example, when creating entries in a common search index) 
## If not specified derived from the site_url
# ckan.site_id = ckan.net

## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net

## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end = 

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

• [namespace]: ? only if KeyValues? (for settings this would then always be settings)
• key
• label
• value (json)
• type (e.g. date and to specify in advance what type should be)
• description
• tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

• /ckan-admin/settings
• Show label, plus description plus text field

Depends

• Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)

Need to research what can easily be done to improve CKAN packages in the search rankings.

Comments from Glen Barnes:

We've been pretty successful at SEO without even really trying (see ​http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (​http://www.ckan.net/package/ascoe -> ​http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly ("Atmospheric Chemistry Studies in the Oceanic Environment" on the example above). Some basic SEO 101 on page optimisations.

Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they?

They appear on the 'Recently changed' list on the home page with an empty 'Packages' column.

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

​http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.

When people link to a package, a track-back link is auto-created. (Similar system as for blogs).

As suggested by Tim Davies:

Allowing some form of ‘track back’ against datasets When a non-technical user comes to look at a dataset it would be really useful for them to be able to see if anyone has created an interface interpretation of it already.

I found quite a few cases in research of end-users struggling to make sense of a dataset when good interfaces to that data had already been built and blogged about, but without there being any link from the dataset listing to those data uses. Accepting track backs could also make it easier for technical users to find blog posts / shared code etc. relating to a given dataset.

Hi Will, this is important again because some CSW servers we use have over 300 documents in. Could you take a look at modifying the filter please?

All writes to the API are captured and you are returned an error explaining the reason.

Possible errors:

• 503 temporary maintenance
• 403 forbidden (if server if permanently read-only)