Because of a clash between two development branches there is some duplication of code to do with code facets (note: at the time of writing the code duplication exists only on the feature-1821-multilingual-extension branch, but this will be merged into master at some point):

The package_search() function is adding the search facets to the search results twice with two different data structures, with keys "facets" and "new_facets". It should be reduced to just the new facets (with the key changed to "facets").

Also the group and package controllers are adding both facets and new_facets to the context, should be new_facets only (but renamed to facets).

The facet_items() function in helpers.py should be removed, it uses the old facets structure and shouldn't be needed anymore with the new facets structure.

In facets.html, facet_sidebar() should be removed as it uses the old facets structure and facet_div() implements the same functionality but uses the new facets.

In facets.html, facet_list_items() will have to be updated to not use the facet_items() helper and to use the new facets structure instead.

Anywhere that "new_facets" appears it will have to be changed to "facets" (e.g. in the ckanext/multilingual/plugin.py.

This is the merge commit that introduced the duplication: ​https://github.com/okfn/ckan/commit/1153aa876f54c22289e460aeececea22d1d4d51d

This is the earlier commit where the search facets were refactored: ​https://github.com/okfn/ckan/commit/3970e52008b75933fda1be1d488bed2578d98c9c

IGroupForm should allow overriding of non-edit templates so that it can specify the index +/or search +/or view.

Documentation is key to the success of projects!

Move material into main docs from wiki

• ​http://wiki.ckan.org/Writing_asynchronous_tasks
• ​http://wiki.ckan.org/Domain_Model
• ​http://wiki.ckan.org/Deployment

Create a User Guide

Basic step-by-step on getting started with CKAN (See start on set of slides here e.g. ​http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/)

• Publishing data
• Authorization and workflows
  • Publisher Profile (Workflow)

Break into sections

Suggest something like:

# General
* Intro
* Conceptual Overview
* What CKAN helps you do (http://ckan.org/)
* FAQ ? (or point to the wiki)

# For Administrators

* Installation and Setup
  * Configuration options
* Customization - Theming etc
* Authorization and Workflows
* Storage

# For Users (Publishers, Data Wranglers, etc etc)
* Walkthrough of publishing a dataset
* Storage
* API (see Developers section)

# CKAN Developers

* Domain Model
* API
* Extensions
* i18n

CKAN Developers
* Buildbot

• Get rid of jquery UI
  • Requires switching to bootstrap lookahead plus a small bit of coding for ajax part (see ​https://gist.github.com/2007241)
• Remove modernizr (what do we use it for?)
• Convert jquery.tmpl to jquery mustache (?)
• Remove ckanjs moving core parts of it directly into our code.

Compress everything for releases and add minified version to build (with switch to enable use of this?)

Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here: ​https://gist.github.com/2007985

need to sort cookies etc

should work with nginx and varnish

It is necessary to add a helper method to Group for adding members to that group so that

• nobody needs to work with the Member model
• It handles existing membership and existing membership with a different capacity

All strings from the ckanext-ecportal extension (and from CKAN core) need to be exported to a pot file to send to tenforce.

This is a reworking of the existing apps extension.

Initial proposal at ​http://wiki.ckan.org/Proposals#Apps_in_CKAN and ​http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN

Naming

What do we call this extension?

• related
  • RP best IMO but perhaps too close to separate relationships concept?
• relatedstuff
• links
• references

Proposal

"Related Stuff": Apps as in an application (website/service/tool) that uses this dataset (as in Apps and Ideas extension)

• This case seems a general example of wanting to link some external thing to a dataset (e.g. this policy paper refs this dataset, this application uses this dataset, this visualization is built on this dataset cf ​http://lists.okfn.org/pipermail/ckan-discuss/2011-August/001607.html)

Implementation

New table named Related with following structure

|| id (int) || type || title || description (markdown) || image_url || owner_id || url || created (timestamp)

• type = Idea | App | API | Visualization | Post | Paper | News Article
  • Suggest we make this a ckan.ini config option (comma separated ...?)
    • Do we want the possibility of different templates for different types of Related objects?

• image: ?? Depends where we store images. Simplest option would be to change to image_url and leave it to users to have already uploaded an image somewhere. If not we need to support image uploading and storage. See #1692 (add image attribute to datasets and groups) for more discussion, once implemented the URL here can be an internal url.

• owner_id = user_id or creating user (see authorization below)

Related2Dataset (note that related_id, dataset_id tuple should be unique). This allows for m2m connections. If a given related item is only with one dataset this could be simplified. May contain status so dataset owner can turn this on/off.

|| id || dataset_id || related_id || status

status should be used to allow for a dataset owner (for dataset_id) to de-activate the relationship between the dataset and the related.

Url

• /dataset/{dataset-name}/related/{related-item-id}/{related-item-title-stringified}
  • If a reference item could exist in its own right (and perhaps refer to multiple datasets then it should get its own url at e.g. /related/{id}
• /dataset/{dataset-name}/related/add => Modal dialog on related tab so we can use API to create them.

/dataset/{id}/related <- list

• use image_url for small icon etc, title description (shortened?)
• Click through to full related item (optional)
• dataset owner is shown show / hide button ... (or on /dataset/{id}/related/{id} )
• related owner sees an edit button / icon (pops up modal)

/dataset/{id}/related/{id} (optional)

/dataset/{id} will have a Related tab (with bubble with count).

• Drop down with Add Related -> Pop-up modal and save via API

(Not used: /related/add with dataset prefilled ... )

/related/{id}/edit

Authorization

Addition of related item be considered orthogonal to datasets (and hence with separate authorization i.e. i can add the info that site X uses dataset Y without needing permission to edit dataset Y).

Thus any logged in user could add a Related item. We set the owner of the related item to creating user and going forward only that user or a sysadmin can update or delete.

NB: we could have a system where datasets owners have to approve related items before they show up next to their dataset. This would add substantial complexity so I propose we leave out of phase 1.

Tasks and estimates {7.5d}

[x] Model + Migration for Related table. {0.75d}

[x] Controller for Related (or relevant sections in Package controller). {0.75d}

[x] Routing setup. {0.25d}

[x] Schema for related. {0.5d}

[x] Logic layer actions. {1.0d}

[x] Auth (default + publisher). {0.5d}

[x] Templates + Dataset changes (new tab etc). {1.0d}

[x] JS Application for adding Related objects in a modal. {1.0d}

[x] Testing. {0.75d}

[x] Dataset owner disabling of Related (via M2M table). {0.5d}

• Updated to allow author of related to delete as well

[x] Documentation. {0.25d}

Just like we display download counts we should display view counts.

Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on ​Storate Metadata API.

Purging datasets (deleting them fully, not just changing the state to 'deleted') is important for users testing dataset creation over the API on a test CKAN instance.

Without this, they need to resort to more difficult methods such as:

• cleaning and reloading the database
• setting the test datasets to state 'deleted' and also appending a suffix '_00' and incrementing the number until there is no clash of names.

Requested for NHSIC.

Implementation

• This could slot into the Action API.
• Of course we would need to ensure the user's had been given the specific right to purge.
• I suggest we log the full details of the dataset being purged.

Allow OpenID to be disabled as a login option.

Currently (CKAN 1.6) there is a config option openid_enabled which stops the login option being displayed, but:

• In the User edit form you can still add your OpenID
• The repoze.who.openid middleware is still required - can it be avoided?

Recline v0.3 has introduced some breaking changes in API and usage. Upgrade to use it.

Currently everything is considered equally when doing a normal search. Use dismax when there is not a fielded search to get round this. This is part of the work for #1822 as the dismax option is needed for multilingual search.

For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page.

We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar.

Dismax query across languages. Titles and selected languages, weighted more highly.

All translated fields and vocabularies need to be added to search index in the correct fields.

Estimate 2.5d

Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset.

We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers? [1], [2].

Things to take into account:

• Integration with the form sections.
• Allow some degree of configuration (default extent, limit geometry types...)

[1] ​http://openlayers.org/dev/examples/draw-feature.html [2] ​http://openlayers.org/dev/examples/modify-feature.html

Estimate [2-3d]

If possible, it should work with both 1.5.1 and 1.6.

Needed for PDEU, also at some point for IATI

Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones.

• see how this currently works with DGU first

• Tag list not be added to CKAN core, what is the best way to do this in an extension?
• Does the list need to be updated via API/WUI?

All data is that is translatable should be translated when viewed.

This includes tag names will be ID codes. When viewing a dataset page, look up the current language of the interface (ckan_lang or something in the environs) and display the string for the tag in that language.

Related to the dataset view extension points for translation, recently added by kindly/=.

language chooser should look like the ec one

e.g. ​http://189.9.137.65/dados/ clicking on Deutsch link is http://189.9.137.65/dados/locale?locale=de&return_to=%2Fdados%2F&hash=1dc17c315c419df850da0dd3599eefa9da76fbeb and redirect goes to http://189.9.137.65/dados/dados/?__cache=97995106 so /dados/dados/ when it should be /dados.

Affects CKAN 1.6b only (not yet released).

See how possible it is to run CKAN on a Windows server.

Plan:

• Get CKAN code and dependencies installed on Windows
  • ideally as similar to standard package install as possible
  • virtualenv/pip should work
  • try Cygwin to help
  • SOLR and Postgres could potentially run on other servers if necessary.
• Try to run on Apache
• See if there are any issues

For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed.

Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality.

You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q.

For example:

http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93

This "N-dash" (Unicode character 2013) causes this exception:

Module ckan.controllers.api:460 in search
<<                      if ver in u'12':
                               # Otherwise, put all unrecognised ones into the q parameter
                               params = convert_legacy_parameters_to_solr(params)
                           query = query_for(model.Package)
                           results = query.run(params)
>>  params = convert_legacy_parameters_to_solr(params)
Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr
<<      for search_key in non_solr_params:
               value_obj = legacy_params[search_key]
               value = str(value_obj).replace('+', ' ')
               if search_key == 'all_fields':
                   if value:
>>  value = str(value_obj).replace('+', ' ')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128)

This problem affects CKAN 1.5 and 1.5.1 only.

lxml is used in CKAN core in two places:

• ckan/lib/helpers.py:239 Parsing Markdown and ensuring it is valid XML (i.e. tags close nicely so the HTML of the rest of the CKAN page is not disruptable).
• reading the SOLR config XML to determine what version it is.

Both of these could be done with xml.dom built into Python and would save us a sizeable and slightly problematic dependency.

lxml is needed by these extensions: ckanext-inspire, ckanext-harvest, ckanext-csw & ckanext-wordpresser, so we'd have to add this to their requirements.

Changes to publishers for customer that would otherwise clutter up the master branch.

The api currently uses the default validation schemas regardless of the form you specified on the front end. There should be a way to post through the api using the same validation rules.

From Pablo:

Editing the tags field is clumsy when there are too many tags. Could show existing effectively as tags (like delicious), then allow clicks to delete. New tags added via text box.

From Antonia Acuno, via Ira:

It appears that the current layout for the datahub does not work in IE7, which would mean a problem for us as most gov uses it and is one of those we have to support...any views?

Tested in several gov departments computers, so a common enough issue.

​http://imgur.com/ip4xa

From Rufus: We haven't officially supported IE7 so far but can put engineering effort into this if absolutely needed. I note that IE 7 is now 3y out of date and is likely a security risk. Obviously this is something to ask Antonio about.

If I have a relationship between two packages, and then I edit any property of one of them (Author email, for example), the relationship will automatically be lost.

The HTML templates for some activity stream activities link to objects (packages, resources, etc.) by their names, e.g. /dataset/foo, but if the object has since been renamed then the link will be broken.

Possible solutions are to use the object ID to find its current name at rendering time and link to that, or link to the object by ID rather than by name.

Linking by ID is both simpler to implement and better. Links by ID are ugly, but they are permanent and don't break when an object is renamed. We should be using links by ID everywhere on CKAN, so that when third-party sites copy-paste our URLs and link to us, their links don't break later.

Fixing this means fixing the dataset_link(), resource_link(), group_link() etc. functions in ckan/lib/helpers.py which will change the links everywhere in CKAN where those functions are used to link to objects (which should be everywhere where we link to an object).

If we want to have nice looking by-name URLs and what them to be permanent, then we need to change CKAN so that objects cannot be renamed.

• Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?]
• Icons for *types* of resources in resource list
• List of 'Owners' (administrators) at top right of sidebar (a la group page)
  • Much prefer owner to administrator ...

Add api to translation to the term_translation table.

• add to logic 2d
• test 2d

In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the get() classmethods in the model/ classes.

Doing it with module functions lets us type module.function() instead of module.Class.method().

In many cases we're importing classes directly with from module import Class and then doing Class.method(), but from module import foo is bad style: ​http://docs.python.org/dev/howto/doanddont.html

Add API calls for adding one or more tags to and removing one or more tags from a vocabulary, without affecting the other tags in that vocabulary and without having to pass the full list of the vocabulary's tags.