Because of a clash between two development branches there is some duplication of code to do with code facets (note: at the time of writing the code duplication exists only on the feature-1821-multilingual-extension branch, but this will be merged into master at some point):

The package_search() function is adding the search facets to the search results twice with two different data structures, with keys "facets" and "new_facets". It should be reduced to just the new facets (with the key changed to "facets").

Also the group and package controllers are adding both facets and new_facets to the context, should be new_facets only (but renamed to facets).

The facet_items() function in helpers.py should be removed, it uses the old facets structure and shouldn't be needed anymore with the new facets structure.

In facets.html, facet_sidebar() should be removed as it uses the old facets structure and facet_div() implements the same functionality but uses the new facets.

In facets.html, facet_list_items() will have to be updated to not use the facet_items() helper and to use the new facets structure instead.

Anywhere that "new_facets" appears it will have to be changed to "facets" (e.g. in the ckanext/multilingual/plugin.py.

This is the merge commit that introduced the duplication: ​https://github.com/okfn/ckan/commit/1153aa876f54c22289e460aeececea22d1d4d51d

This is the earlier commit where the search facets were refactored: ​https://github.com/okfn/ckan/commit/3970e52008b75933fda1be1d488bed2578d98c9c

Because yes please

Before we do this we need to merge the organizations branch into master, we need to implement smarter activity streams for organizations, and we need to implement following of organizations.

Both in 1.5.1b:

​http://thedatahub.org/api/search/dataset?groups=lodcloud&title=

and 1.5.2a (current master):

​http://test.ckan.net/api/search/dataset?groups=lodcloud&title=

Although the error message in 1.5.2a is more verbose:

"Bad request - Bad search option: HTTP code=400, reason=org.apache.lucene.queryParser.ParseException?: Cannot parse 'groups:lodcloud title:': Encountered \"<EOF>\" at line 1, column 22. Was expecting one of: \"(\" ... \"*\" ... <QUOTED> ... <TERM> ... <PREFIXTERM> ... <WILDTERM> ... \"[\" ... \"{\" ... <NUMBER> ..."

Some parameter validation before sending it to Solr should do the trick

Broken out from ticket:216

Bug report regarding getting revisions:

Getting revisons by ID (on the latest ID) GET "​http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff"

Returns a whole load of stuff (it's also quite slow - about 10 seconds)

I'm expecting it to return just an empty array - am I doing something wrong here - if so could you clarify correct use of the API?

GET "​http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45" Returns the empty string - I'd expect an empty array ie []

GET "​http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45" Returns an internal server error 500 - I think it should probably be "bad Request" 400 (the date is invalid)

CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like:

http://<solr-server>/solr/schema-<version>

Different scenarios:

• Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine):
  • Update CKAN
  • Update SOLR schema (symlink to suitable version in CKAN source)
  • Rebuild search index

• Single SOLR instance used by multiple CKAN sites.
  • Update CKAN source
  • Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source)
  • Update solr_url on each of the CKAN sites with the suitable core
  • Rebuild search index

These are the SOLR and CKAN instances that need to be updated (Please add any missing ones):

• Same machine as the CKAN site:
  • test.ckan.net
  • iati.test.ckan.net
  • data.gov.uk/ catalogue.data.gov.uk (confirm)

• s046.okserver.org
  • iatiregistry.org

• s052.okserver.org
  • datacatalogs.org

• s004.okserver.org / eu4.okfn.org / solr.okfn.org
  • thedatahub.org (=www.ckan.net)
  • datagm.org.uk
  • publicdata.eu
  • hri.fi (dev.fvh.fi?)
  • it.ckan.net
  • ie.ckan.net
  • cz.ckan.net
  • register.data.overheid.nl (nl.ckan.net?)
  • no.ckan.net / datakilder.no
  • br.ckan.net
  • colorado.ckan.net
  • at.ckan.net

TODO: which SOLR server are using these instances?

• data.norge.no
• nederland.ckan.net
• lt.ckan.net
• pl.ckan.net
• datadotmontreal.ca/
• ca.ckan.net / datadotgc.com

CKAN gets its license list from a license service, which can be a local file, but is often the ​http://licenses.opendefinition.org/2.0/ckan_original server. This server is currently flakey, but I think we only request the list on start up. The problem is we query it much more often than required. It is queried for every request to api/rest/licenses, and we are returning lots of 500 errors when the license server is timing out.

CKAN is pulling a number of resources from external locations. This causes problems when connectivity is limited and you have to work locally. Maybe some of them cold be moved to CKAN source to avoid external requests.

Quick search:

./ckan/templates/layout_base.html:            <img src="http://assets.okfn.org/images/logo/okf_logo_white_and_green_tiny.png" id="footer-okf-logo" />
./ckan/templates/layout_base.html:            <a href="http://opendefinition.org/"><img alt="This Content and Data is Open" src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" style="border: none ; margin-bottom: -4px;"/></a>
./ckan/templates/package/resource_read.html:                <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/templates/package/read.html:          <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" /></a>
./ckan/templates/_util.html:                    <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/templates/_util.html:                  <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      this.$dialog.html('<h2>Loading results...</h2><img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:          self.setMessage('Uploading file ... <img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      self.setMessage('Checking upload permissions ... <img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" />');
Binary file ./ckan/lib/app_globals.pyc matches
./ckan/lib/app_globals.py:                                  'http://assets.okfn.org/p/ckan/img/ckan.ico')
./ckan/config/deployment.ini_tmpl:ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico

CKAN provides the /data page page of data.gov.uk

Breakdown of tasks:

• [x] Analysis / refinement of spec.
• [x] Log-in / register as publisher

(Waiting on publisher form)

• [x] Population of "browse by publisher"

list groups ordered by most datasets (Waiting on publisher integration)

• [ ] Browse by nation
• [ ] Featured datasets

Now a possible integration point with drupal

• [X] Tag cloud
• [X] Developers section

CKAN should be able to send email notifications to users.

Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users.

One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well.

Need to implement (at least some of) #1634 before this can be implemented, in order to have something to send notifications about.

Analysis here: ​http://ckan.okfnpad.org/27

CKAN uses a database. We can put the database on another machine and then clone the CKAN machine. Then there are two machines calling the database and converting SQL results into HTTP responses.

How many machines can you have before the database becomes the bottleneck? At that point, what QoS can be obtained with normal hardware?

Afterwards, can CKAN be made to scale further than this with a little development? Can its Postgres database be mirrored in realtime? Or can we change SQL writes (create/updates) to write to many databases?

Are there any other options for scaling the persistence mechanism?

CKAN's harvesting facility is now live on DGU but there are some major improvements that could be made to make it more robust and better fit the generic CKAN harvesting framework proposed in #987.

Some of the key issues:

• Error reports do not currently contain the ID or title of the document with the error.
• We only have "added" and "error" logging on jobs when we really need a report of "added", "updated", "not changed" and "errors" with the items in each referencing a real metadata document for which harvesting was attempted
• We need deletion and editing of sources, without deleting the harvested documents or packages
• We need a more robust harvesting mechanism than a cron job or we need to deal with the case of multiple cron jobs running at once.
• We need to know the last time a list of documents was scheduled for harvest and the last time each one was fetched.

CORS - ​http://www.w3.org/TR/cors/ - support.

This is what you do in Apache. Should do this in lib/base.py or similar.

    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Methods "POST, PUT, GET, OPTIONS"
    Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type"

    # Respond to all OPTIONS requests with 200 OK
    # This could be done in the webapp
    # This is need for pre-flighted requests (POSTs/PUTs)
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} OPTIONS
    RewriteRule ^(.*)$ $1 [R=200,L]

Can this be released as a standard CKAN feature?

Can we do this as a localisation?

Can we please add some text next to the format field when user adding a new resource along the lines of:

"This will be generated automatically, but you can edit if you wish"

It always seems to work better when the user does not fill it themselves...

Change login cookie from a default expiry of 50 years to 2 years. You can also uncheck a 'remember me' checkbox on the login form for the cookie to just last the session.

Background conversation on ckan-dev:

DR: I wonder if anyone objects to the expiry of the login cookie to be changed from 50 years to 2 years? 50 years might be appropriate for thedatahub.org, but for government sites it seems (to me) to be too lax.

Toby: is this the repoze.who cookie? If so that seems sensible to me.

Rufus: Definitely agree. I would also like to see introduction of a standard "remember me" checkbox (set to true by default). At the moment a login lasts forever (until you logout) automatically.

Change the visual style of CKAN to be more like these sites:

• GitHub?
• Quora
• Google Projects

This does not include major UX work.

Changes to publishers for customer

Changes:

• tidy up activity feed on home page or replace with search results (by modified date)
• move ckan menu to a left sidebar (see Sam's template) (not doing this now)
• update templates with changes from master
• use local image in language dropdown (ecportal.js)
• update dataset read template

To discuss:

• should we rename 'organization' to 'publisher'? Requires updating all templates and the url mapping.

Characters outside of ASCII range are not supported within data previews.

Steps to reproduce:

1. Visit ​http://ckan.net/package/kivele2010
2. Click on [preview] for any of the resources

Check to see that all the strings are getting translated, i.e. that all user-visible strings are being passed to gettext. Fix any strings that are not getting translated.

Child tickets

• #1198 Publisher hierarchy
• #1050 Authz lib improvement and refactor of ckan/lib/authztool.py
• #1004 Group creation instructions missing
• #1099 Strange interactions between two browsers while playing with authz groups
• #1115 can have two authzgroups with the same name
• #1133 command line rights manipulation doesn't work
• #1138 minor navigations behave inconsistently

Old ticket description:

1. Change name of AuthzGroup? to UserGroup? to reflect what it is for

2. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
   • Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
   • Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

3. Change UserGroups? so that they can have a hierarchical structure,

More info on Hierarchy change

e.g. UserGroup? NHS contains the User nhsysadmin, as well as the UserGroups? SURREY and BERKS, which themselves contain users.

One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not.

Each Group has permissions over various objects.

A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively.

Algorithm:

possible(user, action, package):

if user has permission for action on package

or any of have that permission

or any of his groups group-children (but not user-children), and so on recursively have the permission.

Child tickets:

• #1101 Integrate stats and googlanalytics into site nav

We have a spreadsheet from UKLP of statistics we'd like to generate

Child tickets:

• #1194 "Welcome back" message for newly registered user
• #1202 Links to datapkg utility don't lead to info about it
• #925 Change the search box icon to remove the down arrow
• #923 Search box doesn't work in leaderboard page in stats extension
• #1034 Flash message cached
• #737 Markdown syntax summary page
• #811 Extra field editing form layout breaks when there are long field names

Chrome does not resize iframe after a full refresh/ on first load

Clicking on 'save & add another' button here ​http://s031.okserver.org:2375/dataset/new_resource/yo takes you to step 3 instead of leaving you on step 2 when you haven't added any data.

Instead it should show an alert/ message that no data has been added & that you need to link to a file or upload something

Also, can upload be option 2 instead of 3?

Collect together requirements and top-level design for user/package 'groups': existing tickets, Rufus spec, Sean spec, meeting notes (dread) email, based on existing user authz stuff.

​http://knowledgeforge.net/ckan/trac/wiki/AccessControl

Do we add these into user-role table somehow or new table? To present this to team

Compare these two requests to create a package:

curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'Content-Type: application/json' -H 'X-CKAN-API-KEY: tester'
curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'X-CKAN-API-KEY: tester'

The second one gets the payload through (ckan log):

Retrieving request params: UnicodeMultiDict([('{name:"test"}', u'')])

But the first one causes a ServerError? because the payload (name:"test") doesn't make it to request.POST or request.params:

Retrieving request params: UnicodeMultiDict([])

The only difference is the "ContentType?: application/json" header, which seems a reasonable thing to include. Javascript lib backbone.js (for example) inserts this automatically.

So why does this header cause the payload to not get through to the request object?

Configuration option 'ckan.site_description' isn't documented

Confusion between entering values in the title and name/url fields in datacatalogs.org.

The help text "Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_" applies to the name/url field, yet in Lucy's browser the text that is actually hovering over the title field in my browser.

Consolidate search API interface (and backend) on solr (solrpy) type interface.

• Support for standard query structure
• Support for facet options

Do not need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

1. Disable (happens automatically)
2. Implement - suggest using group by etc

Extras

• Front-page tag cloud: change this to use facets
  • Accept this means that if facets not functional in backend we have no tag cloud

Control using cookies.

Controlling Groups through a REST interface.

Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend.

Core changes to theme in order to make it easier to re-theme:

• page_heading
• side bar menu must be switchable (left to right) (through config or css)
• Add optional_footer to complement optional_head
  • And adopt rule for template writers that all extra js must go in optional footer
  • That way we could move optional js to the bottom of the page to improve page load times.
• Re-add support for body-class ... (seems to be removed by #1108)
• more divs.
• better labelling for css (and javascript)

Cost - 2 days

Search interface has new options to filter and sort the results by the time-related field of the package. Search options are included in both Web UI and Search API.

The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package. Need to decide for a time-related field value that is date range, what date is used for the search.

Example search parameters:

Related to ticket:192

Cost 7 days

When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API.

The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user.

Example at bottom of package page:

Leave a comment:

Comments:

Implementation details:

Comments table is with columns:

Cost: 1h

Cost: 2h

Create /stats/ page displaying main statistics, e.g:

• Most highly rated packages
• Most edited packages
• Largest groups
• Top tags (by packages)
• Package addition rate
• Users with most packages

Related to ticket:181 - Stats side-bar

Details

• Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display)
• For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html

Create a new dataset, don't add any resources to the dataset yet, view the dataset in ckan, if you have firebug enabled you'll see the error "resource is undefined" at line 699 of application.js.

Create a thin web service that will allow users to upload files to OFS on S3 without knowing S3 credentials. This could also provide static urls for all stored resources.

• relates to #307

Create big tag cloud with all CKAN tags - perhaps weighting with size and colour...

Create in the model basic operation of Access Control.

roles table

name | context | action
-----------------------
admin| package | edit
admin| package | edit-permissions
admin| package | read
editor| package | update
editor| package | read
reader| package | read
This data is set-up on db init and will have no interface.

user-roles table:

username | context_type | objectid | role
rgrp     | system  | n/a | admin
visitor  | package | * | reader
bob      | package | geonames | admin
visitor  | package | geonames | editor
visitor  | package | geonames | reader
john     | group | ukgov | admin
dread    | group | ukgov | editor
visitor  | group | ukgov | reader
This data will be added when someone is given permissions for the system, a package or a group.

Pseudo code:

class Package

def is_allowed(name, action):

is_allowed(name, action, context=self)

class Group

def is_allowed(name, action):

is_allowed(name, action, context=self)

def is_allowed(name, action, context=None): name: string - a username or IP for 'visitor'

action: string - 'read', 'edit', 'delete', 'edit-permissions' context: object - a Group or a Package or None (which means system)

Create the form wizard for the package-new form.

Each section of the form will be a separate page as this was decided to be simpler than the alternative of making AJAX calls for validation at each stage. (*)

• separate pages for each section of the form
• validation carried out at each stage against the whole schema. Each section/page declares a list of schema keys that need to validate for that section to validate, and thus move onto the next section.
• no draft saving to be performed in this ticket.

(*) - although the javascript alternative will probably provide better UX (each step would require a page-load in the wizard approach), it was decided that:

• with the javascript approach it would be harder to test the workflow.
• with the javascript approach there would be additional work displaying validation correctly. Although not that complicated, it was felt to add another point of failure.
• the multi-page wizard is quicker and easier to implement, and if it provided poor UX, then the javascript approach would be used instead.
• the multi-page wizard wouldn't preclude a javascript-tabbing create-form for other cases (where the wizard workflow wasn't such a good match, eg on the hedatahub.org)
• the multi-page wizard wouldn't preclude a javascript-tabbing edit-form.

Create/register a user with a unicode character in the password. It creates the user, but the redirect to the user page doesn't work and results in 500 error.

Creation of duplicate packages named with trailing underscore

ww 22/9/10:

Need to ... merge some packages (the "find packages by attributes" is not 100% reliable because of inconsistent data already in there).

The "find by tags" algorithm uses (title, department). Unfortunately some of the data already there has no department. This causes it to make a new package (with trailing underscore).

dread 24/9/10:

I've done a test for the missing department issue and I can't reproduce the problem. This may be due to my general tidy up and getting tests passing using the latest ckanclient. Anyway, can you check over my test to see if this is what you mean?: dgu/ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadMissingDept

ww 20/10/10:

you should uncover it if you run against (a copy of) the dgu database.

Current form stuff is not very good (uses formencode). Switch to formalchemy would improve this, especially on validation.

As an extra we could utilize the formalchemy pylons admin interface (pretty much for free).

Cost: 12h

Details

1. Replace htmlfill and formencode extract in controllers/templates with formalchemy
   1. Crude and simple
   2. Suppress unwanted fields (revision, state, all revisions) and sort out ordering to be similar to before.
   3. Sort out tag field with a special renderer
2. Validation - testing definitely required.
3. Fix up description and pretty css etc