Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ...

Spec - see attached, which is a copy of the Pirate Pad from Lauren: ​http://piratepad.net/2C2iwiLDhd

Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources.

A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package.

It provides a way for people to make suggestions about a package without needing access to main package.

Need to be able to build a distribution. Need:

• new 'build' command
• specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution.

If a package name changes, a simple CKAN client may not be aware of this (not monitoring the push notifications, revisions or feed), so it is preferable to refer to the package by its (invariant) ID.

It is still useful to refer to a package by its name though, so both should be valid arguments in the REST interface.

As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

• ckan.site_title config variable
• set this on g in app_globals.py e.g.
  • from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
• use in head title and in main site title/logo section (use it as alt on logo image)
• Also all other pages (e.g. index, about) which talk about CKAN
  • Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:

• Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
• Probably yes, but not part of this ticket.

Use cases

• Register for package changes
• Register for all revisions
• Notified of a package change
• Notified of a revision
• Deregistration
• Configuration of port in pylons config

Design

• Default port: 5672 (standard for AMQP)
• Exchange name: 'ckan'
• Exchange type: topic exchange (most flexible)
• Routing keys: (see below)

Routing detail

Routing key format: "OBJ_TYPE" (NB tags should be identified by their name, not ID)

Example routing keys

• 'package' - Package edited/created
• 'resource' - Resource edited/created
• 'revision' - Any change
• 'db.clean'
• 'db.rebuild'

Example queue bindings that clients may use:

• * - no filtering - client receives all notifications
• package - only changes to packages
• revision - all revisions
• db - all database operations

Versioning

Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps?

Documentation

• How to use
• simple example of an external client?

Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead.

The indexing shall run in a separate shell/process, managed by supervisord.

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

• Python: ckanclient
• Perl: luke closs wrote something
• PHP: drupal library?
• ...?

Use case: new package

1. An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package.

2. User: clicks 'new package'.

3. CKAN presents the package/new form to the user.

4. (After a couple of previews) User: clicks 'commit'.

5. Notification message goes from CKAN to the front-end detailing the new package.

6. The user is redirected back to the front-end web page displaying the list of packages, which contains the new one.

The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6).

In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages?

Use case: CKAN imports packages

1. CKAN administrator runs a script that adds 100 new packages into CKAN.

2. CKAN sends notification message to front-end to report the new packages/revisions.

3. Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages.

4. The front-end queries CKAN for each new package one-by-one.

5. A new user request to the front-end will include the info about the new packages.

The package addition could be achieved in 1 revision, 100 revisions or some compromise:

• If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'.

• If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN.

This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages.

target = _blank

Cost: 30m

(in the same way that we can refer to packages in the API by ID as well as name.)

As a

client of CKAN

I want to

link to a package page in the UI, referring to it by ID

Need to research what can easily be done to improve CKAN packages in the search rankings.

Comments from Glen Barnes:

We've been pretty successful at SEO without even really trying (see ​http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (​http://www.ckan.net/package/ascoe -> ​http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly ("Atmospheric Chemistry Studies in the Oceanic Environment" on the example above). Some basic SEO 101 on page optimisations.

As an

external front-end

I want to

be notified (by XML-RPC) about package creations and updates.

Implementation

1. A message queue worker waits for package update notifications
2. On reception, it constructs XML detailing the changes and PUSHes it to a configured URI.

Rather than turning the package fields into XML fields, the JSON dump of the list of package dictionaries will become a single XML parameter.

Config - in the CKAN config will be:

• URI to callback to
• API version to use (version 2 gives packages referred by ID not name)

(spun out of ticket:336)

Resource added to model API at:

api/rest/resource

Example model request

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028

returns:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Authorization

1. Have it generic (ie. not per resource) and use an action/role on system
2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
3. Introduce Resource in authorization system (requires migration)

Mixed model

Create / Edit:

if resource associated to package:
    check_permissions(package, update)
else:
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)

If you change a resource then you not only get a PackageNotification?, but also a ResourceNofication?.

Runs on the commandline and prints out notifications.

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

​http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.

The simplest way to improve performance on specific pages is to implement full page caching.

This, however, has challenges on pages with user-specific content. For example, on CKAN when a user is logged at the top of the page we have "You are logged in as: {username}". Similarly we change the visibility of some material depending on the status of users:

• Visibility of Edit or Authz link on packages and groups

This has been done in IATI ckan but should be ported over to CKAN main and put into the UI in a nice fashion.

IATI: ​http://bitbucket.org/okfn/iati/changeset/378431974c76 (related to #520)

Requires SOLR search backend (for postgres should fall back gracefully -- just do not show them).

I suggest we produce a quick Markdown cheat-sheet page, showing the key runes: e.g. create a title and quote some text. This page can link to the full Markdown docs for advanced users.

A user going to the Markdown docs that we link will have to read a couple of pages of the raison-d'etre of Markdown before he gets to the syntax. And it's not very easy to read, and being white on black it looks like proper geek stuff.

Admin configures entering read-only mode in one of two places:

• CKAN config file (e.g. ckan.ini)
• environment variable from Apache config

Once enabled, no writes can occur to the database (including user ratings and other usage stats).

All writes to the API are captured and you are returned an error explaining the reason.

Possible errors:

• 503 temporary maintenance
• 403 forbidden (if server if permanently read-only)

WP twentyten has an excellent core html structure. Furthermore, using that structure makes us compatible with all the WP twentyten compatible themes.

• Convert to wp twentyten html
• Switch css to be based off twentyten css (and perhaps rework somewhat)

Packages should provide create and last modified attributes which are included in package dict exposed via api.

These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc)

Allow choice of language on ckan instance.

• Configuration variable listing language choices (could try doing this automatically but recommend against this)
• Switcher option in UI (perhaps on user profile page)
• Default it automatically based on browser settings (?)

Think this is high value (and cost is low)

Cost: low (4h)

The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource.

This would help improve data quality and consistency.

Data importers should use API instead of object model directly:

• saves load on server
• much better error checking and authentication
• eating our own dogfood (tests the API well)

There are a number of extra properties we want to store in a resource:

• resource size (in bytes, no. triples etc.) - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/673
• resource hash is 'static' (default) or 'changeable' - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/674
• alternative link (e.g. TNA Web Archive) to use the link is broken - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/173

i.e. this is a much needed enhancement

Need to be able to read/write these values in web interface and API.

(At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.)

Implementation

I suggest these are arbitrary key/value pairs, just like package extras, reusing PackageExtra? code as much as possible. (Other much more radical alternative is to move model to RDF.)

Already have the basic capacity using *_extras paths for templates and public directory (see ​http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: ​http://packages.python.org/Flask-Themes/

Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:

• Purge revisions (or sets of revisions) and purge objects #1076
• Set roles for users #1075
• Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
  • WONTFIX
• Overview of activity
  • WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.

​https://bitbucket.org/okfn/ckanext-admin

Tickets

• ticket:1031 - user autocomplete api in ckan

Notes

Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of 
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system

A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html

Instead we should use the site_url config option where necessary.

Cost: 1h

This is a meta ticket for a bunch of UI changes:

• #839 Package page redesign
• #846 Merge css files
• #850 Show welcome message
• #863 Prompt users to enter missing info
• #864 Search results changes
• #866 Add download formats to search results

• Use config variable to switch ETags caching on or off. Consider joining in with 'cache_enabled'.
• Default setting for (all) caching should be off.
• Needs documenting in configuration.rst

An extension that provides a todo list feature on CKAN so that people can register and find things to do.

Extension name: ckanext-todo

User Story

Package page

As a user I come to a package:

• Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
• At the bottom is a section of the package display titled "ToDo?" where I see a list of all toDos for the package most recent at the top
  • If I am logged in
    • See a form for "Add to do" at the top of the todo section and can add one straight away
    • I see a "now resolved" button next to each which goes green when you hover.

When clicked the todo fades away.

• Not logged in: I see a button that says "login to add todo"

Todo list page

When a user comes to todo overview at /todo

At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category.

Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo.

Implementation

The Todo form

• One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed)
• Add a description
• Submit, the todo gets added via AJAX to the list at the top as the most recent todo

Model:

todo table

  id (autoincrement integer)
  package_id
  todo_category_id (required)
  description (required)
  created=NOW()
  resolved=null (unresolved) or a datetime (datetime of resolution)
  creator=user
  resolver=user

todo_category table
  id
  name

Prepopulate with: broken-resource-link, no-author, bad-format, add-description

API at /api/2/todo

• GET / POST / PUT ...

/api/2/todo?package=package_id_or_name&category=...&resolved=0/1

• support limit (?)

/api/2/todo/category -> return list of todo categories

• No GET / PUT / POST (these are auto-created by creation of todo)

Optional Extras (Will not be done atm)

• Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ...

Allow a form to specify fields that are required. Currently name and resource url are required.

• Show required fields with asterisks by the input box.
• Resource fields to be included in this.

This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010).

3 key aspects of this:

• Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN)
  • Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable
  • Asynchroously: via worker with queue
    • #324 Search indexing using notifications
    • #669: Refactor Solr (indexer) to become a generic worker
    • #874: Extract Solr search backend into an extension
• In core support using SOLR backend as well as postgres for queries
  • #317: Make search pluggable (?)
• Integrate into WUI (if changes needed)
  • #672 - Facets in CKAN search results

Extras:

• Solr index testing tool #431

Control using cookies.

Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys.

This would lead to an extension to OFS.

This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN).

• master ticket #852

as per http://ckan.org/wiki/UIRedesignSearch

To be more pythonic, raise exceptions when ckanclient gets status which isn't 200.

Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)

Move the solr SearchBackend? to a seperate code base, remove any mentions of the CKAN model.

Among other things this will allow the tests to run much quicker.