CKAN: Ticket Query

#320: site_title configuration variable which is used in template

rgrp — Thu, 20 May 2010 18:09:27 GMT

As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

ckan.site_title config variable
set this on g in app_globals.py e.g.
- from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
use in head title and in main site title/logo section (use it as alt on logo image)
Also all other pages (e.g. index, about) which talk about CKAN
- Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:

Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
Probably yes, but not part of this ticket.

#340: Web UI theme easier to configure

dread — Tue, 08 Jun 2010 15:08:49 GMT

An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in.

http://wiki.okfn.org/ckan/doc/theme

#226: UI Review - History

dread — Tue, 15 Dec 2009 12:35:15 GMT

Repository History revision

What is this page called? "Recent Changes", "Repository History", "/revision": standardise. Between the link in the nav and the page <title> particularly, but the route is also important. Perhaps /changes or something similar?
- Will change page title on /revision/ to Revision History. Will not change route for the time being.
~~needs copyediting.~~
Pagination has similar issues to elsewhere. Also, most obvious here, is the fact that we don't need to display a link to every possible page. Please can we limit it to, say, a dozen nearby pages and an ellipsis.
~~Without looking at dates, its not clear whether I'm seeing most recent or oldest changes. Change pagination to say "Older"/"More recent" rather than "Previous"/"Next".~~ (wontfix: now have text saying we are showing most recent changes)
Table layout is pretty ugly (yes, I'm aware this is my fault).
- wontfix - nothing better at the moment
Timestamps are horribly unreadable. At the absolute minimum get rid of the micros. Hover for more detail?
Can we link to an author page? Yes!
~~Atom feed should have a feed icon~~~~
Why are we adding another <link rel="alternate"> to this page? We already have one for recent changes on every page in the site and the one we're adding has a less descriptive title. Which is correct? Use that one as a feed for every page.

#227: UI Review - General checks

dread — Tue, 15 Dec 2009 12:35:52 GMT

There is huge inconsistency in the titles of pages across the site: to give one example "Edit Package:" vs "Package: mypkg [not linked]" vs "History - mypkg [linked to package page]"
- Normalize and use terminology: "Data Package" - also change in navbar (but nowhere else for the time being). So hvae Data Package - mypkg, Data Package - mypkg - Edit etc
pagination is currently lost in amongst the results list. needs to be *much* more obvious, and should appear top and bottom, or at the very least at the bottom!

#324: Search indexing using notifications

dread — Mon, 24 May 2010 17:51:23 GMT

Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead.

The indexing shall run in a separate shell/process, managed by supervisord.

#356: Search box in at top of page (UI)

rgrp — Tue, 22 Jun 2010 19:36:51 GMT

A small but useful ui improvement would be to have a search box at top right on every page.

As an example see the one here on trac or on github.com or bitbucket.org.

It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration).

#336: Resource Search API

donovanhide — Tue, 01 Jun 2010 17:02:45 GMT

As a

CKAN client such as ScraperWiki

I want to

search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL.

Proposed implementation

Add resource search API at:

/api/search/resource

AND resource added to model API at:

api/rest/resource

(see ticket:358)

Functional differences from the ScraperWiki suggested patch:

URL is not normalised

URLs are not grouped

All fields of the resource object are returned, not just the URL

Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend 'http://ckan.net/package/' to the package ID)

This is to make our API more general, simple and consistent. It means that the ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok?

Example search

POST

{"url": "scraperwiki.com/", "all_fields": 1}

to: /api/2/search/resource

returns JSON:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors",
   "format": "CSV",
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "",
   "position": 2
 }]

Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done ticket:341 then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe

Search Parameters

Key:  q
Description: Search all resource fields for the value
Key: url / description / format /
Description: Search particular field for the value
Key: all_fields
Value: 0 or 1 (0 is default)
Description: If 1 (true), the full record of the package resource
(and it's package reference) are returned, rather than just the
PackageResource ID.

May also choose to introduce 'offset' and 'limit' to page through a large number of results.

JSONP achieved through API-wide parameter - see ticket:342

Search is case insensitive.

Original request

Hi, have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of #388.

Could also add a search version. Not sure what the best url would be for that though.

Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere?

Donovan

#311: Reordering of package resources can lead to integri

rgrp — Sat, 08 May 2010 20:02:48 GMT

Created a new package resource line and then moved it up above existing one and hit save. Result: 500 error. In logs have:

[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] Error - <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "package_resource_revision_pkey", referer: http://ckan.net/package/edit/cofog
[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30]  'INSERT INTO package_resource_revision (id, package_id, url, format, description, hash, position, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(url)s, %(format)s, %(description)s, %(hash)s, %(position)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'hash': '', 'description': 'The Treasury record of COFOG functions. ', 'format': 'XLS', 'url': 'http://www.hm-treasury.gov.uk/d/cofog_definitions_coins250609.xls', 'package_id': '8482334d-fe2e-4285-9114-5243130f80c0', 'state': 'active', 'continuity_id': '8bf302db-8a80-47d3-b5dc-bc07512a3928', 'position': 3, 'revision_id': 'e4e2cb2d-4bd5-414e-b646-e484f174d9ab', 'id': '8bf302db-8a80-47d3-b5dc-bc07512a3928'}, referer: http://ckan.net/package/edit/cofog

#335: Post-package-edit redirect to configurable URL

dread — Tue, 01 Jun 2010 16:13:54 GMT

As a

third-party interface to a CKAN instance

I want to

link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return.

Design

The 'return URL' is passed as a parameter to CKAN.
CKAN substitutes the package name into the return URL.

Example

Front-end links to: http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/<NAME> (but with the parameter URL-encoded)

When finished editing and the user commits, CKAN redirects the user to: http://datadotgc.ca/dataset/pollution_data

#323: Notification message

dread — Mon, 24 May 2010 17:48:53 GMT

Which events to notify on

Listed by domain object, these are the notification message 'change types' that will be sent:

Package
PackageResource

Also it is clear that it could be useful to know when db-wide maintenance is carried out:

db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration)

Ignored domain objects

These parts of the domain model will not carry notifications as no use case has been identified for them:

Revision
Group
Tag
Rating
User - list of users is sensitive info
Relationships - complicated
Authz - complicated and sensitive info
License - change of a license's metadata is a question for the 'license service'

Message format

A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity.

For the 'db' notifications there shall be no payload.

#242: Miscellaneous tidying up (v0.11)

rgrp — Tue, 09 Feb 2010 11:31:43 GMT

Dumping ground for various minor and miscellaneous items (mainly refactorings):rgrp

controllers/tag.py: why does this not use lib/search.py (but controllers/packages.py search method does)?

done:

lib/search.py: inheritance would be nicer than switching on entity in search (i.e. have SearchPackage?, SearchTag? etc)

#328: Mention code libraries in API documentation

rgrp — Thu, 27 May 2010 19:21:46 GMT

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

Python: ckanclient
Perl: luke closs wrote something
PHP: drupal library?
...?

#317: Make search pluggable

rgrp — Wed, 19 May 2010 18:56:58 GMT

Make lib/search.py pluggable so that we can plug in different search systems (e.g. SOLR).

Suggest we define a base Search class from which specific search implementations inherit (e.g. SQLSearch, SOLRSearch etc). The specific one being used would then be set via a config variable.

#273: Investigate search index options and create tickets

rgrp — Fri, 19 Mar 2010 11:08:53 GMT

Work already here: http://knowledgeforge.net/ckan/trac/wiki/SearchEngine

Add knowledge there or to:

http://wiki.okfn.org/SoftwareTools/Search

#375: Integrate Forms API into Drupal

johnbywater — Tue, 27 Jul 2010 09:32:48 GMT

Requested by DGU.

#315: Improvements and fixes to csv dump

rgrp — Mon, 17 May 2010 12:54:47 GMT

Issues with quote in fields: http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html

Issues with package resource serialization into csv table.

<quote> In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent. </quote>

#263: Improve and test openid login

rgrp — Wed, 03 Mar 2010 08:30:39 GMT

At the moment the user has to figure out to login with providers -- this is not always obvious (e.g. google).

We could improve this with a small bit of javascript. As an example see: http://standalone.demo.civicrm.org/standalone/

Not sure exactly what library that is using (or if bespoke) so alternatives include (NB: we already use jquery):

In addition to implementing this we also need to test logging in with main providers: google, wordpress -- as we have had reports of it not working with e.g. wordpress (not sure if this testing can be automated really -- best hope would be selenium I think -- so OK if done by hand).

Cost: 3h (2h UI), (1h testing)

#376: Implement servicization of CKAN API

johnbywater — Tue, 27 Jul 2010 09:33:46 GMT

Requested by DGU.

#325: Event push notification

dread — Mon, 24 May 2010 17:51:52 GMT

As a

CKAN client program

I want to

be notified when changes to the CKAN metadata occur.

Examples of use

An external search engine needing to (re)index a package. (interest: Package)
A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333.
A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: PackageResource)
Do some processing on resource (e.g. extract sample data for display) (interest: PackageResource)

Context

The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling.

Design

Split-off into two tickets:

Notification message - ticket:323
- Which events to notify on
- Message format
Interface for Notifier Service - ticket:322

Testing

To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine.

Run CKAN search indexing using this system - ticket:324
Run SOLR indexing using this system - ticket:353
Web hooks for notifications - ticket:327

#337: Download links for resources should open in new window

rgrp — Tue, 01 Jun 2010 19:55:51 GMT

target = _blank

Cost: 30m

#374: Design servicization of CKAN API

johnbywater — Tue, 27 Jul 2010 09:31:58 GMT

#310: Commit message box looks wrong in edit page since edit style overhaul

rgrp — Sat, 08 May 2010 19:58:34 GMT

Suggest move this below the label and make full width of screen and only 3/4 rows high (more like a wiki site).

Also change label to: Edit summary (Briefly describe the changes you have made)
Remove: you can markdown formatting here.
Move author: if you have not signed in smaller and closer (like markdown instructions are nwo).
Change commit -> save
Remove "please save" just have the bullet points

#322: Client interface for Notification Service

dread — Mon, 24 May 2010 16:54:02 GMT

Use cases

Register for package changes
Register for all revisions
Notified of a package change
Notified of a revision
Deregistration
Configuration of port in pylons config

Design

Default port: 5672 (standard for AMQP)
Exchange name: 'ckan'
Exchange type: topic exchange (most flexible)
Routing keys: (see below)

Routing detail

Routing key format: "OBJ_TYPE" (NB tags should be identified by their name, not ID)

Example routing keys

'package' - Package edited/created
'resource' - Resource edited/created
'revision' - Any change
'db.clean'
'db.rebuild'

Example queue bindings that clients may use:

* - no filtering - client receives all notifications
package - only changes to packages
revision - all revisions
db - all database operations

Versioning

Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps?

Documentation

How to use
simple example of an external client?

#326: Centralise importation of json library

dread — Tue, 25 May 2010 10:43:43 GMT

Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers.

#329: Bad dates cause exception on Gov form

dread — Fri, 28 May 2010 15:30:17 GMT

Reproduction

Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message.

Affects all versions of CKAN.

Why it's happening

The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught.

#233: Allow simple site-specific customization/overriding of templates

rgrp — Sat, 09 Jan 2010 01:26:36 GMT

Options:

Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works)

Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist).

Can do this using:

<xi:include href="base.html"><xi:fallback /></xi:include>

#313: Allow packages to be specified by IDs in REST interface

dread — Tue, 11 May 2010 19:30:48 GMT

If a package name changes, a simple CKAN client may not be aware of this (not monitoring the push notifications, revisions or feed), so it is preferable to refer to the package by its (invariant) ID.

It is still useful to refer to a package by its name though, so both should be valid arguments in the REST interface.

#333: CKAN front end requirements for package notifications

dread — Mon, 31 May 2010 16:40:42 GMT

Use case: new package

An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package.

User: clicks 'new package'.

CKAN presents the package/new form to the user.

(After a couple of previews) User: clicks 'commit'.

Notification message goes from CKAN to the front-end detailing the new package.

The user is redirected back to the front-end web page displaying the list of packages, which contains the new one.

The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6).

In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages?

Use case: CKAN imports packages

CKAN administrator runs a script that adds 100 new packages into CKAN.

CKAN sends notification message to front-end to report the new packages/revisions.

Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages.

The front-end queries CKAN for each new package one-by-one.

A new user request to the front-end will include the info about the new packages.

The package addition could be achieved in 1 revision, 100 revisions or some compromise:

If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'.

If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN.

This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages.

#338: Reference groups by ID in addition to name, since group names can change

johnbywater — Mon, 07 Jun 2010 08:58:57 GMT

CKAN: Ticket Query

#320: site_title configuration variable which is used in template

#340: Web UI theme easier to configure

#226: UI Review - History

#227: UI Review - General checks

#324: Search indexing using notifications

#356: Search box in at top of page (UI)

#336: Resource Search API

As a

I want to

Proposed implementation

Example search

Search Parameters

Original request

#311: Reordering of package resources can lead to integri

#335: Post-package-edit redirect to configurable URL

As a

I want to

Design

Example

#323: Notification message

Which events to notify on

Ignored domain objects

Message format

#242: Miscellaneous tidying up (v0.11)

#328: Mention code libraries in API documentation

#317: Make search pluggable

#273: Investigate search index options and create tickets

#375: Integrate Forms API into Drupal

#315: Improvements and fixes to csv dump

#263: Improve and test openid login

#376: Implement servicization of CKAN API

#325: Event push notification

As a

I want to

Examples of use

Context

Design

Testing

Related

#337: Download links for resources should open in new window

#374: Design servicization of CKAN API

#310: Commit message box looks wrong in edit page since edit style overhaul

#322: Client interface for Notification Service

Use cases

Design

Routing detail

Versioning

Documentation

#326: Centralise importation of json library

#329: Bad dates cause exception on Gov form

Reproduction

Why it's happening

#233: Allow simple site-specific customization/overriding of templates

#313: Allow packages to be specified by IDs in REST interface

#333: CKAN front end requirements for package notifications

Use case: new package

Use case: CKAN imports packages

#338: Reference groups by ID in addition to name, since group names can change