Expect to page through results.

Only page 1 is shown, all other pages fail to display.

Reproduce by searching for something common and browsing to the second page.

1. List in sidebar 3 latest created/updated packages.

2. Put package search box on front page (might also want to move some of front page content into an about page).

3. Similarly might want package creation box on front page.

It is essential packages have a version attribute to assist in tracking releases etc (and to allow compatibility with other packaging systems e.g. distutils which is being used by datapkg).

cost: 2h

Switch from license domain object to a simple license field and use license list from new centralised license repo:

<​http://knowledgeforge.net/okfn/licenses/>

• This will require a migration

Cost: 4h (plus migration ...)

As A

Visitor/User?

I want to

View information about a package history, i.e. what revisions it has.

Details

• History info should be available at package/history/pkg-name
• Most basic info is just info from revision (when it occurred, author etc)
• More sophisticated would be to have diff support to see differences between revisions

Using datapkg a user can:

• list packages on CKAN
• get info about package on CKAN
• register and update a package on CKAN

When a package listing is shown (as on ​http://ckan.net/package/list) more information than just name should be shown. For example package titles should be shown as well as names (could also list tags and other stuff but simplest is just title).

Cost: 1

Move CKAN codebase to use sqlalchemy (rather than sqlobject) along with vdm v0.2.

NB: the upgrade itself should not be too hard, the work will be in providing a satisfactory migration script to convert existing data on ckan.net service.

As a

Sysadmin

I want to

Dump (serialize) CKAN data to a simple transport format (e.g. JSON) and be able to load it again.

Details

• Already have simple db dump. However dumping to JSON has various advantages, particularly where changes to the data need to be made upon reloading (e.g. during a migration).
• Dump should include *all* CKAN data (i.e. all the data in the CKAN db tables)

Notes

• Currently have a text box where users can add space separated tags
• Tags should auto-complete in a manner similar to delicious
• would also be good to have a list of suggested tags. This could either be:
  • [simpler] a list below the tag box chosen by existing popularity or
  • [more complex] a list of suggestions that updates as you add tags (based again on popularity but 'filtered/reranked' to reflect choices so far).
  • For first attempt to should probably just go for the simpler option.

As A

Visitor

I Want To

View lists of items but not have too many on a page (as this leads to slow page loads and difficulty navigating the list). Where there are more items in the results of my action that can be fit on a single page the results should be 'paged' in some manner.

Details

Main place this issue arises:

• List of tags
• List of packages
• Repository history (/revision)

Should probably have no more than 50 (or so) items on a page. Obvious solution is for actions that return lists to have some kind of pageno parameter and then use this in controller to select what to display.

Spam entries have started to occur on ckan.net in the last few months. It would be useful to be able to purge these revisions -- that is permanently delete all changes to domain objects associated with those revision (one might also permanently delete that revision -- or alternatively simply mark it as purged).

In addition to providing this facility from the shell it would also be useful to be able to do this from the web interface (with associated restrictions on usage via an authorization controller of some sort).

A url such as: ​http://someurl.com/xyz?x=1&VERSION=1.1&Service=WFS when set as url or download_url breaks the rendering of the package with an error like:

There was an error rendering the package: not well-formed (invalid token): line 1, column 181

Have checked that removing the & stuff makes the error go away so this looks like an issue with escaping urls when displaying them ...

When IP address is used to identify non-logged in users (for example on create and editing of packages) it is always 127.0.0.1 when using a reverse proxy (standard deployment configuration on a production machine and used e.g. on ​http://www.ckan.net).

This could be corrected by using HTTP_X_FORWARDED_FOR when available.

As a

Visitor

I want to

Browse a list of packages resulting from a search or browse request (see other use cases)

So that

I can select one of the packages to view in more detail (-> viewing an individual package ticket:6)

Notes

• When browsing a list of packages you should be able to see summary information about the package such as title (though this may be shortened in order to conveniently fit the list
• The list should be broken up into pages so that the number of packages per page should be kept to a reasonable number (<= 50). Response time should be kept reasonable

As A

Site User

I Want To

Create/Edit/Update/Delete? tags associated with my package

Notes

• Tags are service-wide and not package specific
• Tags should not allow spaces (but allow everything else e.g. ':')
• web interface details
  • should provide a text box where users can add space separated tags (a la delicious).
  • tag creation should be seemless (so if a tag does not already tag exists it should automatically be created in the background)
  • would like to provide a list of relevant existing tags. For time being could just list all existing tags below entry box but in long run would like auto-updating/auto-suggesting listing a la delicious

Support for download metadata (file size, file formats, version information).

See also: ticket:87 (multiple download links)

'paster db upgrade' should result in the same database as 'paster db create' on a clean system. In particular, all tests should pass.

Note: equivalent of paster db upgrade isn't being run from tests at the moment (we haven't been able to get this to work).

Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN.

Cost: 30m

Seems some package have ended up with same tag twice: ​http://www.ckan.net/package/fred2dot0

Check whether this is still possible and if it is fix it ...

Cost: 2h

Based on number of views.

Related to ticket:143.

Follows on from ticket:148. Sister to ticket:150.

User object should have a "created" attribute initialized to current datetime.

Require a db migration but o/w very simple.

Cost: 1.5h

This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there.

Suggested by Jonathan Gray

Log message on package edit page is no longer persisted through a preview (no doubt due to recent form upgrades). Let's fix this.

cost: 0.5h

Search of form: tags: ... behaves differently depending on whether there is a leading space:

• tags: postcode - tags found correctly but no packages found
• tags:postcode - tags incorrectly found but correct packages found

Let's fix this.

Cost: 0.5h

E.g. ​http://ckan.net/tag/read/postcodes this has no-postcode listed but when you look at that package 'postcodes' isn't a tag ...

I suspect this may be to do with search using deleted tags as I think no-postcode had postcodes as a tag at some point earlier ...

Related to: ticket:90 Link to RDF version of CKAN data on Talis Connected Commons ticket:164 Creating RDF data from CKAN

Regularly re-upload the CKAN RDF data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

Only first tag is updated.

Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible.

NB: improving the documentation is another matter (as is integrating e.g. existing api docs).

Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net):

• ​http://knowledgeforge.net/ckan/ckan/doc/
• (Other options: ​http://www.ckan.net/doc/ ​http://doc.ckan.net)

the "only openly licensed packages" tick box seems to return blank page every time - as do queries for only openly licensed and only downloadable packages. The "only downloadable packages" tick box by itself seems to work fine. E.g. try searching for 'science'.

Raised by Jonathan Gray

/tool New 'tools' section of the WUI with 'Excel package import' page.

/tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:

• license - name of license
• tags - space separated
• groups - space separated
• anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups.

Standard search should search notes field in addition to name, title and tags (discussed in ticket:108 but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results.

If we do this we need:

1. To weight across fields in a sensible way
2. We can also use proper text search on title or ...

Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: ​http://www.postgresql.org/docs/8.3/static/textsearch.html

Using this with sqlalchemy: ​http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/

Issues with fulltext search:

• tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking.
• if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better.
• can we split the name on dash or underscore before being indexed?
• natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'.
• do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres?
• we want to weight name and title higher than other fields - achievable with custom trigger.

See also: PackageResources which defines a 'distribution' as a package's payload (perhaps with metadata). In our case, at least within CKAN, the representation of the distribution will just be the reference to get it (plus some minor metadata). We are expanding this functionality to be any kind of 'resource' associated with a package since it is likely in some cases people will want to link to a file containing a part, but not the whole, of a package's data.

As a

User (package editor and package user)

I want to

(package editor) Associate multiple distributions with a package.

(package user) Easily have access to multiple distributions of a package should they exist.

Implementation

There is an object called 'Resource' with the following attributes:

• url (text) (required in UI, but not in db table)
• format (text)
• description (text)
• package (association) - a resource is always associated with 1 package

A package's list of resources will have an order which is preserved. The first in the list will be known as the 'default resource' (the presumption will always be that this is a 'distribution').

How to replace download url

download_url removed and replaced as a proxy attribute. Read returns the url of the default distribution. Write changes the url of the default resource. Ensures backward compatibility.

REST API has new attribute in package dictionary 'resouces' - an ordered list.

In WUI, replace download url in package read with a table of the distributions.

In the WUI package edit, be able to add/remove/edit/reorder distributions.

Example

Example part of package:

Distributions:

Existing data to be migrated to new structure.

Cost: 4 days

Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name.

Cost: 2h

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: ​http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: ​http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

1. i18n templates
   • Extract content using Babel
   • Get genshi translation working
   • Do the translation!
2. i18n all strings in controllers (if any)
3. i18n forms

As a

admin

I want to

input government data in a constrained way

so that

you get consistency in the db which is easily searched, discovered and automatically linked.

implementation

Form has fields suitable for government data. (related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)

Simple fields

• all basic ckan fields
• default license to OKD::Crown Copyright
• co_id, precision, taxonomy url, agency

Date fields

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

• date_released

• date_updated

• temporal_coverage_from, temporal_coverage_to - two date fields combined

Strict selection

Drop down combo

• department

Selection or free text

Editable combo

• update_frequency - selections: never, weekly, monthly, annually, discontinued

• geographic_granularity - selections: national, regional, local authority, ward, point

• temporal_granularity - selections: yes, months, weeks, days, hours, point

Specialised fields

• geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

• national_statisic - UI is check box, store as 'Yes' or 'No'

• resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.

• categories - multi values from strict selection of ONS categories. Stored as comma-separated.

Field data processed

• tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.

Future - optional

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

We are already using standard kforge css though copied locally and with a few modifications.

We should switch to using common copy on m.okfn.org. Some work will be required to factor out any changes we have made.

Cost: 2h

• Set hover names on navigation links.
• Do we really need to display "About CKAN" everywhere? Just on home page.
• Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions.

• openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS.
• maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence.
• results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise.
• maybe links should go on the full titles? Get rid of package names.
• Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS?
• same comments as search listing (ticket:217)

Groups landing page group

• Yeesh! What's up with the typography for "Just login..."?
• Truncated description along with group name.
• Explanation text about what groups are for (RHS)
• Another case of <li> bullets adding nothing.

As A

User

I Want To

Add arbitrary named attributes to packages (an attribute being a name, type, value triple).

Details

• We will do this using a dedicated (versioned) table associated to Package
• Do we allow multiple attributes of the same name?
  • For the present: No (since we will key by attribute name)
  • Could allow for single attribute but with multiple values using json list ...
• What types do we allow or do we just rely on JSON to take care of this?

Questions (Original)

• How complex is this to implement?
• What would an arbitrary user be able to edit? Possibilities:
  1. 'create new attribute' and setting the value (so name and type would be chosen from predefined list).
  2. 'create', setting of name and value (but not type -- type already set in predefined list)
• Could just use (machine) tags -- though this could be seen as a bit of a hack.
• Would solve having to create special file/url attributes (though I think that perhaps file stuff is important enough to merit first class support in the domain model -- though, that said, since one won't want to have a file limit adding unlimited file support is very similar to unlimited attributes of arbitrary type).

Requires us to url encode the tag names when displaying them ...

1. Link at top should say "Your Account" when you are logged in
   • when not logged in should say: "Log in via openid"
   • Open Id info on /account/ should move to /account/login/ or be deleted
   • Going to /account/ when not logged in should redirect to /account/login/

2. When logged in the basic account index page should not give generic information but should reflect fact you are logged in by:

• Saying something like: "You are logged in as: ..."
• Showing your apikey or a link to apikey
• Giving you a logout link

We can think of a bunch of other stuff that could go on their (recent edits, packages you own etc) but these will be separate tickets.

Cost: 2h