As a

Visitor

I want to

View a tag's page which should list all packages associated with that tag (or link to a page containing that list).

Notes

• The tags should have a nice primary url such as /tag/<tag-name> (could also use the plural i.e. /tag/<tag-name>)

When IP address is used to identify non-logged in users (for example on create and editing of packages) it is always 127.0.0.1 when using a reverse proxy (standard deployment configuration on a production machine and used e.g. on ​http://www.ckan.net).

This could be corrected by using HTTP_X_FORWARDED_FOR when available.

Spam entries have started to occur on ckan.net in the last few months. It would be useful to be able to purge these revisions -- that is permanently delete all changes to domain objects associated with those revision (one might also permanently delete that revision -- or alternatively simply mark it as purged).

In addition to providing this facility from the shell it would also be useful to be able to do this from the web interface (with associated restrictions on usage via an authorization controller of some sort).

A url such as: ​http://someurl.com/xyz?x=1&VERSION=1.1&Service=WFS when set as url or download_url breaks the rendering of the package with an error like:

There was an error rendering the package: not well-formed (invalid token): line 1, column 181

Have checked that removing the & stuff makes the error go away so this looks like an issue with escaping urls when displaying them ...

As A

Visitor

I Want To

View lists of items but not have too many on a page (as this leads to slow page loads and difficulty navigating the list). Where there are more items in the results of my action that can be fit on a single page the results should be 'paged' in some manner.

Details

Main place this issue arises:

• List of tags
• List of packages
• Repository history (/revision)

Should probably have no more than 50 (or so) items on a page. Obvious solution is for actions that return lists to have some kind of pageno parameter and then use this in controller to select what to display.

As A

sysadmin

I Want To

Have revisions to the CKAN data filtered in order to reduce the spam in the system.

Details

In the long run this is a quite a generic problem common across several OKF systems and probably can become a general component in the okfmisc repo. For time being focus on a well-factored CKAN-specific solution.

Suggest we follow path of trac: ​http://trac.edgewall.org/wiki/SpamFilter

Could have a general engine that aggregates spam scores from many different 'plugins' and then marks spam appropriately (actions should be configurable depending on spam level from 'purge' to 'delete' (mark revision as inactive) to 'flag' to 'do nothing').

Main initial plugins would be:

• regex filter (this would seem very useful here, e.g. do not allow urls in commit messages ...)
  • could augment using the badcontent list approach (can find list on e.g. moinmoin)
• spambayes and/or akismet

Move CKAN codebase to use sqlalchemy (rather than sqlobject) along with vdm v0.2.

NB: the upgrade itself should not be too hard, the work will be in providing a satisfactory migration script to convert existing data on ckan.net service.

As A

Visitor/User?

I want to

View information about a package history, i.e. what revisions it has.

Details

• History info should be available at package/history/pkg-name
• Most basic info is just info from revision (when it occurred, author etc)
• More sophisticated would be to have diff support to see differences between revisions

As A

User or Visitor

I Want To

See simple icons (e.g. tick/cross) next to packages (in lists or on main page) indicating openness status of package.

So That

It is easy to see the openness status of a package.

Details

• Suggested by Liz Turner
• Openness = open license at present (no easy way to determine whether accessible)
• Have separate tick for downloadability (i.e. existence of download url)
• Could do more stuff in future (e.g. money where tag price- exists, etc etc)

Make bookmarklet to enable easy package registration on CKAN

Details

1. This requires redoing web interface to allow 1-step package creation/editing (as opposed to name then everything else approach at the moment)

Cost: 1h

Switch to repoze.who + openid plugin for authentication from authkit.

Already did this in microfacts so should be fairly easy.

1. Link at top should say "Your Account" when you are logged in
   • when not logged in should say: "Log in via openid"
   • Open Id info on /account/ should move to /account/login/ or be deleted
   • Going to /account/ when not logged in should redirect to /account/login/

2. When logged in the basic account index page should not give generic information but should reflect fact you are logged in by:

• Saying something like: "You are logged in as: ..."
• Showing your apikey or a link to apikey
• Giving you a logout link

We can think of a bunch of other stuff that could go on their (recent edits, packages you own etc) but these will be separate tickets.

When listing revisions (at /revision/ or in atom feed) show affected packages. Minor but v. useful UI improvement.

Cost: 1h

Cost: 1h

Integrate new logo from ​http://wiki.okfn.org/ckan/logo into site:

• In title

Also can do favicon (separate ticket:48)

Cost: 1h

When you login you get given a cookie but it is very short (life of your browser session I think).

Work out how to extend to something reasonable (30 days or perhaps forever with logout unsetting cookie).

1. A quick investigation to see whether this is possible
2. If possible do it

Cost: ?

Should have function/facility to get a diff of a version object between 2 revisions. Function should return a dictionary of fields with diffs.

Details:

• Do we deal with m2m relationships (and m2one on the many side)?
• What is diff?
  • For text fields output of python diff command
  • For non-text field guess just simple +/- for what was there before and now

Cost: 4h (don't think this is a huge request)

No real benefit of having State object as opposed to a simple text field with enumerated values and this second option is both simpler and more flexible.

Main hassle here is that it requires migration in all projects that use vdm.

If we're going to do this we should do it sooner rather than later.

Cost: 2h (+ 1h for a conversion method)

Multiple download links, including links to mirrors and multiple formats/versions

Support for listing files (manifest) contained within a package.

Traditional manifests just list the files. Suggest in addition we have support for optional metadata in form of key, value pairs.

(2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option).

Cost: 1h

Test DbIndex? with sqla 0.5 (and get it working if it does not work).

Cost: ??

Priority: low because DbIndex? is not currently that important to us (also wonder whether we should try using CKAN code here to stop us reinventing the wheel).

1. Given a url download from that url to disk
2. Given a package use download url to download to disk

Once on disk should uncompress (if necessary).

Details

• Should download either to tmp directory (defaulted in config) or a specified path on disk
• In download would like to support as many types of target urls as possible:
  1. Single files (usually compressed filesets but could be individual files)
  2. Index pages listing files to download
  3. Revision control repositories
  4. ...

This will obviously be a significant amount of work. Suggest:

1. Start off just supporting the first option.
2. Have something like a plugin system to handle new target types
3. Reuse existing work (e.g. easy_install and pip handle standard vcs such as svn, hg, git etc)

Cost: 1d

As a user i want to see a package at "version" X (NB: not revision X).

• When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version)
• Implementation:
  • Find revision for this version (search revision history for when version field was last 0.7)
  • Show pacakge at that revision (as in ticket:103)
• On history page also shows versions in list of revisions associated with the package

Similar to ticket:108 but for tags:

/api/search/tag?q

For tags search is extremely simple since you can only search by name.

Possible Extras

• Allow "exotic" ordering of results e.g. ordering by number of packages with that tag.
• Return this number with tag list.

'paster db upgrade' should result in the same database as 'paster db create' on a clean system. In particular, all tests should pass.

Note: equivalent of paster db upgrade isn't being run from tests at the moment (we haven't been able to get this to work).

Controllers and templates should not access db objects directly - they should do all access via authz module giving username. They are handed by a query that has already been filtered by the packages they are authorized to read.

(Additional idea to be discussed: When they request a package object, they are handed an copy of the db object - disconnected from the database - so it the db object can't be changed.)

A couple of tests can be reenabled when this is done: ckan.tests.functional.test_authz.TestUsage?.test_admin_list_deleted ckan.tests.functional.test_authz.TestUsage?.test_search_deleted

Using WUI or REST interface you can search packages and groups without authorization being checked.

On the REST interface you can also read all the attributes of the packages using the 'all-fields' option.

Can be fixed using more advanced query to check authz.

• Generic text box for markdown about the user 'About'

Model's user table reflects these:

• 'about' attribute

Seems some package have ended up with same tag twice: ​http://www.ckan.net/package/fred2dot0

Check whether this is still possible and if it is fix it ...

Cost: 2h

When viewing a group you see a list of the users who have admin rights to it. (Akin to owners / moderators)

Related to: ticket:90 Link to RDF version of CKAN data on Talis Connected Commons ticket:164 Creating RDF data from CKAN

Regularly re-upload the CKAN RDF data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

Create consolidated set of config by merging config from .datapkgrc and command line options and use that instead of mixture of command line options and other stuff ... (likely to be less buggy!)

Use vdm diff functionality (see ticket:82) to implement diffing of packages between versions in WUI.

In package history you should be able to select to revisions and show the diff.

Estimated cost: 12h

'created' attribute on User, Group and Rating

datetime object: default=datetime.datetime.now()

See example on revision

Requires migrate script

/tool New 'tools' section of the WUI with 'Excel package import' page.

/tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:

• license - name of license
• tags - space separated
• groups - space separated
• anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups.

With up to date statistics such as:

• total packages
• total tags
• total users
• most active users
• last update
• etc.

Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See ticket:184

Make parts of the package form hidden by default, to make it look simpler. Users are invited to click a button to reveal the less important fields.

Fields hidden by default: version, author, author_email, maintainer, maintainer_email

Standard search should search notes field in addition to name, title and tags (discussed in ticket:108 but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results.

If we do this we need:

1. To weight across fields in a sensible way
2. We can also use proper text search on title or ...

Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: ​http://www.postgresql.org/docs/8.3/static/textsearch.html

Using this with sqlalchemy: ​http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/

Issues with fulltext search:

• tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking.
• if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better.
• can we split the name on dash or underscore before being indexed?
• natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'.
• do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres?
• we want to weight name and title higher than other fields - achievable with custom trigger.

Cost - 2 days

The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09).

Example part of package:

Date: 5/10/09
or
Date: 1/1/09 – 1/4/09

rel=nofollow or robots.txt ?

See how we did it already for other things.

Note: on ckan.net older PackageRevision?.id might not be identical to Package.id but this may need sorting at this point.

Current write method just creates a bare bones package and does not write any of the available metadata other than name.

Importer function is exposed to logged in users.

Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'.

• Basic is as it is currently
• Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords.

All fields should be in step with the government custom input form.

For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text.

Estimate: