As A

User

I want To

Create/register and update a package via a web api.

As A

Site User

I Want To

Create/Edit/Update/Delete? tags associated with my package

Notes

• Tags are service-wide and not package specific
• Tags should not allow spaces (but allow everything else e.g. ':')
• web interface details
  • should provide a text box where users can add space separated tags (a la delicious).
  • tag creation should be seemless (so if a tag does not already tag exists it should automatically be created in the background)
  • would like to provide a list of relevant existing tags. For time being could just list all existing tags below entry box but in long run would like auto-updating/auto-suggesting listing a la delicious

As a

Visitor

I want to

View a list of the tags on the system and search for a tag by name

Notes

• As for a search of packages should search for tags that contains a given search string (or could use a glob approach allowing wildcards).

When editing a package one would like to be able to preview one's changes before saving.

Currently all data is only available via the web interface which makes it difficult to obtain in bulk. Data should be provided in other more convenient, machine usable forms:

• DB Dump
• JSON for each package page (maybe Atom as well)

A url such as: ​http://someurl.com/xyz?x=1&VERSION=1.1&Service=WFS when set as url or download_url breaks the rendering of the package with an error like:

There was an error rendering the package: not well-formed (invalid token): line 1, column 181

Have checked that removing the & stuff makes the error go away so this looks like an issue with escaping urls when displaying them ...

As A

Visitor

I Want To

View lists of items but not have too many on a page (as this leads to slow page loads and difficulty navigating the list). Where there are more items in the results of my action that can be fit on a single page the results should be 'paged' in some manner.

Details

Main place this issue arises:

• List of tags
• List of packages
• Repository history (/revision)

Should probably have no more than 50 (or so) items on a page. Obvious solution is for actions that return lists to have some kind of pageno parameter and then use this in controller to select what to display.

As A

Visitor

I Want To

Get an RSS/Atom Feed of the Repository History to use in my feed reader (or elsewhere).

Details

• Preference for Atom.
• should just add parameter to /revision/list/ (or /revision/) to select atom format e.g. ?format=atom.
• should have a 'days' attribute specifying number of days back to go e.g. &days=30

Cost

Low

Because CKAN's worth it!

Move CKAN codebase to use sqlalchemy (rather than sqlobject) along with vdm v0.2.

NB: the upgrade itself should not be too hard, the work will be in providing a satisfactory migration script to convert existing data on ckan.net service.

As a

Sysadmin

I want to

Dump (serialize) CKAN data to a simple transport format (e.g. JSON) and be able to load it again.

Details

• Already have simple db dump. However dumping to JSON has various advantages, particularly where changes to the data need to be made upon reloading (e.g. during a migration).
• Dump should include *all* CKAN data (i.e. all the data in the CKAN db tables)

As A

User or Visitor

I Want To

See simple icons (e.g. tick/cross) next to packages (in lists or on main page) indicating openness status of package.

So That

It is easy to see the openness status of a package.

Details

• Suggested by Liz Turner
• Openness = open license at present (no easy way to determine whether accessible)
• Have separate tick for downloadability (i.e. existence of download url)
• Could do more stuff in future (e.g. money where tag price- exists, etc etc)

Make bookmarklet to enable easy package registration on CKAN

Details

1. This requires redoing web interface to allow 1-step package creation/editing (as opposed to name then everything else approach at the moment)

1. List in sidebar 3 latest created/updated packages.

2. Put package search box on front page (might also want to move some of front page content into an about page).

3. Similarly might want package creation box on front page.

Requires us to url encode the tag names when displaying them ...

Switch to repoze.who + openid plugin for authentication from authkit.

Already did this in microfacts so should be fairly easy.

1. Link at top should say "Your Account" when you are logged in
   • when not logged in should say: "Log in via openid"
   • Open Id info on /account/ should move to /account/login/ or be deleted
   • Going to /account/ when not logged in should redirect to /account/login/

2. When logged in the basic account index page should not give generic information but should reflect fact you are logged in by:

• Saying something like: "You are logged in as: ..."
• Showing your apikey or a link to apikey
• Giving you a logout link

We can think of a bunch of other stuff that could go on their (recent edits, packages you own etc) but these will be separate tickets.

When listing revisions (at /revision/ or in atom feed) show affected packages. Minor but v. useful UI improvement.

Cost: 1h

Cost: 1h

Integrate new logo from ​http://wiki.okfn.org/ckan/logo into site:

• In title

Also can do favicon (separate ticket:48)

Cost: 1h

Current form stuff is not very good (uses formencode). Switch to formalchemy would improve this, especially on validation.

As an extra we could utilize the formalchemy pylons admin interface (pretty much for free).

Cost: 12h

Details

1. Replace htmlfill and formencode extract in controllers/templates with formalchemy
   1. Crude and simple
   2. Suppress unwanted fields (revision, state, all revisions) and sort out ordering to be similar to before.
   3. Sort out tag field with a special renderer
2. Validation - testing definitely required.
3. Fix up description and pretty css etc

When you login you get given a cookie but it is very short (life of your browser session I think).

Work out how to extend to something reasonable (30 days or perhaps forever with logout unsetting cookie).

1. A quick investigation to see whether this is possible
2. If possible do it

Cost: ?

Get rid of modes in the RESTful API. Do json stuff directly in controllers.

• package will have to_dict and from_dict methods (called by rest controller) which provide and consume JSON friendly dictionaries representing the object. from_dict - class_method. stuff in forms for tags as_string, maybe factor out. reuse validation stuff.

Should have function/facility to get a diff of a version object between 2 revisions. Function should return a dictionary of fields with diffs.

Details:

• Do we deal with m2m relationships (and m2one on the many side)?
• What is diff?
  • For text fields output of python diff command
  • For non-text field guess just simple +/- for what was there before and now

Cost: 4h (don't think this is a huge request)

No real benefit of having State object as opposed to a simple text field with enumerated values and this second option is both simpler and more flexible.

Main hassle here is that it requires migration in all projects that use vdm.

If we're going to do this we should do it sooner rather than later.

Cost: 2h (+ 1h for a conversion method)

Support for download metadata (file size, file formats, version information).

See also: ticket:87 (multiple download links)

We should link to representations on semantic.ckan.net

• Each package page could link to its RDF representation
  • "Alternative formats" near star rating. Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f).
  • Some instances of ckan won't need this, so have option in ini file for this. rdf_store_url=xyz
  • Include machine-readable link to the RDF:

    <meta http-equiv=... />
    

  • 303 redirect to RDF when content negotiation requires RDF. Test:

    curl -I -H "Accept: application/rdf+xml" http://localhost:5000/package/warandpeace
    

Cost: 3h

Future

• Search page to link to SPARQL interface: ​http://semantic.ckan.net/sparql
• License/dump page could link to full RDF dump.

Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: AccessControl.

Cost: 8d

Support for listing files (manifest) contained within a package.

Traditional manifests just list the files. Suggest in addition we have support for optional metadata in form of key, value pairs.

(2008-10-29) convert existing directory on disk to a datapkg distribution (PythonDistribution? by default). This would be presented in the CLI as a convert command.

Details:

• Add a metadata file (setup.py), perhaps prompting for input
• List all existing data in that directory

Cost: 4h

This is better for long term sustainability, especially if we want to "push and pull" from unrelated repositories in the future.

Details:

• May wish to retain a number field updated in the usual incremental way (what's the cost/benefit here?)

As a user i want to see a package at "version" X (NB: not revision X).

• When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version)
• Implementation:
  • Find revision for this version (search revision history for when version field was last 0.7)
  • Show pacakge at that revision (as in ticket:103)
• On history page also shows versions in list of revisions associated with the package

Add package search facility in the rest api at /api/search

Queries can be provided as for the normal package search either by posting to that url or by performing a get with a query string.

E.g. .../api/search/package?q=xyz

Query parameters:

• q is keyword string (searches name, title, tags by default)
  • split by words and ANDed
• additional parameters for specific fields
• qjson is alternative to q for Searching by specific fields (in addition to keyword string). Payload to json is a JSON-encoded dict which is a dictionary with a q field for free text (keywords) and additional key/value pairs for specific fields
  • if q and qjson specified ignore q and just process qjson value

Additional parameters in addition to query ("q" or "qjson") are:

• limit
• offset
• fullinfo=0/1 - return full record for each result (default=0)
• order_by=field_name
• search_notes=0/1 (default 0) do we search notes field in a keyword search

Return value is json encoded dictionary with keys:

• 'results': list of results
• 'count': total number of results

Extras (for the future)

• Prioritisation is not part of this ticket (requires fulltext support in DB or in external app such as Xapian)

1. Package needs a groups property (backref Group.packages).
2. Put groups in the search api.
3. Display them on the package, but not editable. (Only editable from the group page.)

Follow up to ticket:105 (Groups).

Cost: 1d

Must not alter the db.

Related to ticket:112.

Based on a section of AccessControl design: Reading and writing to packages in the WUI and REST API are now dependent on the authz tables.

Based on a section of AccessControl design: Group reads and edits are controlled by access control. WUI and REST interfaces covered.

Ensure pending packages don't show up in search or browse

cost: 4h

And associated page to browse group.

Add Group editing page.

If no permissions to change group can't edit group. Also cannot view edit page.

Editable attributes: name, title, description

No preview needed

As a Package Admin I want to change the state of the package. In particular I wish to delete and undelete it.

(NB: this is quite separate from "purging" objects which is the term we shall use for irrevocable removal of an object from the domain model).

• Only Package Admins (and sysadmins) should be able to change state

Implementation Suggestions

• 'delete' action should be renamed to 'change-state' (NB: this requires a db migration ...)
• Have new package formalchemy form (created via inheritance?) to incorporate state attribute. Suggest this is rendered as a dropdown (and may be simple object rendering of state, i.e. do NOT need to change it to a single name such 'active').
• This form should then be used when the user satisfies is_authorized(..., model.Action.CHANGE_STATE) instead of the usual fieldset

Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN.

Cost: 30m

Controlling Groups through a REST interface.

Using WUI or REST interface you can search packages and groups without authorization being checked.

On the REST interface you can also read all the attributes of the packages using the 'all-fields' option.

Can be fixed using more advanced query to check authz.

• Generic text box for markdown about the user 'About'

Model's user table reflects these:

• 'about' attribute
• 'created' attribute is a default=datetime.now() - so when user was created

Need edit page for this.

Only user should be able to edit it.

Follows on from ticket:136 and ticket:142

Cost: 2h

Suggested fields:

• id, name (username/profile name), openid, email, password, extras (JsonType?)
  • + current items: apikey, about, created)
  • current contents of name is usually openid and should migrate to openid field (could match based on starting with ​http://)
• May also want a fullname field (or perhaps that can

just go in extras - may be better to have its own field as useful to search on this )

Related changes (probably separate tickets):

• Improve user home page to include this additional information (and have it be editable)
• Use user 'name' (username) for home page links (rather than current user/<id>)
• Where we show lists of revisions, each revision author is now a link to the author's user page.

Related to ticket:136 and ticket:138.

• parse like data4nr code

Follow up tickets: ticket:149 and ticket:150