As a

Authenticated User

I want to

Create/register a package

So that

The package will be listed in the system and visible to users

Notes

• Full package registration process may be broken down into:
  1. Creating a package by provision of minimal attributes:
     1. name: unique and web-usable
     2. auto-generation of default values such as:
        1. date-registered (today's date)
        2. first owner (one owner should be created corresponding to the authenticated user performing this action)
     3. (?) auto-generate a default release
  2. updating the created package -> ../UseCase12

As a

Authenticated user

I want to

Delete a package

So that

So that the package is no longer displayed for visitors of the site

Notes

• The permissions for this action are the same as for updating the package details (see ticket:10)
• When a package is deleted it is put in an inactive state and is invisible to users and vistors of the service. However its name remains reserved and it may be undeleted or purged by the service administrator

Provide a new test install of DataGM based on the ODS demo site being completed through June.

Should have all supported extensions enabled, those that don't work should be disabled and ticketed. Also needs a customised homepage.

Requirements

The ODS demo site will need data adding, initially as fixtures but it would also be useful if we started evaluating datasets that we can ship with ODS installations (at least in the UK) from places such as DGU and ONS.

May wish to create a ticket for making sure the datasets within the system are reset every X hours. Perhaps.

Interface

None

User Stories

• As a new system administrator for an ODS instance, I don't want to have a site devoid of any data. Geographically relevant datasets would be welcomed.

• As a bizdev person I would like to be able to demonstrate how ODS works with real datasets.

Tasks

[ ] Identify relevant sources for datasets

[ ] Pick datasets

[ ] Set them up for import

Estimates

Write a harvester for data.london.gov.uk to import catalogue metadata into PDEU. API (or at least documentation) is available at: ​http://sourceforge.net/projects/londondatastore/files/

Change the visual style of CKAN to be more like these sites:

• GitHub?
• Quora
• Google Projects

This does not include major UX work.

RDFa can be used as a simple way to expose linked data or at least sameAs the API version of the data but we need to make sure we do not expose anything different from the "official" representation.

We need to offer a strongly simplified version, read-only of CKAN under publicdata.eu, with a focus on its role as search engine instead of a data catalogue.

This ticket relates to work on the PDEU theme only!

A nice map in the homepage showing the availability of data across Europe

As CKAN grows, more and more optional representations of packages will become used. Besides RDF (which is the motivation for this ticket), support for DSPL, home-grown XML, or CSV listings is imaginable.

To properly support this CKAN should have an extensible representation registry that can be extended when new output types become available (and without changing the API in the process). This needs to integrate in two places:

• We need to add support for file format extensions to the package handler and REST API.
• We need to add HTTP content type negotiation (Accept headers).

To support this we should have a registry with two registers:

• to map format extensions to mime types (e.g. "json" -> "application/json")
• to map mime types and entity types to converter functions (e.g. 8"application/json", ckan.model.Package) -> func(obj, mime_type) )

This should be set up on load_environment so that IConfigurer plugins can feed into it.

The registry should then first be added to _finish in the REST API (which needs to be refactored to be passed the {format} part of the URL if one is given. Based on the format part and HTTP headers, an appropriate representation can be generated by the registry and then be returned to the user.

Forwarding of requests to the regular WUI controllers with Accept headers set or a format specified can be implemented in a separate effort.

Cost: 1h

Switch to repoze.who + openid plugin for authentication from authkit.

Already did this in microfacts so should be fairly easy.

Additional feature related to ticket:66.

cost: 2h

Integrate new logo from ​http://wiki.okfn.org/ckan/logo into site:

• In title

Also can do favicon (separate ticket:48)

Cost: 1h

Current form stuff is not very good (uses formencode). Switch to formalchemy would improve this, especially on validation.

As an extra we could utilize the formalchemy pylons admin interface (pretty much for free).

Cost: 12h

Details

1. Replace htmlfill and formencode extract in controllers/templates with formalchemy
   1. Crude and simple
   2. Suppress unwanted fields (revision, state, all revisions) and sort out ordering to be similar to before.
   3. Sort out tag field with a special renderer
2. Validation - testing definitely required.
3. Fix up description and pretty css etc

Get rid of modes in the RESTful API. Do json stuff directly in controllers.

• package will have to_dict and from_dict methods (called by rest controller) which provide and consume JSON friendly dictionaries representing the object. from_dict - class_method. stuff in forms for tags as_string, maybe factor out. reuse validation stuff.

Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: AccessControl.

Cost: 8d

As a user I want to be able to create groups of packages.

This functionality is different from tags:

• The ability to add a package to a given group is restricted whereas anyone can add a given tag to a package
• Specifically groups have owners and only the owners can add a package to that group

Group properties

• id (primarykey, uuid style)
• name (restricted content - same rules as package)
• title (no restrictions)
• owners -- many:many with user object
• description (markdown)
• packages -- many:many with package object

Don't version groups for now.

Groups address in the WUI will be:

• /group - browse list of groups (reuse list action)
• /group/list - same thing for now
• /group/<groupname> - display: group properties with links to packages. No links to user pages (yet).

Group editing and searching will be another ticket.

Cost: 3d

1. Package needs a groups property (backref Group.packages).
2. Put groups in the search api.
3. Display them on the package, but not editable. (Only editable from the group page.)

Follow up to ticket:105 (Groups).

Cost: 1d

Create in the model basic operation of Access Control.

roles table

name | context | action
-----------------------
admin| package | edit
admin| package | edit-permissions
admin| package | read
editor| package | update
editor| package | read
reader| package | read
This data is set-up on db init and will have no interface.

user-roles table:

username | context_type | objectid | role
rgrp     | system  | n/a | admin
visitor  | package | * | reader
bob      | package | geonames | admin
visitor  | package | geonames | editor
visitor  | package | geonames | reader
john     | group | ukgov | admin
dread    | group | ukgov | editor
visitor  | group | ukgov | reader
This data will be added when someone is given permissions for the system, a package or a group.

Pseudo code:

class Package

def is_allowed(name, action):

is_allowed(name, action, context=self)

class Group

def is_allowed(name, action):

is_allowed(name, action, context=self)

def is_allowed(name, action, context=None): name: string - a username or IP for 'visitor'

action: string - 'read', 'edit', 'delete', 'edit-permissions' context: object - a Group or a Package or None (which means system)

Based on a section of AccessControl design: WUI gives controls to user and administrator to change permissions on a package.

Take out import of paginate in setup.py. Use paginate in webhelpers instead. Make changes to take account of any i/f changes.

Look for all places where model is accessed and check authorization is checked.

Document holes (and, as necessary, suggestions for fixes) as new tickets. Likely areas that need looking at:

• search i/f
• package WUI commit

Write holes are obviously much more significant to us than read holes.

As a Package Admin I want to change the state of the package. In particular I wish to delete and undelete it.

(NB: this is quite separate from "purging" objects which is the term we shall use for irrevocable removal of an object from the domain model).

• Only Package Admins (and sysadmins) should be able to change state

Implementation Suggestions

• 'delete' action should be renamed to 'change-state' (NB: this requires a db migration ...)
• Have new package formalchemy form (created via inheritance?) to incorporate state attribute. Suggest this is rendered as a dropdown (and may be simple object rendering of state, i.e. do NOT need to change it to a single name such 'active').
• This form should then be used when the user satisfies is_authorized(..., model.Action.CHANGE_STATE) instead of the usual fieldset

As a

User

I want to

Create a new group via the WUI

Details

• locate at /group/new/
• You must be logged in to create a group
• Group creator is automatically given role of group admin
• Edit screen is same as edit screen ...

Support for download metadata (file size, file formats, version information).

See also: ticket:87 (multiple download links)

'paster db upgrade' should result in the same database as 'paster db create' on a clean system. In particular, all tests should pass.

Note: equivalent of paster db upgrade isn't being run from tests at the moment (we haven't been able to get this to work).

Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN.

Cost: 30m

Seems some package have ended up with same tag twice: ​http://www.ckan.net/package/fred2dot0

Check whether this is still possible and if it is fix it ...

Cost: 2h

Based on number of views.

Related to ticket:143.

Follows on from ticket:148. Sister to ticket:150.

User object should have a "created" attribute initialized to current datetime.

Require a db migration but o/w very simple.

Cost: 1.5h

This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there.

Suggested by Jonathan Gray

Log message on package edit page is no longer persisted through a preview (no doubt due to recent form upgrades). Let's fix this.

cost: 0.5h

Search of form: tags: ... behaves differently depending on whether there is a leading space:

• tags: postcode - tags found correctly but no packages found
• tags:postcode - tags incorrectly found but correct packages found

Let's fix this.

Cost: 0.5h

E.g. ​http://ckan.net/tag/read/postcodes this has no-postcode listed but when you look at that package 'postcodes' isn't a tag ...

I suspect this may be to do with search using deleted tags as I think no-postcode had postcodes as a tag at some point earlier ...

Related to: ticket:90 Link to RDF version of CKAN data on Talis Connected Commons ticket:164 Creating RDF data from CKAN

Regularly re-upload the CKAN RDF data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

Only first tag is updated.

Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible.

NB: improving the documentation is another matter (as is integrating e.g. existing api docs).

Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net):

• ​http://knowledgeforge.net/ckan/ckan/doc/
• (Other options: ​http://www.ckan.net/doc/ ​http://doc.ckan.net)

the "only openly licensed packages" tick box seems to return blank page every time - as do queries for only openly licensed and only downloadable packages. The "only downloadable packages" tick box by itself seems to work fine. E.g. try searching for 'science'.

Raised by Jonathan Gray

/tool New 'tools' section of the WUI with 'Excel package import' page.

/tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:

• license - name of license
• tags - space separated
• groups - space separated
• anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups.

Standard search should search notes field in addition to name, title and tags (discussed in ticket:108 but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results.

If we do this we need:

1. To weight across fields in a sensible way
2. We can also use proper text search on title or ...

Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: ​http://www.postgresql.org/docs/8.3/static/textsearch.html

Using this with sqlalchemy: ​http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/

Issues with fulltext search:

• tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking.
• if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better.
• can we split the name on dash or underscore before being indexed?
• natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'.
• do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres?
• we want to weight name and title higher than other fields - achievable with custom trigger.

See also: PackageResources which defines a 'distribution' as a package's payload (perhaps with metadata). In our case, at least within CKAN, the representation of the distribution will just be the reference to get it (plus some minor metadata). We are expanding this functionality to be any kind of 'resource' associated with a package since it is likely in some cases people will want to link to a file containing a part, but not the whole, of a package's data.

As a

User (package editor and package user)

I want to

(package editor) Associate multiple distributions with a package.

(package user) Easily have access to multiple distributions of a package should they exist.

Implementation

There is an object called 'Resource' with the following attributes:

• url (text) (required in UI, but not in db table)
• format (text)
• description (text)
• package (association) - a resource is always associated with 1 package

A package's list of resources will have an order which is preserved. The first in the list will be known as the 'default resource' (the presumption will always be that this is a 'distribution').

How to replace download url

download_url removed and replaced as a proxy attribute. Read returns the url of the default distribution. Write changes the url of the default resource. Ensures backward compatibility.

REST API has new attribute in package dictionary 'resouces' - an ordered list.

In WUI, replace download url in package read with a table of the distributions.

In the WUI package edit, be able to add/remove/edit/reorder distributions.

Example

Example part of package:

Distributions:

Existing data to be migrated to new structure.

Cost: 4 days

Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name.

Cost: 2h

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: ​http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: ​http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

1. i18n templates
   • Extract content using Babel
   • Get genshi translation working
   • Do the translation!
2. i18n all strings in controllers (if any)
3. i18n forms