At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys)

e.g. Text on ckan.net home page:

"All material available under an /licenseopen license"

Also seen on the package edit page: "/user/loginClick here to sign in"

I believe this is due to the i18n additions from Benoit. Maybe needs new Genshi. Maybe best to work round in the meantime.

Dumping ground for various minor and miscellaneous items (mainly refactorings):rgrp

• controllers/tag.py: why does this not use lib/search.py (but controllers/packages.py search method does)?

done:

• lib/search.py: inheritance would be nicer than switching on entity in search (i.e. have SearchPackage?, SearchTag? etc)

Currently it is possible (v0.11) to have a PackageResource? object without a url. This should be impossible to do (either at creation time or via modification).

Broken out from ticket:216

Strapline (?): Sharing, discovery and reuse of data and content

Need to consult on this.

Cost: 1h

• Requires Package Read to not use the side-bar
• Need to sync params to a package object that is taken out of the session so that it doesn't go into the database.

This needs to be thought out ...

Repository History revision

• What is this page called? "Recent Changes", "Repository History", "/revision": standardise. Between the link in the nav and the page <title> particularly, but the route is also important. Perhaps /changes or something similar?
  • Will change page title on /revision/ to Revision History. Will not change route for the time being.
• needs copyediting.
• Pagination has similar issues to elsewhere. Also, most obvious here, is the fact that we don't need to display a link to every possible page. Please can we limit it to, say, a dozen nearby pages and an ellipsis.
• Without looking at dates, its not clear whether I'm seeing most recent or oldest changes. Change pagination to say "Older"/"More recent" rather than "Previous"/"Next". (wontfix: now have text saying we are showing most recent changes)
• Table layout is pretty ugly (yes, I'm aware this is my fault).
  • wontfix - nothing better at the moment
• Timestamps are horribly unreadable. At the absolute minimum get rid of the micros. Hover for more detail?
• Can we link to an author page? Yes!
• Atom feed should have a feed icon~~
• Why are we adding another <link rel="alternate"> to this page? We already have one for recent changes on every page in the site and the one we're adding has a less descriptive title. Which is correct? Use that one as a feed for every page.

Groups landing page group

• Yeesh! What's up with the typography for "Just login..."?
• Truncated description along with group name.
• Explanation text about what groups are for (RHS)
• Another case of <li> bullets adding nothing.

Package history page package/history/mypackagename

• "Revisions" is redundant
• "Compare" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it!
• Ditto to overall "Recent Changes" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links.

Package page package/mypackagename

• Design
  • In general: a huge amount of visual and typographic noise: "Package:" is redundant.
  • "{edit}", "{history}" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is "make an enquiry with Is It Open?" bigger than these?
  • Openness/downloadability icons: need help, as discussed earlier
  • "Rating:" redundant, as is the second set of stars -- why can't both the display and rate widget
    • Rating - alternative to GET on set_rating.
  • "Metadata:" technospeak, probably redundant.
  • <li> bullets add nothing.
  • Title, Version, URL can probably go in one line.
  • Resources can be made more prominent.
  • Author, author email and Maintainer, maintainer email can be merged.
  • Groups, tags in little boxes?
  • Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface?
  • Don't show extras unless there are any.

• openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS.
• maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence.
• results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise.
• maybe links should go on the full titles? Get rid of package names.
• Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS?
• same comments as search listing (ticket:217)

• allow clicking on 'only openly licensed'/'only downloadable' to select/deselect checkboxes: make them <label>s.
• cut 'only's -- use word 'filter' instead
• doesn't appear to match substrings!? A search for 'anna' doesn't return match the 'annakarenina' package.
• autocomplete package names & tags

• Set hover names on navigation links.
• Do we really need to display "About CKAN" everywhere? Just on home page.
• Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions.

We are already using standard kforge css though copied locally and with a few modifications.

We should switch to using common copy on m.okfn.org. Some work will be required to factor out any changes we have made.

Cost: 2h

Native registration scenario: Complete registration form, require unique email address (indicate this not displayed on site), full name (indicate this is displayed on site), password. Dispatch email confirmation message. Respond to email confirmation message causes user to be logged in. Users are assigned a uuid.

Obfuscate author email and maintainer email field.

Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@…', encode='javascript')

Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate?

Investigate and list (on the wiki at wiki:UiReviewNotes) CKAN's UI issues, in particular:

• Form usability
  • best example: ​http://ckan.net/package/new
  • "Extras" fields
  • Adding to groups (from group page or from package page?)

• Overview pages: reduction in visual noise and unnecessary elements.
  • best example: package overview page
  • consider openness/downloadability icons
  • display common metadata fields in a more structured fashion?

Est cost: 2 hrs.

As a

admin

I want to

input government data in a constrained way

so that

you get consistency in the db which is easily searched, discovered and automatically linked.

implementation

Form has fields suitable for government data. (related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)

Simple fields

• all basic ckan fields
• default license to OKD::Crown Copyright
• co_id, precision, taxonomy url, agency

Date fields

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

• date_released

• date_updated

• temporal_coverage_from, temporal_coverage_to - two date fields combined

Strict selection

Drop down combo

• department

Selection or free text

Editable combo

• update_frequency - selections: never, weekly, monthly, annually, discontinued

• geographic_granularity - selections: national, regional, local authority, ward, point

• temporal_granularity - selections: yes, months, weeks, days, hours, point

Specialised fields

• geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

• national_statisic - UI is check box, store as 'Yes' or 'No'

• resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.

• categories - multi values from strict selection of ONS categories. Stored as comma-separated.

Field data processed

• tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.

Future - optional

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: ​http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: ​http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

1. i18n templates
   • Extract content using Babel
   • Get genshi translation working
   • Do the translation!
2. i18n all strings in controllers (if any)
3. i18n forms

Current write method just creates a bare bones package and does not write any of the available metadata other than name.

See how we did it already for other things.

Note: on ckan.net older PackageRevision?.id might not be identical to Package.id but this may need sorting at this point.

Provide RDF version of a package at the RDF URI: ​http://ckan.net/package/rdf/32000-naples-florida-businesses-kml

See email on OKFN help:

For the information to be Linked Data, the following URI (from the api.talis.com SPARQL call) should be resolvable to RDF, or an RDFa enhanced HTML page.

rel=nofollow or robots.txt ?

Cost - 2 days

The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09).

Example part of package:

Date: 5/10/09
or
Date: 1/1/09 – 1/4/09

Cost 7 days

When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API.

The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user.

Example at bottom of package page:

Leave a comment:

Comments:

Implementation details:

Comments table is with columns:

Propose change to tabular-like format showing these attributes (perhaps should be configurable?)

• Openness status
• Title (not sure name is needed)
• Tags

Cost: 4h

(Follows on from ticket:107). We want to provide facility for users to automatically upload material.

• Create an upload page
• Store the material on archive.org using s3 - ​http://www.archive.org/help/abouts3.txt
• Main issue is how we cache large datasets ...

Create /stats/ page displaying main statistics, e.g:

• Most highly rated packages
• Most edited packages
• Largest groups
• Top tags (by packages)
• Package addition rate
• Users with most packages

Related to ticket:181 - Stats side-bar

Details

• Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display)
• For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html

Tick and cross aren't clear enough. Propose:

• Icon for downloadability (nothing if absent ...(?))
• Icon for open license? (Need something if closed ...)

Cost: 2h (?)

Create big tag cloud with all CKAN tags - perhaps weighting with size and colour...

/tool New 'tools' section of the WUI with 'Excel package import' page.

/tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:

• license - name of license
• tags - space separated
• groups - space separated
• anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups.

(Related to ticket:169 - Package derivations)

A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.)

e.g. london-traffic-visualisation depends on road-map

'Dependency' relationship is:

• directional
• many:many
• stateful

'dependency' table columns:

• id (primary key)
• dependent (foreign key)
• dependency (foreign key)

Further tickets:

• WUI - package view - have list of dependencies (do not need to list packages which depend on this one)
• WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package')
• REST api - expose reading and writing 'depends on' property.

Issues

• How do we deal with dependency at a particular version?

the "only openly licensed packages" tick box seems to return blank page every time - as do queries for only openly licensed and only downloadable packages. The "only downloadable packages" tick box by itself seems to work fine. E.g. try searching for 'science'.

Raised by Jonathan Gray

Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible.

NB: improving the documentation is another matter (as is integrating e.g. existing api docs).

Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net):

• ​http://knowledgeforge.net/ckan/ckan/doc/
• (Other options: ​http://www.ckan.net/doc/ ​http://doc.ckan.net)

e.g.

​http://www.ckan.net/package/search?q=science&search=Search+Packages+%C2%BB

Results are capped at 20, but can't see past first page.

Ensure logging is working on server.

Related to: ticket:90 Link to RDF version of CKAN data and ticket:163 (regularly convert CKAN data to RDF).

Convert ckan data to RDF with suitable ontology and put on semantic.ckan.net.

Rather than raising a validation error automatically lower-case tags on submission.

Cost: 0.5h

Suggest form is: key:value (with optional space after ':'.

Keys would be for an object, i.e. 'package', 'tag', 'group'.

Becomes a link.

This would be implemented by implementing a 'Parser' or 'Processor' object which is run before markdown is done. Regex replace.

Cost: 4h

• Ensure you can add/change/delete extra fields from packages using the REST API.
• Update API docs to cover this.

On Package get, ensure separation of extra attributes from other attributes.

What is the meaning of posting an incomplete dictionary (updating a package)? Option 1 - if field not present, leave field alone

• Delete field if field value is None

Option 2 - user must supply all fields

• Delete field if field absent

Prefer option 1. Document this.

Follows on from ticket:152

Not just links that need to be displayed. Other html should be displayed too, such as tables which can't be represented in markdown. Do a search of the existing fields to see what is needed.

Give guidance on what to put in 'author' and 'maintainer' fields (e.g. name? URL?)

Suggestion from Jonathan Gray

Package edit page has fields for extra key-value pairs. Don't use formalchemy. The form displays the existing ones and new ones as follows:

Key           Value            delete
[ country   ] [ uk, spain   ]  [x]

When presenting the form, existing pairs and 3 blank pairs are provided.

When saving the data, keys that are not changed use the existing PackageExtra object (perhaps with new value). Deleted keys put the PackageExtra state to deleted. For new keys, only create a new PackageExtra if there isn't a state=deleted one to resurrect.

Validators ensure no repeated keys.

(instead of sqlalchemy interface, to facilitate general usage of the loader)

• use ckanclient

Follows on from ticket:148 and ticket:149

• parse like data4nr code

Follow up tickets: ticket:149 and ticket:150