(Related to ticket:169 - Package derivations)

A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.)

e.g. london-traffic-visualisation depends on road-map

'Dependency' relationship is:

• directional
• many:many
• stateful

'dependency' table columns:

• id (primary key)
• dependent (foreign key)
• dependency (foreign key)

Further tickets:

• WUI - package view - have list of dependencies (do not need to list packages which depend on this one)
• WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package')
• REST api - expose reading and writing 'depends on' property.

Issues

• How do we deal with dependency at a particular version?

/tool New 'tools' section of the WUI with 'Excel package import' page.

/tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:

• license - name of license
• tags - space separated
• groups - space separated
• anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups.

Create big tag cloud with all CKAN tags - perhaps weighting with size and colour...

Tick and cross aren't clear enough. Propose:

• Icon for downloadability (nothing if absent ...(?))
• Icon for open license? (Need something if closed ...)

Cost: 2h (?)

Create /stats/ page displaying main statistics, e.g:

• Most highly rated packages
• Most edited packages
• Largest groups
• Top tags (by packages)
• Package addition rate
• Users with most packages

Related to ticket:181 - Stats side-bar

Details

• Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display)
• For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html

(Follows on from ticket:107). We want to provide facility for users to automatically upload material.

• Create an upload page
• Store the material on archive.org using s3 - ​http://www.archive.org/help/abouts3.txt
• Main issue is how we cache large datasets ...

Propose change to tabular-like format showing these attributes (perhaps should be configurable?)

• Openness status
• Title (not sure name is needed)
• Tags

Cost: 4h

Cost 7 days

When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API.

The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user.

Example at bottom of package page:

Leave a comment:

Comments:

Implementation details:

Comments table is with columns:

Cost - 2 days

The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09).

Example part of package:

Date: 5/10/09
or
Date: 1/1/09 – 1/4/09

rel=nofollow or robots.txt ?

Provide RDF version of a package at the RDF URI: ​http://ckan.net/package/rdf/32000-naples-florida-businesses-kml

See email on OKFN help:

For the information to be Linked Data, the following URI (from the api.talis.com SPARQL call) should be resolvable to RDF, or an RDFa enhanced HTML page.

See how we did it already for other things.

Note: on ckan.net older PackageRevision?.id might not be identical to Package.id but this may need sorting at this point.

Current write method just creates a bare bones package and does not write any of the available metadata other than name.

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: ​http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: ​http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

1. i18n templates
   • Extract content using Babel
   • Get genshi translation working
   • Do the translation!
2. i18n all strings in controllers (if any)
3. i18n forms

As a

admin

I want to

input government data in a constrained way

so that

you get consistency in the db which is easily searched, discovered and automatically linked.

implementation

Form has fields suitable for government data. (related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)

Simple fields

• all basic ckan fields
• default license to OKD::Crown Copyright
• co_id, precision, taxonomy url, agency

Date fields

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

• date_released

• date_updated

• temporal_coverage_from, temporal_coverage_to - two date fields combined

Strict selection

Drop down combo

• department

Selection or free text

Editable combo

• update_frequency - selections: never, weekly, monthly, annually, discontinued

• geographic_granularity - selections: national, regional, local authority, ward, point

• temporal_granularity - selections: yes, months, weeks, days, hours, point

Specialised fields

• geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

• national_statisic - UI is check box, store as 'Yes' or 'No'

• resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.

• categories - multi values from strict selection of ONS categories. Stored as comma-separated.

Field data processed

• tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.

Future - optional

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

Investigate and list (on the wiki at wiki:UiReviewNotes) CKAN's UI issues, in particular:

• Form usability
  • best example: ​http://ckan.net/package/new
  • "Extras" fields
  • Adding to groups (from group page or from package page?)

• Overview pages: reduction in visual noise and unnecessary elements.
  • best example: package overview page
  • consider openness/downloadability icons
  • display common metadata fields in a more structured fashion?

Est cost: 2 hrs.

Obfuscate author email and maintainer email field.

Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@…', encode='javascript')

Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate?

Native registration scenario: Complete registration form, require unique email address (indicate this not displayed on site), full name (indicate this is displayed on site), password. Dispatch email confirmation message. Respond to email confirmation message causes user to be logged in. Users are assigned a uuid.

We are already using standard kforge css though copied locally and with a few modifications.

We should switch to using common copy on m.okfn.org. Some work will be required to factor out any changes we have made.

Cost: 2h

• Set hover names on navigation links.
• Do we really need to display "About CKAN" everywhere? Just on home page.
• Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions.

• allow clicking on 'only openly licensed'/'only downloadable' to select/deselect checkboxes: make them <label>s.
• cut 'only's -- use word 'filter' instead
• doesn't appear to match substrings!? A search for 'anna' doesn't return match the 'annakarenina' package.
• autocomplete package names & tags

• openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS.
• maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence.
• results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise.
• maybe links should go on the full titles? Get rid of package names.
• Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS?
• same comments as search listing (ticket:217)

Package page package/mypackagename

• Design
  • In general: a huge amount of visual and typographic noise: "Package:" is redundant.
  • "{edit}", "{history}" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is "make an enquiry with Is It Open?" bigger than these?
  • Openness/downloadability icons: need help, as discussed earlier
  • "Rating:" redundant, as is the second set of stars -- why can't both the display and rate widget
    • Rating - alternative to GET on set_rating.
  • "Metadata:" technospeak, probably redundant.
  • <li> bullets add nothing.
  • Title, Version, URL can probably go in one line.
  • Resources can be made more prominent.
  • Author, author email and Maintainer, maintainer email can be merged.
  • Groups, tags in little boxes?
  • Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface?
  • Don't show extras unless there are any.

Package history page package/history/mypackagename

• "Revisions" is redundant
• "Compare" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it!
• Ditto to overall "Recent Changes" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links.

Groups landing page group

• Yeesh! What's up with the typography for "Just login..."?
• Truncated description along with group name.
• Explanation text about what groups are for (RHS)
• Another case of <li> bullets adding nothing.

Repository History revision

• What is this page called? "Recent Changes", "Repository History", "/revision": standardise. Between the link in the nav and the page <title> particularly, but the route is also important. Perhaps /changes or something similar?
  • Will change page title on /revision/ to Revision History. Will not change route for the time being.
• needs copyediting.
• Pagination has similar issues to elsewhere. Also, most obvious here, is the fact that we don't need to display a link to every possible page. Please can we limit it to, say, a dozen nearby pages and an ellipsis.
• Without looking at dates, its not clear whether I'm seeing most recent or oldest changes. Change pagination to say "Older"/"More recent" rather than "Previous"/"Next". (wontfix: now have text saying we are showing most recent changes)
• Table layout is pretty ugly (yes, I'm aware this is my fault).
  • wontfix - nothing better at the moment
• Timestamps are horribly unreadable. At the absolute minimum get rid of the micros. Hover for more detail?
• Can we link to an author page? Yes!
• Atom feed should have a feed icon~~
• Why are we adding another <link rel="alternate"> to this page? We already have one for recent changes on every page in the site and the one we're adding has a less descriptive title. Which is correct? Use that one as a feed for every page.

This needs to be thought out ...

• Requires Package Read to not use the side-bar
• Need to sync params to a package object that is taken out of the session so that it doesn't go into the database.

Strapline (?): Sharing, discovery and reuse of data and content

Need to consult on this.

Cost: 1h

Broken out from ticket:216

Integrate new test upgrade into testmigrate stuff.

New "migrate.ini" config file programmed with the db to be used for migrate tests and a param saying where the db dump is.

4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package.

​http://ckan.net/group/publicdomain has:

"There is 16 package in this group. There are 16 packages in this group."

Problem here is that foreign key then becomes "complicated" (composite).

• Could also deprecate continuity_id field in favour of the basic foreign key on ie

"Alternative formats". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f).

• How do we know if RDF available? Ans: see details in ticket:90.

Cost: 1h

Search of package name and title (and other fields) using regular expressions.

Current example use-case: Wanting to specify packages with title beginning with 'B'.

Issues:

• Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields?

Implementation:

• Postgres reg ex searching detailed here:

​http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE

WUI:

• View: show both sides of the relationship (but think carefully -- e.g. a given package may have *many* dependents ...)

Add uuids into package in REST interface.

Currently use old "bullet-point" style. Should change to now standard style used on e.g. package search results page.

Cost: 0.5h

Need a way to mark packages as duplicate.

Ideas/Questions?:

• Could work off a tag (meta-duplicate or somesuch)
• What happens if we want to delete the package (to we leave a place-holder, do we merge content?)

Revision.packages should not include all changes so why are we getting revisions with no associated changes? See e.g. ​http://ckan.net/revision/read/814e39e1-2540-4449-90d1-ac9a92ab40fb?

Investigate and fix.

NB: if this is bugfix this should go against stable or metastable branch if at all possible and then be merged into default.

Cost: 1.5h (?)

As a user I want to track changes to a particular package.

Provide an Atom feed, linked from the package read page.

There is code to create packages in-process (ckan/lib/importer) and via HTTP (ckanclient/loaders/base). There is also code to read different kinds of package source (Google Spreadsheets spreadsheet, other spreadsheet).

Propose to refactor all this to have template method in base command class (subclasses may implement command line interface, graphical user interface, or Web interface), where the template method calls on aggregated strategy objects for: reading raw spreadsheet data from spreadsheet application; for inferring CKAN package entities from the raw spreadsheet data; and for writing package entities to CKAN service.

Spreadsheet readers to be factored to allow for variations of spreadsheet application.

Raw data inferrers to allow for variations in spreadsheet cell structure.

CKAN writer strategies to allow for either CKAN client or direct linking.

Actual cases then supported by specifying existing strategies (or defining new ones) in a script that invokes the command class.

I want to be able to easily check when a package is added (or removed?) from a given tag or groups. To do this I would like access to an Atom/RSS feed for a given tag or group.

• This would allow people to embed feed widgets in other places

At the moment only support a very limited set of extra fields (see docs for details). Should support arbitrary extra fields (note no need to put these in special full text index though ...)

Cost: 1h

Create a new package with a name 'blah' and resource format 'blah'. Hit preview. There is an error because of the lack of resource url, but in the resource input boxes, there are now four resources with format 'blah'.

With forms configurable to show a subset of licenses and ckan instances moving packages from server to server, there is a chance that a user will edit a package and the existing package's license is not displayed in the form. If the user submits this form without touching the license dropdown, then the value is lost.

So for this case we should add the package's license to the list displayed.