CKAN: Ticket Query

#21: Undelete a package

johnbywater — Mon, 10 Jul 2006 17:02:16 GMT

#77: Rate packages

rgrp — Fri, 17 Jul 2009 11:13:13 GMT

As a User or Visitor

I wish to rate a package in some way. On the package view, below Openness, there is a 'Rating' heading with the star rating which is selectable.

Rating out of 5. One user or IP, one vote.

package_rating table:

#88: Download (link) metadata

rgrp — Mon, 27 Jul 2009 11:16:25 GMT

Support for download metadata (file size, file formats, version information).

See also: ticket:87 (multiple download links)

#90: Link to RDF versions of CKAN data

rgrp — Thu, 30 Jul 2009 08:49:02 GMT

We should link to representations on semantic.ckan.net

Each package page could link to its RDF representation
- "Alternative formats" near star rating. Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f).
- Some instances of ckan won't need this, so have option in ini file for this. rdf_store_url=xyz
- Include machine-readable link to the RDF:
```
<meta http-equiv=... />
```
- 303 redirect to RDF when content negotiation requires RDF. Test:
```
curl -I -H "Accept: application/rdf+xml" http://localhost:5000/package/warandpeace
```

Cost: 3h

Future

Search page to link to SPARQL interface: http://semantic.ckan.net/sparql
License/dump page could link to full RDF dump.

#113: Simple script to test data migration on a production db

dread — Mon, 07 Sep 2009 09:42:03 GMT

Must not alter the db.

Related to ticket:112.

#128: Add ckan_url attribute to REST JSON representation of a Package

rgrp — Fri, 25 Sep 2009 08:18:33 GMT

Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN.

Cost: 30m

#134: admin interface is only available to sysadmins

dread — Mon, 05 Oct 2009 09:35:14 GMT

#135: sysadmins defined in the db

dread — Mon, 05 Oct 2009 09:36:08 GMT

#136: User has publicly viewable page

dread — Mon, 05 Oct 2009 11:20:50 GMT

A user's 'home page' is at: user/<user.id> e.g. user/28394723982-03849472

Step 1:

Current user home page with recent edits

Step 2: Readonly

Number of edits
Number of Packages they are admin of

How do we do a nicer URL for the home page - can we use their openid login?

Follow on tickets: ticket:138 and ticket:142

#141: Search api docs

rgrp — Wed, 07 Oct 2009 08:10:08 GMT

Write up search api documentation and put it in a template that shows up at api/search/.

Cost: 2h

#145: Group shows its administrators

dread — Thu, 08 Oct 2009 14:00:03 GMT

When viewing a group you see a list of the users who have admin rights to it. (Akin to owners / moderators)

#148: Parser and loader for esw.org data

dread — Tue, 13 Oct 2009 13:31:36 GMT

parse like data4nr code

Follow up tickets: ticket:149 and ticket:150

#149: Loader for esw data recognises existing packages

dread — Tue, 13 Oct 2009 13:32:31 GMT

Follows on from ticket:148. Sister to ticket:150.

#150: Loader for esw data uses REST API

dread — Tue, 13 Oct 2009 13:33:48 GMT

(instead of sqlalchemy interface, to facilitate general usage of the loader)

use ckanclient

Follows on from ticket:148 and ticket:149

#152: Package has editable 'Extra' fields

dread — Thu, 15 Oct 2009 14:01:56 GMT

Package edit page has fields for extra key-value pairs. Don't use formalchemy. The form displays the existing ones and new ones as follows:

Key           Value            delete
[ country   ] [ uk, spain   ]  [x]

When presenting the form, existing pairs and 3 blank pairs are provided.

When saving the data, keys that are not changed use the existing PackageExtra object (perhaps with new value). Deleted keys put the PackageExtra state to deleted. For new keys, only create a new PackageExtra if there isn't a state=deleted one to resurrect.

Validators ensure no repeated keys.

#157: Log message not persisted through preview

rgrp — Fri, 16 Oct 2009 11:59:56 GMT

Log message on package edit page is no longer persisted through a preview (no doubt due to recent form upgrades). Let's fix this.

cost: 0.5h

#158: Add 'extras' to REST API

dread — Fri, 16 Oct 2009 14:27:39 GMT

Ensure you can add/change/delete extra fields from packages using the REST API.
Update API docs to cover this.

On Package get, ensure separation of extra attributes from other attributes.

What is the meaning of posting an incomplete dictionary (updating a package)? Option 1 - if field not present, leave field alone

Delete field if field value is None

Option 2 - user must supply all fields

Delete field if field absent

Prefer option 1. Document this.

Follows on from ticket:152

#159: Searching for tags:... resulting in lots of tags being found

rgrp — Tue, 20 Oct 2009 09:14:57 GMT

Search of form: tags: ... behaves differently depending on whether there is a leading space:

tags: postcode - tags found correctly but no packages found
tags:postcode - tags incorrectly found but correct packages found

Let's fix this.

Cost: 0.5h

#160: Implement internal links in CKAN notes markup

rgrp — Tue, 20 Oct 2009 09:25:51 GMT

Suggest form is: key:value (with optional space after ':'.

Keys would be for an object, i.e. 'package', 'tag', 'group'.

Becomes a link.

This would be implemented by implementing a 'Parser' or 'Processor' object which is run before markdown is done. Regex replace.

Cost: 4h

#161: Tag listing is incorrect

rgrp — Tue, 20 Oct 2009 09:28:45 GMT

E.g. http://ckan.net/tag/read/postcodes this has no-postcode listed but when you look at that package 'postcodes' isn't a tag ...

I suspect this may be to do with search using deleted tags as I think no-postcode had postcodes as a tag at some point earlier ...

#162: Tags should be automatially lowere-cased if not already

rgrp — Tue, 20 Oct 2009 14:17:49 GMT

Rather than raising a validation error automatically lower-case tags on submission.

Cost: 0.5h

#163: Regularly put CKAN RDF on Talis CC

dread — Tue, 20 Oct 2009 15:29:47 GMT

Related to: ticket:90 Link to RDF version of CKAN data on Talis Connected Commons ticket:164 Creating RDF data from CKAN

Regularly re-upload the CKAN RDF data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

#166: Add extras editing to ckanclient

dread — Thu, 22 Oct 2009 09:26:21 GMT

Ensure logging is working on server.

#167: REST interface you can't update multiple tags

dread — Thu, 22 Oct 2009 11:13:13 GMT

Only first tag is updated.

#170: Paginate search results

dread — Fri, 23 Oct 2009 15:59:44 GMT

e.g.

http://www.ckan.net/package/search?q=science&search=Search+Packages+%C2%BB

Results are capped at 20, but can't see past first page.

#172: Build ckan documentation using sphinx and upload

rgrp — Sun, 25 Oct 2009 16:43:39 GMT

Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible.

NB: improving the documentation is another matter (as is integrating e.g. existing api docs).

Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net):

http://knowledgeforge.net/ckan/ckan/doc/
(Other options: http://www.ckan.net/doc/ http://doc.ckan.net)

#173: Diffing WUI

dread — Mon, 26 Oct 2009 13:56:52 GMT

Use vdm diff functionality (see ticket:82) to implement diffing of packages between versions in WUI.

In package history you should be able to select to revisions and show the diff.

Estimated cost: 12h

#174: Packages search broken for 'only openly licensed'

dread — Tue, 27 Oct 2009 13:41:07 GMT

the "only openly licensed packages" tick box seems to return blank page every time - as do queries for only openly licensed and only downloadable packages. The "only downloadable packages" tick box by itself seems to work fine. E.g. try searching for 'science'.

Raised by Jonathan Gray

#175: Creation date

dread — Wed, 28 Oct 2009 18:21:10 GMT

'created' attribute on User, Group and Rating

datetime object: default=datetime.datetime.now()

See example on revision

Requires migrate script

#178: Excel package importer

dread — Tue, 03 Nov 2009 10:27:14 GMT

/tool New 'tools' section of the WUI with 'Excel package import' page.

/tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:

license - name of license
tags - space separated
groups - space separated
anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups.

#187: Full-text search

rgrp — Tue, 10 Nov 2009 16:19:10 GMT

Standard search should search notes field in addition to name, title and tags (discussed in ticket:108 but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results.

If we do this we need:

To weight across fields in a sensible way
We can also use proper text search on title or ...

Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: http://www.postgresql.org/docs/8.3/static/textsearch.html

Using this with sqlalchemy: http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/

Issues with fulltext search:

tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking.
if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better.
can we split the name on dash or underscore before being indexed?
natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'.
do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres?
we want to weight name and title higher than other fields - achievable with custom trigger.

#189: Multiple download urls

dread — Mon, 16 Nov 2009 15:42:00 GMT

See also: PackageResources which defines a 'distribution' as a package's payload (perhaps with metadata). In our case, at least within CKAN, the representation of the distribution will just be the reference to get it (plus some minor metadata). We are expanding this functionality to be any kind of 'resource' associated with a package since it is likely in some cases people will want to link to a file containing a part, but not the whole, of a package's data.

As a

User (package editor and package user)

I want to

(package editor) Associate multiple distributions with a package.

(package user) Easily have access to multiple distributions of a package should they exist.

Implementation

There is an object called 'Resource' with the following attributes:

url (text) (required in UI, but not in db table)
format (text)
description (text)
package (association) - a resource is always associated with 1 package

A package's list of resources will have an order which is preserved. The first in the list will be known as the 'default resource' (the presumption will always be that this is a 'distribution').

How to replace download url

download_url removed and replaced as a proxy attribute. Read returns the url of the default distribution. Write changes the url of the default resource. Ensures backward compatibility.

REST API has new attribute in package dictionary 'resouces' - an ordered list.

In WUI, replace download url in package read with a table of the distributions.

In the WUI package edit, be able to add/remove/edit/reorder distributions.

Example

Example part of package:

Distributions:

http://some.com/pollution.pdf	PDF	Original report
http://some.com/pollution.csv	csv	Original data
http://some.com/pollution	sparql	Linked version of data

Existing data to be migrated to new structure.

Cost: 4 days

#194: Check star ratings aren't influenced by search engine crawlers

dread — Tue, 17 Nov 2009 15:25:12 GMT

rel=nofollow or robots.txt ?

#195: Association listings should be alphabetically sorted (e.g. a group's packages)

rgrp — Tue, 17 Nov 2009 16:30:48 GMT

Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name.

Cost: 2h

#197: Simple dumper

dread — Mon, 23 Nov 2009 10:34:55 GMT

As a

admin / user

I want to

Admin: export all HEAD data to JSON / CSV / RDF format dump.

User: wants to get hold of this dump.

Details

Just do JSON & CSV for now.
Put this in CLI.
Just do list of packages - as_dict info. CSV needs a bit of flattening.

Cost: 2h

#202: Internationalization (i18n) try-out

dread — Mon, 07 Dec 2009 09:46:32 GMT

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

i18n templates
- Extract content using Babel
- Get genshi translation working
- Do the translation!
i18n all strings in controllers (if any)
i18n forms

#203: show package counts for groups in WUI

dread — Mon, 07 Dec 2009 12:08:10 GMT

In two places in the WUI

list of groups
group page in title e.g. ukgov (5)

Cost: 1h

#204: Custom package input form

dread — Mon, 07 Dec 2009 12:29:25 GMT

As a

admin

I want to

input government data in a constrained way

so that

you get consistency in the db which is easily searched, discovered and automatically linked.

implementation

Form has fields suitable for government data. (related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)

Simple fields

all basic ckan fields
default license to OKD::Crown Copyright
co_id, precision, taxonomy url, agency

Date fields

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

date_released

date_updated

temporal_coverage_from, temporal_coverage_to - two date fields combined

Strict selection

Drop down combo

department

Selection or free text

Editable combo

update_frequency - selections: never, weekly, monthly, annually, discontinued

geographic_granularity - selections: national, regional, local authority, ward, point

temporal_granularity - selections: yes, months, weeks, days, hours, point

Specialised fields

geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

national_statisic - UI is check box, store as 'Yes' or 'No'

resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.

categories - multi values from strict selection of ONS categories. Stored as comma-separated.

Field data processed

tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.

Future - optional

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

#208: Obfuscate author email and maintainer email field

rgrp — Mon, 07 Dec 2009 16:38:58 GMT

Obfuscate author email and maintainer email field.

Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@…', encode='javascript')

Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate?

#212: Switch to using m.okfn.org "kforge" media (css and behaviour)

rgrp — Sun, 13 Dec 2009 11:28:48 GMT

We are already using standard kforge css though copied locally and with a few modifications.

We should switch to using common copy on m.okfn.org. Some work will be required to factor out any changes we have made.

Cost: 2h

#213: Automate CKAN deployment

dread — Mon, 14 Dec 2009 17:28:27 GMT

fabric script to automate ckan deployment

#214: UI Review - General template

dread — Tue, 15 Dec 2009 12:19:41 GMT

Set hover names on navigation links.
Do we really need to display "About CKAN" everywhere? Just on home page.
Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions.

#217: UI Review - Search results

dread — Tue, 15 Dec 2009 12:24:30 GMT

Package search results page package/search

need clearer delineation between search entry form and results.
0 results needs to be clearer, and the alternative suggestion: 'would you like to create a new package' needs to not look like a search result.
"N tags found" -- why do I care? this should be presented as "filter results by tag" -- very unlikely that I'm looking for a tag: I'm looking for a package, but tags might help me narrow down my search.
tags should be displayed more like the way they're displayed when you add them to a package (i.e. blue 'tag' background)
filter out particular tags
tag cloud?
copywriting: "Packages - Search" -> "Search packages"

#218: UI Review - Package listing

dread — Tue, 15 Dec 2009 12:25:03 GMT

openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS.
maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence.
results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise.
maybe links should go on the full titles? Get rid of package names.
Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS?
same comments as search listing (ticket:217)

#220: UI Review - Package read

dread — Tue, 15 Dec 2009 12:28:56 GMT

Package page package/mypackagename

Design
- In general: a huge amount of visual and typographic noise: "Package:" is redundant.
- "{edit}", "{history}" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is "make an enquiry with Is It Open?" bigger than these?
- Openness/downloadability icons: need help, as discussed earlier
- "Rating:" redundant, as is the second set of stars -- why can't both the display and rate widget
  - Rating - alternative to GET on set_rating.
- "Metadata:" technospeak, probably redundant.
- <li> bullets add nothing.
- Title, Version, URL can probably go in one line.
- Resources can be made more prominent.
- Author, author email and Maintainer, maintainer email can be merged.
- Groups, tags in little boxes?
- Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface?
- Don't show extras unless there are any.

#224: UI Review - Groups

dread — Tue, 15 Dec 2009 12:32:40 GMT

Groups landing page group

Yeesh! What's up with the typography for "Just login..."?
Truncated description along with group name.
Explanation text about what groups are for (RHS)
Another case of <li> bullets adding nothing.

#229: Resource hashes

dread — Tue, 05 Jan 2010 10:11:27 GMT

New field for resources - hash of the resource file.

CKAN itself will not calculate the hash value - user just pastes it in.
Display text field in resource table.
Requires migration script.

Questions for the field's value:

Which hash to use? Restrict to python hashlib and other major hash libraries.

Should we use merkle trees?

Thanks to Julien D'Assanges for the suggestion.

#112: Test db schema created via migration scripts

dread — Mon, 07 Sep 2009 09:41:35 GMT

'paster db upgrade' should result in the same database as 'paster db create' on a clean system. In particular, all tests should pass.

Note: equivalent of paster db upgrade isn't being run from tests at the moment (we haven't been able to get this to work).

#154: Package form guidance

dread — Thu, 15 Oct 2009 15:46:12 GMT

Give guidance on what to put in 'author' and 'maintainer' fields (e.g. name? URL?)

Suggestion from Jonathan Gray

#144: Most popular packages listed on homepage

dread — Thu, 08 Oct 2009 13:59:51 GMT

Based on number of views.

Related to ticket:143.

#147: Parser and loader for esw.org data

dread — Tue, 13 Oct 2009 13:31:35 GMT

#151: User object should have a created attribute

rgrp — Thu, 15 Oct 2009 06:54:54 GMT

User object should have a "created" attribute initialized to current datetime.

Require a db migration but o/w very simple.

Cost: 1.5h

#153: Group's packages listed alphabetically

dread — Thu, 15 Oct 2009 15:45:15 GMT

This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there.

Suggested by Jonathan Gray

#180: Tag cloud as way to view CKAN tags

jwyg — Fri, 06 Nov 2009 19:04:14 GMT

Create big tag cloud with all CKAN tags - perhaps weighting with size and colour...

#188: Improve package listing views

rgrp — Tue, 10 Nov 2009 16:20:31 GMT

Propose change to tabular-like format showing these attributes (perhaps should be configurable?)

Openness status
Title (not sure name is needed)
Tags

Cost: 4h

#139: Ensure you cannot add same tag twice to a package

rgrp — Mon, 05 Oct 2009 18:49:12 GMT

Seems some package have ended up with same tag twice: http://www.ckan.net/package/fred2dot0

Check whether this is still possible and if it is fix it ...

Cost: 2h