Look for all places where model is accessed and check authorization is checked.

Document holes (and, as necessary, suggestions for fixes) as new tickets. Likely areas that need looking at:

• search i/f
• package WUI commit

Write holes are obviously much more significant to us than read holes.

Split out from ticket:43

• group form: description field for needs to be a text area
• group form: make title and name field wider
• package view: need to move tags and license above notes

As a

User

I want to

Create a new group via the WUI

Details

• locate at /group/new/
• You must be logged in to create a group
• Group creator is automatically given role of group admin
• Edit screen is same as edit screen ...

Using REST interface you can list packages and groups without authorization being checked.

Can be fixed using more advanced query to check authz.

A user's 'home page' is at: user/<user.id> e.g. user/28394723982-03849472

Step 1:

• Current user home page with recent edits

Step 2: Readonly

• Number of edits
• Number of Packages they are admin of

How do we do a nicer URL for the home page - can we use their openid login?

Follow on tickets: ticket:138 and ticket:142

Write up search api documentation and put it in a template that shows up at api/search/.

Cost: 2h

When viewing a group you see a list of the users who have admin rights to it. (Akin to owners / moderators)

Follows on from ticket:148. Sister to ticket:150.

Package edit page has fields for extra key-value pairs. Don't use formalchemy. The form displays the existing ones and new ones as follows:

Key           Value            delete
[ country   ] [ uk, spain   ]  [x]

When presenting the form, existing pairs and 3 blank pairs are provided.

When saving the data, keys that are not changed use the existing PackageExtra object (perhaps with new value). Deleted keys put the PackageExtra state to deleted. For new keys, only create a new PackageExtra if there isn't a state=deleted one to resurrect.

Validators ensure no repeated keys.

Log message on package edit page is no longer persisted through a preview (no doubt due to recent form upgrades). Let's fix this.

cost: 0.5h

Search of form: tags: ... behaves differently depending on whether there is a leading space:

• tags: postcode - tags found correctly but no packages found
• tags:postcode - tags incorrectly found but correct packages found

Let's fix this.

Cost: 0.5h

E.g. ​http://ckan.net/tag/read/postcodes this has no-postcode listed but when you look at that package 'postcodes' isn't a tag ...

I suspect this may be to do with search using deleted tags as I think no-postcode had postcodes as a tag at some point earlier ...

Related to: ticket:90 Link to RDF version of CKAN data on Talis Connected Commons ticket:164 Creating RDF data from CKAN

Regularly re-upload the CKAN RDF data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

Ensure logging is working on server.

e.g.

​http://www.ckan.net/package/search?q=science&search=Search+Packages+%C2%BB

Results are capped at 20, but can't see past first page.

Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible.

NB: improving the documentation is another matter (as is integrating e.g. existing api docs).

Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net):

• ​http://knowledgeforge.net/ckan/ckan/doc/
• (Other options: ​http://www.ckan.net/doc/ ​http://doc.ckan.net)

the "only openly licensed packages" tick box seems to return blank page every time - as do queries for only openly licensed and only downloadable packages. The "only downloadable packages" tick box by itself seems to work fine. E.g. try searching for 'science'.

Raised by Jonathan Gray

/tool New 'tools' section of the WUI with 'Excel package import' page.

/tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:

• license - name of license
• tags - space separated
• groups - space separated
• anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups.

Tick and cross aren't clear enough. Propose:

• Icon for downloadability (nothing if absent ...(?))
• Icon for open license? (Need something if closed ...)

Cost: 2h (?)

Standard search should search notes field in addition to name, title and tags (discussed in ticket:108 but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results.

If we do this we need:

1. To weight across fields in a sensible way
2. We can also use proper text search on title or ...

Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: ​http://www.postgresql.org/docs/8.3/static/textsearch.html

Using this with sqlalchemy: ​http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/

Issues with fulltext search:

• tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking.
• if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better.
• can we split the name on dash or underscore before being indexed?
• natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'.
• do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres?
• we want to weight name and title higher than other fields - achievable with custom trigger.

Cost 7 days

When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API.

The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user.

Example at bottom of package page:

Leave a comment:

Comments:

Implementation details:

Comments table is with columns:

Cost - 2 days

The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09).

Example part of package:

Date: 5/10/09
or
Date: 1/1/09 – 1/4/09

Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name.

Cost: 2h

As a

admin / user

I want to

Admin: export all HEAD data to JSON / CSV / RDF format dump.

User: wants to get hold of this dump.

Details

• Just do JSON & CSV for now.
• Put this in CLI.
• Just do list of packages - as_dict info. CSV needs a bit of flattening.

Cost: 2h

As of 80:099bdc5b07a6 datapkg has 4 test fails and 1 test error.

There is no documentation on how to run them, so I am using 'python setup.py test'.

datapkg.tests.test_cli.TestCLI.test_ckan ... FAIL
datapkg.tests.test_cli.TestCLI.test_walkthrough ... FAIL
datapkg.tests.test_index.TestDbIndex.test_get ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/index.py:97: SADeprecationWarning: Use session.add()
  self.session.update(pkg)
FAIL
datapkg.tests.test_index.TestDbIndex.test_get_when_loaded_as_new_and_init_not_called ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/tests/test_index.py:57: SADeprecationWarning: Use session.expunge_all()
  self.index.session.clear()
FAIL

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: ​http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: ​http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

1. i18n templates
   • Extract content using Babel
   • Get genshi translation working
   • Do the translation!
2. i18n all strings in controllers (if any)
3. i18n forms

As a

admin

I want to

input government data in a constrained way

so that

you get consistency in the db which is easily searched, discovered and automatically linked.

implementation

Form has fields suitable for government data. (related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)

Simple fields

• all basic ckan fields
• default license to OKD::Crown Copyright
• co_id, precision, taxonomy url, agency

Date fields

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

• date_released

• date_updated

• temporal_coverage_from, temporal_coverage_to - two date fields combined

Strict selection

Drop down combo

• department

Selection or free text

Editable combo

• update_frequency - selections: never, weekly, monthly, annually, discontinued

• geographic_granularity - selections: national, regional, local authority, ward, point

• temporal_granularity - selections: yes, months, weeks, days, hours, point

Specialised fields

• geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

• national_statisic - UI is check box, store as 'Yes' or 'No'

• resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.

• categories - multi values from strict selection of ONS categories. Stored as comma-separated.

Field data processed

• tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.

Future - optional

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

Remove the current dependency on Yahoo YUI toolkit for doing tag autocompletion on package edit forms.

The (KForge-derived) behaviour files for CKAN can easily be switched to the new jQuery-based ones: see ticket:212. Once this is done, it would make sense to implement the tag-selection code with jQuery.

Relevant pages include:

• ​http://bassistance.de/jquery-plugins/jquery-plugin-autocomplete/
• e.g. ​http://ckan.net/package/edit/acawiki

p:wui 2009-09-09 validate email address when creating an enquiry

fabric script to automate ckan deployment

• 'Login via OpenID' -> 'Login with OpenID'?
• Elements:
  • Search
  • about CKAN
  • 3 column Packages view (see ticket:219)
  • Small tag cloud (RHS)

Also, what about ticket:143 ?

Package search results page package/search

• need clearer delineation between search entry form and results.
• 0 results needs to be clearer, and the alternative suggestion: 'would you like to create a new package' needs to not look like a search result.
• "N tags found" -- why do I care? this should be presented as "filter results by tag" -- very unlikely that I'm looking for a tag: I'm looking for a package, but tags might help me narrow down my search.
• tags should be displayed more like the way they're displayed when you add them to a package (i.e. blue 'tag' background)
• filter out particular tags
• tag cloud?
• copywriting: "Packages - Search" -> "Search packages"

Package page package/mypackagename

• Design
  • In general: a huge amount of visual and typographic noise: "Package:" is redundant.
  • "{edit}", "{history}" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is "make an enquiry with Is It Open?" bigger than these?
  • Openness/downloadability icons: need help, as discussed earlier
  • "Rating:" redundant, as is the second set of stars -- why can't both the display and rate widget
    • Rating - alternative to GET on set_rating.
  • "Metadata:" technospeak, probably redundant.
  • <li> bullets add nothing.
  • Title, Version, URL can probably go in one line.
  • Resources can be made more prominent.
  • Author, author email and Maintainer, maintainer email can be merged.
  • Groups, tags in little boxes?
  • Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface?
  • Don't show extras unless there are any.

Package history page package/history/mypackagename

• "Revisions" is redundant
• "Compare" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it!
• Ditto to overall "Recent Changes" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links.

Groups landing page group

• Yeesh! What's up with the typography for "Just login..."?
• Truncated description along with group name.
• Explanation text about what groups are for (RHS)
• Another case of <li> bullets adding nothing.

• There is huge inconsistency in the titles of pages across the site: to give one example "Edit Package:" vs "Package: mypkg [not linked]" vs "History - mypkg [linked to package page]"
  • Normalize and use terminology: "Data Package" - also change in navbar (but nowhere else for the time being). So hvae Data Package - mypkg, Data Package - mypkg - Edit etc
• pagination is currently lost in amongst the results list. needs to be *much* more obvious, and should appear top and bottom, or at the very least at the bottom!

• Requires Package Read to not use the side-bar
• Need to sync params to a package object that is taken out of the session so that it doesn't go into the database.

Options:

1. Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works)

1. Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist).

Can do this using:

<xi:include href="base.html"><xi:fallback /></xi:include>

Currently it is possible (v0.11) to have a PackageResource? object without a url. This should be impossible to do (either at creation time or via modification).

Dumping ground for various minor and miscellaneous items (mainly refactorings):rgrp

• controllers/tag.py: why does this not use lib/search.py (but controllers/packages.py search method does)?

done:

• lib/search.py: inheritance would be nicer than switching on entity in search (i.e. have SearchPackage?, SearchTag? etc)

"Alternative formats". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f).

• How do we know if RDF available? Ans: see details in ticket:90.

Cost: 1h

Want to have:

• /api/search/revision/

Support querying by "since time" and "since revision". Each individual revision should be available in the REST API (read-only):

• /api/rest/revision/{revision ID}
  • should include revision attributes plus revision.packages e.g. like { "id": "...", ... "packages": [ list of packages names from e.g. revision.packages ] }

WUI:

• View: show both sides of the relationship (but think carefully -- e.g. a given package may have *many* dependents ...)

Add uuids into package in REST interface.

Currently use old "bullet-point" style. Should change to now standard style used on e.g. package search results page.

Cost: 0.5h

Need a way to mark packages as duplicate.

Ideas/Questions?:

• Could work off a tag (meta-duplicate or somesuch)
• What happens if we want to delete the package (to we leave a place-holder, do we merge content?)