This ticket depends on #2423, #2424, #2425, #2426 and #2427.

• Try to remove as many of CKAN's dependencies as possible

• Try to update as many of the remaining ones as possible

• Try to get rid of some dependencies that are fetched from random hg or git servers instead of pypi

• Then try to get the order of deps in the requirements files right so that you end up with the right versions of everything

• Each module in the dependencies file should specify the exact version, to protect us from changes due to them upgrading versions under us and breaking things or changing their own deps

• Consider shipping dependencies with ckan instead of pip installing them?

• This is related to moving CKAN to Ubuntu 12.04

• Add a test case that runs pip freeze and tests that all the right versions of deps are installed?

Reported in IRC by @floapps

In previous ckan versions i could use paster commands to remove reader rights from visitors on all packages and this would then throw the user to login screen when trying to see a dataset but, when i did this in ckan 1.7, i could still see all datasets as a visitor

Using ​http://docs.ckan.org/en/latest/authorization.html#permissions-publisher-mode

paster rights remove visitor reader package:all

Also has an empty list of roles in config for default visitor roles

I tested on my local install and could replicate with that command doing nothing. Neither affects existing or newly created datasets.

create a depreciated decorator to wrap functions we intend to remove

logs a warning maybe check that 'depreciated' is included in the docstring too

Currently facets in the new demo site disappear from the sidebar once they are selected.

​http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa

Instead they should be marked with an "active" class and clicking the link should remove the filter in the same way as clicking the cross in a token below the search field works.

Currently we have far too many types that are essentially the same format. The new demo theme is using icons for common types. So we need to normalise the common formats into pretty strings. e.g. application/json, JSON, .json and json all should be output as “json”

See the following Basecamp thread for UI examples and discussion.

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056

Core CKAN currently has a dependency on LXML and it would be nice if we could remove it (although this *may* be indirectly through genshi).

When attempting to add more than one resource at once, if more than one resource invalidates, this results in a js error, leaving the form in an inconsistent state.

Repro:

1. Go to /dataset/new
2. Add a new resource. Fill in one of the fields with an invalid value. eg - last_modified, or size...
3. Add another resource, doing the same thing: make one of the fields invalid.
4. Try to save the dataset.
5. The entered resource information will be lost, and a js error "Uncaught Error: Can't add the same model to a set twice,: backbone.js:586" will be thrown.

Taken from ​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002218.html

A few months ago, the RDF provided when dereferencing a dataset's URI was very useful. It described the datasets using many of the "right" RDF vocabularies (dcterms, void, etc).

The RDF that is returned now no longer has these nice descriptions. Instead, it seems to be a mindless transcription of the CKAN record.

For example, the older descriptions gave:

<​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>

void:subset [

void:target <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>, <​http://thedatahub.org/dataset/geonames-semantic-web> ; void:triples 50 ; a void:Linkset

], [

void:target <​http://thedatahub.org/dataset/dbpedia>, <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states> ; void:triples 52 ; a void:Linkset

], [

void:target <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>, <​http://thedatahub.org/dataset/govtrack> ; void:triples 52 ; a void:Linkset

] ; void:triples 130005 ;

but now it's simply:

<​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>

dct:relation [

rdf:value "130005" ; rdfs:label "triples"

], [

rdf:value "52" ; rdfs:label "links:dbpedia"

], [

rdf:value "50" ; rdfs:label "links:geonames-semantic-web"

], [

rdf:value "52" ; rdfs:label "links:govtrack"

] ;

My old copies of the dataset descriptions mentioned a "GoLD CKAN 1.0" that was credited for the nicer modeling.

Where did that go? Can we get it back?

Thanks, Tim Lebo

<​http://semantic.ckan.net/record/91d2c0de-75a4-4bb6-b260-bc2946e1be8b>

dc:source <​http://ckan.net/> ; opmv:wasGeneratedBy [

opmv:used <​http://ckan.net/api/rest/package/farmers-markets-geographic-data-united-states>, <​http://ckan.net/api/rest/revision/9c2545b8-ebf1-42bf-ac92-633ff5ceba3f> ; opmv:wasControlledBy [

a foaf:Agent ; foaf:name "GoLD CKAN 1.0"

] ; opmv:wasPerformedAt [

a time:Instant ; time:inXSDDateTime "2012-01-31T05:34:32Z"^^xsd:dateTime

] ; a opmv:Process

] ; a dcat:CatalogRecord ; dcat:dataset <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states> ;

…

The settings > extras page for a dataset allows extra fields to be added, but only four can be added a time in a single atomic update. It would be helpful to have an 'add another' button on the form that can be used any number of times.

There are a few errors or unclarities in the source install doc - see

​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html ​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html

As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere

Implementation: (?) Use phantomjs or something like ​http://snapito.com/ Investigate for 1.8, will be needed for TDH Oct.

Search results are slow as we do many package_dictize. Store the package_dict in the search index so that we do not have to hit the database for each of them.

One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed

"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

We need to separate the CLI part of CreateTestData? from the bit which creates test fixtures. This is because of the problem with imports and logging. Otherwise we have to do unpythonic things to get logging to work when deriving from CreateTestData? from an extension such as ecportal / dgu. I've only introduced logging into CreateTestData? but I think it is a good thing to have. This change doesn't cause breakages

An extra field on datasets called "metadata_language". The language that the meta-data is in. Drop-down list needs to be translatable. Default value is "English".

It would be useful (well for DGU, but no doubt for core too) to have be able to create multiple users with full properties using one commit. Also to be able to create rights (UserObjectRoles?).

• Only logged in users able to create
• Creation permissions have only creators able to update a dataset
• Synchronous search is enabled (#1678)

The command line tool 'rights' is quite handy but it is glued to the CLI. I'm going to factor out the bit which searches for objects etc so it can be used by CreateTestData? etc and will be used by DGU.

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)

Provide a new test install of DataGM based on the ODS demo site being completed through June.

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( ​http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

This needs to be fixed in the CKAN templates ("no-sidebar" should not be marked for translation) but for 1.7 just fixed it in the translation files.

Currently in the logic function tag_search you can specify limit and offset, but no count is returned. Therefore pagination is not possible for tag results. This is desired though.

This should cover formatting and best practices for HTML, CSS and JavaScript?

Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch.

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

Solr install instructions install the wrong version

1.7 will hide the link on the footer and put a deprecation notice on the related pages, but they should be completely removed on 1.8.

Need to investigate alternative payment gateways

Improve documentation for plugins and add examples. Could use ckanext-example as a base.

I've been writing a doc with different issues regarding INSPIRE support in CKAN. It is mostly done, except for some sections on CSW support. Depends on #2359

Various stuff related with release 1.7:

• Review patches sent to the release branch
• Regularly update TDH with latest changes
• write changelog, announce email, etc
• ...

Try and increase the page load times for the front end.

Now

• Always include scripts at the bottom
• Include CSS at the top
• Load fonts with JS
• Concatenate and minify JavaScript? files
• Create a build script to load dependancies seperately
• Enable Gzip compression on resources
• Enable future cache headers on resources

Next

• Support serving assets from a CDN
• Look into optimising the CSS - Remove large portions of unused CSS
• Remove extreaneous libraries, can jQuery UI be replaced with bootstrap?

Tools

• Webkit Inspector Audits
• YSlow Firebug Plugin - ​http://yslow.org/
• Speed Tracer - ​https://chrome.google.com/webstore/detail/ognampngfcbddbfemdapefohjiobgbdl

Continuous Integration

YSlow can be hooked up to Jenkins via Phantom JS

When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data. Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to PackageController? subclasses to do form validation/processing.

Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override.

NB - form processing/validation *is* different to validating data. For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&Cs.

• [ ] Add hooks
• [ ] Use in ckanext-example

Start a Java library for interacting with the CKAN Action API.

Currently requires support for Groups, Tags, Resources, and Search.

An example app would be very useful.

Changes:

• tidy up activity feed on home page or replace with search results (by modified date)
• move ckan menu to a left sidebar (see Sam's template) (not doing this now)
• update templates with changes from master
• use local image in language dropdown (ecportal.js)
• update dataset read template

To discuss:

• should we rename 'organization' to 'publisher'? Requires updating all templates and the url mapping.

After checking out a commit to test from CKAN's GitHub?, Jenkins runs a script that creates a new virtual environment and installs CKAN and its dependencies into it, and does some other necessary tasks. Jenkins then runs the tests in this virtualenv.

The install script may have to change from one commit to the next as CKAN's install instructions change, so it would be good if the script was shipped in CKAN core. That way Jenkins will run different versions of the script depending on which commit it's testing and if the tests fail because the script is wrong then that's actually a bug that needs to be fixed in CKAN core. Also the CKAN install instructions could be simplified a lot by just having users run this script instead of doing each step by hand.

I won't be needed for this during the current sprint, closing.

Placeholder for thinking about this ...

• Allow users to configure info for CSV import (e.g. separators, quotes etc)

A place to list crucial fixes for v1.7 release:

Related extension:

All not complete now moved to #2347

• Related Item has comments on index page (they should not show up here -- why is this happening!)
• Related item should be to left of history (history could in fact move to RHS)
  • What about losing icons since we duplicate for related item
• Related item plus sign is very poorly aligned within box
  • Add related items should be in drop down like for resources (?) to make it easy to add items
  • Hover should show a title that gives some information about what related are (not at all clear)
• Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

Search results:

• #2331 - search should add not or
• Search results should be similar across group pages and normal search results (suggest adopting group page style)
  • Search results resource link should link to resource page rather than url for that resource
  • Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?)

Data viewer:

• Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) ​https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07
  • Ensure still using read-only (this seems to have been lost in upgrade)
• Do not show failed preview with red-bar (esp when just empty datastore ...)

a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy b) DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

• (? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state)
  • Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...)
  • Do #2322 ...
• Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...)
• Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
• Data viewer embed url includes all the resource info

Dataset page:

• Put home page link somewhere more prominent (e.g. the top of the sidebar)
• Put author more prominent (at top right)
• (? - new work) List CKAN owners at top right in sidebar
• Make resources stand out more
• Rename resources tab and heading to Data & Resources

Dataset edit and create:

• Corrections to dataset creation form for v1.7 - #2317
• Better auto-complete for groups on dataset edit/new page - #2319
  • Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?)
• Description / notes text area is not wide enough.
  • Should also be larger on edit page
• Problem with date instructions on add/edit resource: "Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d."
• Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either
  • A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar)
  • Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like "checking url and gathering url info ...)

Resource view:

• Do not show no description in no description (just show dataset info)

Theme:

• Fix spacing above page title (i.e. dataset title etc)
• Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer
• Fix subnav width on resources pages (does not extend all the way to the right like it should)
• Fix spacing of top menu and search box (do not look right)
• Group pages:
  • view page: image / logo is really poorly locate and sized, see e.g. ​http://datahub.io/group/wikimedia
  • view page: admin heading on RHS sidebar is wrongly indented

Miscellaneous:

• Ensure that we out-of-the-box config for CKAN has (#2388):
  • Only logged in users able to create
  • Creation permissions have only creators able to update a dataset

Deployments (without deployment we cannot know these are working):

• Social extension
• Analytics