Need to add way for extensions to trigger based on resource change and just resource url change.

Superticket: #1343

Creation of accounts from OpenID causes large number of problems:

• No guarantee of other profile info
• Poor username (just from openid)

We therefore will:

1. Require creation of user account via register (no auto-creation of accounts via openid)
   • sidebar of register page should point to login page rather than openid signin

2. Permit association of an openid with an account ...
   • How: paste in your openid url (requires change to edit form to have this)
     • won't work for google with their weird openid urls
   • [future] Nicer way is to have login via openid while logged in (which association then happening in background)

3. Migration (for 1+2):
   • (?) Generate decent usernames for all existing accounts
   • As this is nontrivial suggest instead we allow editing of usernames (by account owner and sysadmins). This is useful in its own right and is a reasonable 80/20 solution.
     • This does present the problem of re-associating commits with the new username. Best solution to this would be to switch revisions to point to userid rather than username. See #1534

4. [optional] Change display_name to always be username
   • Change My Account to Username at top right of all pages (once we know usernames are short ...)
     • could do this straight away by truncating long usernames (e.g. truncate at 20 chars ...)
     • Also should we lose the icon?

Now we don't use postgres search, all the tests involving search now don't need to be skipped when running on sqlite. Should help coders spot earlier if these tests break.

We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: ​http://ckan.okfnpad.org/queue (most relevant parts inlined here)

Preliminaries

• Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)

Configuration setup for daemons

Pass config through to workers i.e site_url, user, api_key. Need to make site user account. #1408

celeryd config:

All providers of tasks will add an item to the following entry point:

[ckan.tasks]
name = ckanext.{name}.tasks:....

celeryconfig.py

from pkg_resources import iter_entry_points
for entry in iter_entry_points:
     celeryimports.appen(....)

CELERY_IMPORTS = celeryimports

Work Items

• Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383
• Generate archiving request on resource url change [0.25d][0.25d] - #1399
• Make site user account.
• Make entry point system for celery config
• Archiver daemon #891
  1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
  2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
• Write to resource and task status table.[1d][0.75d]
• [Required?] Make archived data available in WUI - #892
• Documentation - #1400

Superticket: #1397

Generate archiving request on resource url change by implementing IResourceUrlChange and sending tasks to celery. [0.25d][0.25d]

Will be part of of ckanext-archiver extension.

Process:

1. A resource is added to CKAN
2. IResourceCreate event generated
3. IF: resource url points to ckan storage or falls within some other set of exclusion conditions then END else continue
4. Generate a Archiver.Update task with resource.id

Plan to migrate from mercurial git

• Convert from mercurial to git
  • Map named branches to normal branches
  • Also, if possible map usernames for RP: rgrp <​http://rufuspollock.org/> (mercurial) -> rgrp <rufus.pollock@…>
• Place at ​http://github.com/okfn/ckan (can delete current repo there as an out of date mirror)

Process:

• Do trial run
• Announce conversion date / time
• Require everyone to have pushed all outstanding changes at that time
• Do conversion
• Test
• Announce on list and ckan.org/

[Refactored] :: Follows on from #1506 UX changes.

• Declutter the sidebar. No yellow box.
• Facets to go on the left, rather than the right. More logical flow.
• Did you know you can search by author? Probably not. Find a nice way of presenting extended search options.
• Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results).
  • Update that look-and-feel to include the new resource icons created in #1506

Give site user system administrative rights.

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

I'd like us to display a one-off banner that prompts users who log into thedatahub.org to go and update & their profile & fill in their email address (if we could do it just for those who don't have an email address, then even better) That is, once they log in, they see a banner which says: Please update your profile [here] and add your email address, so you can receive notifications, be able to reset your password and get updates

Then when we build notifications or if we decide to survey people, we can actually contact them. Email address should be required as standard

DR: Also, mention their Full Name too - v. useful for selecting correct user in group curation

This is a scratch pad ticket with some comments on the current status of our ckan deb packages. I know that some of it is the deb packaging roadmap anyway, please forgive me if i mention them here again.

Rufus and me re-deployed some community ckan instances onto s022 (see ​http://trac.okfn.org/ticket/926). We followed the documentation ​http://docs.ckan.org/en/latest/install-from-package.html

• Deb package version number: the version of the deb package is "python-ckan 1309471251~149be76faabc+lucid-1", and it's hard to guess from there that it contains a ckan 1.4.2a
• When is 1.4.3/1.5.x expected as deb package?
• There was a bug in the DB upgrade script /usr/share/pyshared/ckan/migration/versions/029_version_groups.py (line 150) which looks like it was fixed 1.4.1==>1.4.2 but was nevertheless present in this deb package.
• The current script /usr/bin/ckan-std-install
  • does not set the Apache ServerName? according to the $INSTANCE variable
  • automatically configures a ckan extension named after $INSTANCE
  • depends on local postgres
  • could be replaced with "/usr/bin/ckan-deploy --name=ckan-std --domain=ckan-std.localhost (see next point)
• (i think this is exactly James' plan): have more generic deployment script /usr/bin/ckan-deploy as part of python-ckan which takes arguments like
  • --domain=cc.ckan.net
  • --aliases=$list-of-domains
  • --name=cc (defaults to "domain")
  • --no-db (does not configure a DB)
  • --sql-alchemy=$DB_CONFIG_STRING (also runs "paster --plugin ckan db upgrade --config")
  • --extension $list-of-extesions
  • ...

Currently can run as a celery task launched by paster command.

Still have to add a CKAN plugin so that QA will run when a resource is added or URL changed, and then update the extension controller/templates to look for QA data from the task_status table instead of directly on the resource.

• Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: ​http://thedatahub.org/dataset/edit/iati-registry
• Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in.

From Jonathan gray:

At the moment to delete a package you have to go to 'state' and change from 'active' to 'deleted'.

For me this was not very obvious, and someone recently wrote to me about this too.

Its a small issue - but unless you know what 'state: active' refers to, its not obvious what this means, nor that this is where to go if you want to delete a package.

From UI perspective this could be replaced by either a trash can or some text that says 'delete', perhaps with some prompt that explains what happens when you delete a package - explaining that non-admins won't be able to see it, and how you purge, etc.

DOI = digital object identifier = ​http://www.doi.org/

As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way.

Details

• Probably implement as extension rather than core

Why: current stable version of sqlalchemy. geoalchemy stuff required 0.7 and likely that some other things will require it soon.

Probably requires work on vdm ​https://bitbucket.org/okfn/vdm

NB: should have discussion before making 0.7 the default required version in CKAN core.

A good service API needs to be discoverable, so you are not always having to refer to the documentation html.

Maybe /api/action should return a list of actions available? (Currently this returns a 404.)

• It would be nice to sort these into get/create/update/delete.
• #1438 Parameters for each of the actions must be discoverable too

/api/action/{action_name} should also return the help text / parameters allowable. (Currently this returns 400 error)

Super ticket: #1602 (Data Previewer v2)

We already have first pass of Data Explorer that was released as part of #1357.

• Merging ​https://github.com/okfn/dataexplorer with ​http://github.com/okfn/recline and developing it
  • ​https://github.com/okfn/recline/issues

Tickets include (* indicates improvement over current explorer)

• Core Backbone Models representing Dataset and Tabular data: ​https://github.com/okfn/recline/issues/10
• New theme: ​https://github.com/okfn/recline/issues/22
• Read-only mode: ​https://github.com/okfn/recline/issues/17
• Introduce hash navigation / state support (*): ​https://github.com/okfn/recline/issues/19
• Re-enable editing in DataTable? (*): ​https://github.com/okfn/recline/issues/13
• [super] DataTable? view (in Backbone) (*): ​https://github.com/okfn/recline/issues/14
• DataExplorer? parent view (*): ​https://github.com/okfn/recline/issues/12
• Simple graph widget using flot: ​https://github.com/okfn/recline/issues/11

Est: 10-15d (should be broken down -- partly is in recline issues)

Should list resources in the form of search results rather than as a table.

• css class: resource-url-analytics
• Assign to the link tag (a) everywhere we want to count (dataset, resource view) - 0.25d
• Display counts in same place ...

And deploy on ​http://thedatahub.org/ - 0.25d

Possible: Also move analytics extension into core (decided not to).

Allowing more flexible tag names:

• allowing spaces
• allow capital letters (search is case in-sensitive)
• allow all punctuation except for commas and double-quotes '"'
• allow unicode
• commas delimit tag names in the package create/edit form

Effects:

• package creation/edit form.
• /tag/{tagname} uri
• search action
• api controller (search/package-create/edit)
• web controller (search/package-create/edit)
• search api documentation
• autocomplete for tag names

If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those.

Provide way to mark a dataset as featured. Featured database show up on the front page.

TODO: detail this more.

The current harvesting implementation can only have one instance per server.

We could either:

• Accept this
• Change the RabbitMQ code to support multiple instances
• Switch the entire harvesting to use Celery
• Wait for James's redis based live feedback based system

Nils flagged up that Ubuntu launches another LTS in April which we should probably move to. This ticket is to get the ball rolling with what we do. We can always include in our package the exact package versions of deps that we want, but it is good to be in line with the release if possible.

Here's how to see the packages included with it: ​http://packages.ubuntu.com/search?keywords=python-pylons&searchon=names&suite=all&section=all

Pylons 0.9.7 -> 1.0-2 Requires a small amount of work SQLAlchemy 0.7.3-2 OK Babel 0.9.6-1 OK Genshi 0.6-2 Perfect

Another alternative is to move to another web framework, especially minding that Pylons is not being further developed.

navl tests are being skipped unskip them!

This is the implementation of the final draft in

​http://wiki.ckan.org/Groups_Refactor

Steps to complete this will follow (somewhat) in the following order.

Change package_group table to membership. (#1478) 3d

• model migration. (done)
• simplify revisioning (done)
• Remove any sqlalchemy releations from groups. (done)
• change logic functions (done)

Single type per group (#1531)

• migration add type to group table. (done) 1d

Add capacity table (#1522)

• model migration. (done)
• add to logic functions.

Allow other objects to be added to groups. (#1531)

• change logic functions (done)

Adapt thedatahub.org to use new model.

• move auth groups over to groups/members
• change user_object roles to use groups? keep user object roles for time being.
• make create update package/group put the correct items in the group/members/capacity tables.
• work out how ui works.

--not doing the below as part of this super ticker now #1669.

Create model for dgu.

• adapt auth functions to use auth.
• adapt package_create/update group_create/update to fill in the correct capacties/members.

Adapt Iati, and potentially other custom instances.

Sub-tickets:

• #1279 Deprecate Authorization Groups
• #1198 Publisher hierarchy

munge_title_to_name, munge_tag v. useful for client software importing things into CKAN. I want it for ONS.

ordering on homepage by name instead of group count

ckanext-example needs updating for CKAN 1.5:

• theme changes
• new forms

About: 'ckanext-exampletheme' was created in Spring 2011 as an example CKAN extension that showed how to customise the look & operation of CKAN. This moved to github and renamed 'ckanext-example'.

This could be implemented as a separate API call, or the rendered text versions of the activities could be added into the JSON returned by the existing API call.

This requires setting up templates for rendering activity streams items and detail items as nice, human-readable text.

There are some open questions, e.g.: Do we want the entire activity stream rendered as a block of plain text? As HTML? Or do we want a list of JSON objects, where each object contains its textual and/or HTML representations as fields?

Activity stream items and their related detail items are separate objects that each have their own textual representations.

For a mockup of the kind of text messages we want, see:

​http://datahub.pudo.org/pudo

but note that this ticket is just for creating the text snippets themselves, not rendering then in an HTML page or RSS feed. Also the mockup only show activity items and not their detail items.

At some point the SOLR schema used by CKAN will need to be changed and the changes won't be always backward compatible, so we need a mechanism to support instances that use older versions of the schema. This process will involve:

• Better documentation on how to deploy SOLR, as single instance or multicore
• A convention on how to tag schema versions and SOLR endpoints to ensure that CKAN instances point to the correct SOLR URL.
• (If possible) Checks in CKAN core that deal with different schema versions.

Historical Parent Tickets

• Parent-ticket: #1450
• Parent-ticket: #1032

Step Zero

• COMPLETELY REDESIGN resource edit process

Dataset Edit-Resources

• Force format field to be lowercase
  • Modify backend to serve it up in that format
• Subticket #978: Add extra fields to resources in Web UI
• Fix display of large datasets (​http://test.ckan.net/dataset/europeana-lod)
• When submitting forms with errors, open the correct resource pane.
• No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10
• Resource description field to become the markdown editor we use everywhere else.

Dataset Settings

• Subticket #1648: Clarify edit terminology
• Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).
• Fix (i.e. allow / show) name editing on dataset edit.
  • Fix the slugify so it does not overwrite if i change title after fixing name.
  • Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'
• Autocomplete the author/maintainer box (requires API extension?) [WONTFIX what? it's not like that field is linked to users of the site or anything]

Dataset View

• Fix animation glitch in expanding description
• Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?
• Verify that there are no more (none) placeholder <td> objects (i believe they've been refactored out)

Dataset Create

• "Upload or link data files, APIs and other materials related to your dataset."
• Include the description markdown editor again (keep it small)
• Licence should maybe be restored
  • "(Don't worry if you don't know what your licence is)"
• Show the log message field
  • Beautify
• Pre-populate the log message field?
• ? Show Home Page field when creating dataset [WONTFIX don't push it]
• ? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation? [WONTFIX doesnt make sense with new resource editor awesomeness]

User pages

• Improvements to user page UX - 1d - #1396
• Broken links for non-Gravatar use icons - 0.25d - #1719

Etc

• Take before and after screenshots

This ticket is about finalizing, then deploying (to ​http://thedatahub.org/), and finally announcing some of recent major new features such as:

• Deploy wordpresser extension for thedatahub (so we can start blogging) - #1533 - WONTFIX
• (Deploy new Resource and Dataset pages (invite comments))
• Get thedatahub geo-ified i.e. ckanext-spatial - #1601 (?)
  • Any additional dev that needs to be done? (e.g. geo in search ...?)
• (Deploy / re-enable ckanext-googleanalytics on thedatahub - #1451 (more stuff #1566))
  • Rename the extension to ckanext-analytics?
• Custom forms (and use of 'dataset types') - Ticket (??)
• (Get QA deployed and running - #1582)

It would be best if this was done by someone *unfamiliar* with the extensions so feedback could be provided on quality of install docs etc etc.

Add a function to logic.action.get that returns a user's public activity stream as a list of dicts.

Makes more sense to only have one comprehensive stats/analytics extension, so when people are looking to add a stats extension they won't have to add two which may be confusing (is one an old version of the other? why do I have to add two? what's the difference? etc)

User-wise we need a way to display our stats with google analytics in the same place

Need to add capacities to member tables.

Re-create behaviour of old form without using form-alchemy. And unit test.

• [X] write high-level functional tests for the form. Creation / edition / validation.

Original estimate: 6 days (3 days x 2)

Time spent: 3 days

Time remaining: 0 days

• [X] write the html

Original estimate: 2 days (1 day x 2)

Time spent: 1.3 days

Time remaining: 0 days

• [X] modify the existing validation schema

Original estimate: 2 days (1 day x 2)

Time spent: 1.3 days

Time remaining: 1 days

• [X] split resources into types: individual, time-series, and additional.

Original estimate: 2 days

Time spent: 2.5 days

Time remaining: 0 days

• [X] contact information is derived from Publisher, but can be altered for each dataset. #1617

Original estimate: 4 days

Time spent: 0 days

Time remaining: 4 days

(Waiting on publisher integration)

• [X] foi contact information is derived from user, but can be altered for each dataset. #1617

Original estimate: 1 days

Time spent: 0 days

Time remaining: 1 days

(Waiting on publisher integration)

• [X] provide stubs for the theme and sub-themes #1618

Original estimate: 1 days

Time spent: 0 days

Time remaining: 0 days

• [X] provide stubs for the publisher hierarchy #1619

Original estimate: 2 days

Time spent: 0.5 days

Time remaining: 1.5 days

Note - the groups refactor is essentially ready to be used. And this would be a good test case for it.

• [X] don't allow package-type selection when editing a package

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa. (#1620)

Original estimate: 0.5 days

Time spent: 0 days

Time remaining: 0 days

Rather than the anonymous 'user' icon in the top-right corner, we could display your gravatar on all pages when you are logged in.

This was suggested by Richard Cyganiak: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-November/001825.html

The new members table needs a way so you can add arbitrary domain objects against them.

We need to extend the group schema to accept types, and instead of just being able to add packages to groups add members with their capacities that associated with different table rows.

4d

When using the API, the apikey needs to be supplied in a header called 'Authorization'. Because some proxys / deployments use this header for other things, a configurable header was provided as an alternative, with default "X-CKAN-API-KEY".

Rufus suggests having *one* way for this. a) making this not configurable any more b) making X-CKAN-API-KEY the default

(keep Authorization allowed, but not documented, for backwards compatibility)

Allow a user to save a partially-complete and probably invalid package-create form in order to come back to it later.

We decided on blob storage over a state field on the package:

• prevents a history of the drafts being stored in the revision history. This is a particular problem if we start auto-saving, say every 20s.
• the blob can store arbitrary extra data, such as how far through the wizard we are.

A sysadmin should be able to easily examine a suspect group or package, determine if it was created by a spammer (as opposed to being a legitimate object that has been graffitied by a spammer) and purge it.

The existing two-stage revision delete is currently unreliable and perhaps too laborious.

Olav and Richard have needs along this line.

Time estimate: 1d

Add a two search method to the IPackageController interface so extensions can:

• Modify the query being sent to Solr

def before_search(search_params):

    return search_params

• Modify the results returned (after_search)

def after_search(search_results, search_params):

    return search_results

Please see this pad for implementation details:

​http://ckan.okfnpad.org/search

It would be great to have a CKAN extension that allowed users (or CKAN itself) to generate short links to other URIs (both internal and external). Once created, shortlinks made by CKAN should be changeable. This would allow uploaded content to be moved without the user's link changing at all. The tool itself might also be of use as a general link-shortener to users other than the CKAN system itself.

Another useful feature would be for this to also collect some simple analytics such as the referrer and client IP for future reference. I'm not yet sure what we would do with the analytics other than some sort of popularity metric.

Questions:

• Core, or Extension, or Self-hosted?