Run latest ckan on eu0. Automate some queries and searches. Check load and database connections / processes.

Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor.

The attributes we need to read in are being defined here: ​http://okfnpad.org/uklii

The design advice is to prepare, test, and maintain a dictionary of XPaths for each of the attributes.

Work effort is to be directed to the maintainability of the XPath statements. We need to know which are needed for which documents, and to have a way of preventing cruft.

When a package listing is shown (as on ​http://ckan.net/package/list) more information than just name should be shown. For example package titles should be shown as well as names (could also list tags and other stuff but simplest is just title).

Cost: 1

Cost: 1h

The tests in CKAN only test single pages (functional tests) or single methods (unit tests). In order to make sure that the whole system still works as expected, we need tests that cover whole processes.

An example user story to be tested would be: "As a user with create permissions I want to be able to log in, create a new resource, save it and then make sure that it is in the system." or: "As a user I want to use the search functionality and open a resource from the search page."

These user stories require a different testing system with interaction and processes.

We propose the following push approach using individual packages:

1. New, updated and deleted packages are pushed to a url endpoint on Wordpress (WPURL). This endpoint will be a configuration option on the CKAN side. A POST or PUT http method will get used. The payload will be a json document of the following form:

{

payload: The entity data as available from the REST API entity-type: 'Package', operation-type: 'create'|'update'|'delete' # one of these options

}

It will be sent as the body of the request with content-type set to application/json.

2. The wordpress side will provide a 200 on success. Any other response will be taken as a failure. On failure, submission will be archived and failure logged and notified to system administrator. Submission can be resent later automatically by sysadmin after review.

3. Pushes will happen continuously and approximately simultaneously with updates (a webhooks type model)

4. [optional] CKAN side will support configuring authorization for basic authentication if applied on WP side.

5. [optional] List queue status (including failures) on ckan adminstrative dashboard.

(in the same way that we can refer to packages in the API by ID as well as name.)

As a

client of CKAN

I want to

link to a package page in the UI, referring to it by ID

User story #037 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

Title

Visitor visits service

As a

Visitor (general web user)

I want to

Visit the website of the service (frontpage)

So that

• See the front page and discover information about the service
• Perform various activities related to the service such as registering and finding packages, registering as a user and logging in

Default visitor roles in default config is reader, not anon_editor.

Problem caused by changes in #1066 (released in 1.3.3)

New installs will be affected, although simple to just increase permissions when the installer realises a visitor can't create packages.

The solution to the config getting out of sync with the code like this is to not have the default_roles in the config - refer to the code in the configuration instructions.

There's a problem with getting old versions of packages using @date. The query looks for the PackageRevision? related to the date, but ignores TagRevisions? etc.

So for example if you compare this table: ​http://ckan.net/package/history/osm with this view: ​http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830 you actually see the 2010-01-13 11:13 revision - the wrong revision.

As a

Visitor

I want to

View the files associated with a package.

Notes

• See also ticket:16 (updating a file).
• Files should either be listed on the same page as a release or there should be a link from the release page to a page listing all the files associated with the release. See also ticket:14 (update a release)

See Update a release (ticket 14) for more information on what should be shown.

As a user I want to view a package at a given revision:

• When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
• package history page should provide links to these pages

Cost: 2h

As a

Visitor

I want to

View the details of an individual package

So that

I know more about the package. Attributes available are detailed in updating a package (../UseCase12)

As A

Visitor/User?

I want to

View information about a package history, i.e. what revisions it has.

Details

• History info should be available at package/history/pkg-name
• Most basic info is just info from revision (when it occurred, author etc)
• More sophisticated would be to have diff support to see differences between revisions

• Make sure that the extension works as it should and isn't broken on current releases [3d]

• Make it work with the EC schema, including extras [2d]

With the organizations and organizations_dataset plugins enabled login and create a new dataset, adding it to a group that you are a member of. Edit the dataset and set the group to none. Now you can no longer edit your own dataset. (You can also specify None for the organization when first creating the dataset, same problem.)

Requirements

In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub).

Required interface

Once changes have been made to the user schema, for a given user we want to be able to:

• determine if they have a paid or a free account, and
• get a string name of the type of paid account.

Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin.

User Stories

User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases. For now it is adequate that we can manually control these things through paster commands.

Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all.

• As a sysadmin I would like to be able to use a paster command to manually set a user's payment level, or remove it entirely.

• As a sysadmin I would like to be able to run a paster command to view a list of users who have a payment plan, grouped by the plan that they have.

• As a sysadmin I would like to be able to use the API to change the payment status of a specific user through user_create and user_update. This shouldn't be available to anybody else.

• As a user, and only if I have one, I'd like to see my current payment level on my user profile page.

Tasks

[x] Tests

[x] Plugin based migration

[x] Code

[x] Model

[x] API

[x] Documentation

Estimates

They seem to be not exactly well thought out. Needs some thought on better display of the information that they contain.

The user management in organizations ( ​http://localhost:5000/organization/users/NAME ) should autocomplete names in the dropdown box so that only valid usernames are added to the table.

Native login scenario: Complete login form, require email address. Any session is resumed.

Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from BlastOff?.

See CKAN or Microfacts for usage of repose with openid.

Use case:

• Login using OpenID
• Click on 'My account' - results in 404

Solutions:

• User user.id instead of their name
• Escape the URL properly.

• Generic text box for markdown about the user 'About'

Model's user table reflects these:

• 'about' attribute

User story #035 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

Only happens if the user does not update the password. Most probably introduced during #1229.

Details of how to go about this here <​http://www.rufuspollock.org/2009/07/27/sqlalchemy-migrate-with-pylons/>

Cost: 2h

A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html

Instead we should use the site_url config option where necessary.

Cost: 1h

If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those.

The current search implementation uses the output of the the as_dict method of the domain Package object to update the index

​https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48

It also uses package_to_api1 in the SynchronousSearch? plugin:

​https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93

This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model).

The search should use the logic function to get the package properties:

get_action('package_show')(context,data_dict)

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

​http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.

Because people might try and do "tags:gov sweden" when "tag:gov sweden" makes more sense (sweden is searched for in all fields in these cases).

But in REST i/f search allow "tag"="gov" OR "tags"="gov sweden" in parameters.

I'm not seeing any javascript or python error, the upload just hangs.

Upload to ckan.net: ​https://sites.google.com/site/scotlandsdata/

I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from TheDataHub? core, for example.)

• Switch to the latest minifed CSS + JS
• QA-sweep the site, especially dataset/edit

Cost: 2h

At the moment most of the harvesting functionality is only available to sysadmins. This is fine for most cases but going forward we will need to grant each publisher access to their own sources, harvest objects and harvest object errors. This will probably be done based on the groups refactor.

One more pressing concern is giving DGU publishers the ability to make calls to /harvest/object/id. We want them to be able to see documents that faild validation and also to see *all* errors in each document.

• Add Ascii folding filter to text fields.
• Add capacity field for public, private access.
• Add title_string so you can sort alphabetically on title.
• Fields related to analytics, access and view counts.
• Add data_dict field for the whole package_dict.

Actual implementation of data_dict adding will be added later.

• Move code base, fix any breakage.
• Deploy to eu8
• Test
• Deploy to eu14

Why

Later CKAN codebase has a bunch of improvements / fixes that it would be good to have.

Estimate

• 8h

Pylons 1.0 has been out for a while and appears to be running stable on a few production sites. It also introduces a few new features, among them the ability to specifiy controllers using the entrypoint mechanism.

​http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py

This might be a bit ugly since the default behaviour of some pylons objects, such as tmpl_context, app_globals and redirect has changed.