... probably by adapting aidtransparency.net

• Change "Publishing Entity" to "Publisher"

Move from hetzner box to s3.

Creation of duplicate packages named with trailing underscore

ww 22/9/10:

Need to ... merge some packages (the "find packages by attributes" is not 100% reliable because of inconsistent data already in there).

The "find by tags" algorithm uses (title, department). Unfortunately some of the data already there has no department. This causes it to make a new package (with trailing underscore).

dread 24/9/10:

I've done a test for the missing department issue and I can't reproduce the problem. This may be due to my general tidy up and getting tests passing using the latest ckanclient. Anyway, can you check over my test to see if this is what you mean?: dgu/ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadMissingDept

ww 20/10/10:

you should uncover it if you run against (a copy of) the dgu database.

Robust against slightly incorrect URLs (e.g. with trailing slash, or without /csw), and non-CWS service which also return <ows:ExceptionReport ... />.

Both cases would be resolved by checking capabilities (with CswClient's check_get_capabilities() method).

...obviously, not all packages will have a harvested metadata document.

Need to make sure #700 is fixed.

Also configure hmgckan, hmgapi, test-hmg and dev-hmg machines.

Need to be able to create (and remove) sysadmins easily (used to be able to do this via ini file but this functionality was removed a while back in authz system reorganization it seems).

Suggest creating a new paster command as first step.

# user identifier = openid (?)
paster sysadmin add {user identifier}
paster sysadmin remove {user identifier}

Suggest this also be added to main CKAN docs.

This should go into metastable as well as default so that it goes into the v1.2 release (can do this via a transplant from one branch to the other)

Auth Proposal

Use case: We'd like an authenticated and authorized Drupal user to be able to edit/delete packages using the API

To do this a user would need a CKAN API key so we need some way of Drupal telling CKAN who a user is so that they can get their API key.

Proposed Implementation

A user visits the CKAN API key page to get their key. Because CKAN is at catalogue.data.gov.uk (a subdomain of the Drupal site) it can read Drupal cookies.

If there is no DRXtrArgs or DRXtrArgs2 cookie, we know the user isn't signed in so we redirect them to Drupal to sign in.

WISHLIST: It would be really nice if we could pass the URL to redirect back to Drupal so tha Druapl can send them back to the CKAN API key page

Either way, they get back to CKAN and now the cookies exist. When the first HTTP request header is sent, CKAN will read the Drupal session ID and then call a Drupal API, server to server.

TODO: Implement an API on the drupal server which accepts a Drupal session ID as an argument and returns the username and credentials, but only if the request if from the CKAN server (perhaps we specify an API key in the request)

If the session is valid CKAN will set its own auth cookie and show them the page with the API key. CKAN only considers a user signed in if both the CKAN cookie *and* Drupal session ID are present. If at any time they sign out of Drupal the Drupal session disappears so they will be signed out of CKAN too.

Now the user has an API key they can use the standard CKAN command line API tools.

The API key is the only thing the user will need to use the CKAN API. This poses a problem. What if a user is removed from Drupal but still has a CKAN API key?

Two solutions:

1. API keys will be set to only be valid for the length of a Drupal session so that a user will only be able to use the write API for a little time after credentials are revoked

2. CKAN provides an API to allow Drupal to revoke keys

Option 1. seems easier to me. If we choose this we will write a get_api_key command line tool so that Drupal user can write things like this:

::

datapkg list_packages --api_key get_api_key

The get_api_key function will prompt for username and password and then perform the steps necessary to get an API key.

WP twentyten has an excellent core html structure. Furthermore, using that structure makes us compatible with all the WP twentyten compatible themes.

• Convert to wp twentyten html
• Switch css to be based off twentyten css (and perhaps rework somewhat)

We need an incremental plan that connects with current state of DGU and reflects what is actually required by UKLP.

Packages should provide create and last modified attributes which are included in package dict exposed via api.

These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc)

A link to the “register new package” page would be handy somewhere on the front page. As a frequent editor, I often go to ​http://ckan.net in order to create a new package. I don't want to hunt around for the link to the “new package” form.

The link could go into the bar next to “About / Statistics”. Or into the “More information” sidebar. Neither of these truly make sense, but it's a frequently accessed link and that's reason enough to put it somewhere onto the frontpage.

The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource.

This would help improve data quality and consistency.

Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in.

WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'apikey' 						

URL: http://test-hmg.ckan.net/user/apikey

1. Create a package.
2. Edit package and add a resource.
3. View package in browser - no sign of the resource - ERROR.
4. View package in API - resource is there.

Reason: hash doesn't change because package.revision only takes account of core fields.

Intermittent failure on ckan default branch. Looks like brittle test.

======================================================================
FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py", line 250, in test_02_metadata_created_and_modified
    assert out == exp, (out, exp)
AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938))

​http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio

Already have the basic capacity using *_extras paths for templates and public directory (see ​http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: ​http://packages.python.org/Flask-Themes/

Convert stats section to a ckan plugin.

Stats need not be in core and making it a plugin would make it easier to extend and expand.

As a sysadmin I want to take CKAN out of wiki-mode into normal mode. In normal mode visitor and logged in users cannot create or add packages.

Suggest put this as an internal command in authz system and add to cli as paster command.

This is a meta ticket for a bunch of UI changes:

• #839 Package page redesign
• #846 Merge css files
• #850 Show welcome message
• #863 Prompt users to enter missing info
• #864 Search results changes
• #866 Add download formats to search results

• Use config variable to switch ETags caching on or off. Consider joining in with 'cache_enabled'.
• Default setting for (all) caching should be off.
• Needs documenting in configuration.rst

• perhaps we should show the openid as well to distinguish between users with the same name.

• perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name.

• also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it.

Allow a form to specify fields that are required. Currently name and resource url are required.

• Show required fields with asterisks by the input box.
• Resource fields to be included in this.

This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010).

3 key aspects of this:

• Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN)
  • Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable
  • Asynchroously: via worker with queue
    • #324 Search indexing using notifications
    • #669: Refactor Solr (indexer) to become a generic worker
    • #874: Extract Solr search backend into an extension
• In core support using SOLR backend as well as postgres for queries
  • #317: Make search pluggable (?)
• Integrate into WUI (if changes needed)
  • #672 - Facets in CKAN search results

Extras:

• Solr index testing tool #431

Control using cookies.

3 (4) use cases: (With and without access to auth keys.)

1. upload form on ckan (by default local system has auth key for storage)
2. datapkg upload (client) (by default without key) (plumbing done)
3. background job archiving downloads (by queue client)

Most important are:

1. Upload form on ckan.net (with access to keys)
2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

• #826 - Resource 'extra' fields
• #358 - Resources in REST API
• #235 - Resource format normalization and detection

User upload via Web UI

• #877 - File upload in WUI
• #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

• #853 - plumbing for credential-less rest upload
• #879 - Storage auth API

Also related:

• #405 - Retrieval options for package resources

More Info

• Upload/storage discussion: ​http://ckan.okfnpad.org/storage
  • For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/

The coverage tool (run by buildbot in the ckan build) reports that only 24% of lines of ckan.controllers.authorization_group are run in tests and 38% of ckan.forms.authorization_group. This suggests a need for more tests.

...

as per http://ckan.org/wiki/UIRedesignSearch

as above

The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach.

We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!)

Some suggestions for achieving this include:

• Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database
• Not setting up and tearing down the database so frequently.

Since the main menu items have changed (part of the UI redesign) the selected items are not functioning correctly.

Specifically if Home / About or Add Package / Search are selected both menu items are highlighted.

This is because the code assumes only one top-level menu item per controller.

e.g. national_child_measurement_programme had no department but "Information Centre for Health and Social Care" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_.

Consolidate search API interface (and backend) on solr (solrpy) type interface.

• Support for standard query structure
• Support for facet options

Do not need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

1. Disable (happens automatically)
2. Implement - suggest using group by etc

Extras

• Front-page tag cloud: change this to use facets
  • Accept this means that if facets not functional in backend we have no tag cloud

Support uploading files to a storage system from WUI with a simple upload form.

Implementation

• Implement as an Extension
• Locate at /upload
• Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
• Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
• This wants to be tied in to general package workflow. See parent ticket:852.

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

master ticket

• #884 implement harvest job delete command for an easier life
• #885 move csw code into dedicated ckanext plugin
• #886 move forms api into dgu extension module
• #887 move harvesting code to use generic harvesting module

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

• Testing for dataproxy
  • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
• Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
• Google docs style row/column selections
• Use the swiss library - ​https://bitbucket.org/okfn/swiss
  • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
• Handle redirects for content-length?
• Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

• Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository.

Questions

• Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
• Should we have clever ways to dump APIs? ANS: no.
• Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
• Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
  • Complete support for ETags
  • Expires, Max-Age etc.
• Check

Functionality

• Download files via HTTP, HTTPS (will not do FTP)

Process:

1. [Archiver.Update checks queue (automated as part of celery)]
2. Open url and get any info from resource on cache / content-length etc
   1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
   2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get content-length and content-length unchanged GOTO step 4
3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
   • Add cache url to resource and updated date
   • Add other relevant info to resource such as md5, content-type etc
4. Update task_status

Optional functionality

• If result object is HTML, search for references to "proper data" (CSV download pages etc.)
• Download from POST forms (accepting licenses or weird proprietary systems)
• Support running on Google Apps Engine to save traffic costs.

Existing work

• ​https://bitbucket.org/okfn/ckanext-qa/overview
• out of date: ​https://bitbucket.org/pudo/ckanextarchive - Old archiver extension, largely experimental.
• out of date: ​https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65 - Openness scores by ollyc

Every change to every object is being stored in memory, which could add up to quite a lot of memory.

This fixes it by making sure the objects are in a weakref. ​https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f

There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision. We add the uuid earlier to fix this. ​https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e.

From David Raznick

On a package page there are three buttons to the right - View, Edit, History. Each of these buttons contain an icon and the text, but you can only activate the button by clicking on the icon. When you click on the part of the button where the text is (about 50% of the button) then nothing happens.

Using: Chrome, Firefox

What's bad at the moment?

• lack of documentation e.g. config (very poorly documented)
• too many sources of documentation
• no common theming

Sources:

• http://ckan.org/
• ​http://packages.python.org/ckan
• ​http://wiki.okfn.org/ckan/ - user
• Deprecate: ​http://knowledgeforge.net/ckan/doc/ckan/

Resulting meta-ticket with things to do: ticket:927