Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted.

There is no way to delete objects at the moment - changes to 'state' are ignored in the API.

Do we need an alternative way to purge objects in the API?

Subticket of forms super ticket #961

• Forgot password (email a new password)
• Confirm email
• Do not show register page if you are logged in (redirect to home page)
• ticket:1010 - listing of users.
  • Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)

Whilse using CKAN web interface, you are not tempted to edit stuff:

• You know at all times this CKAN is read-only
• All editing facilities are still seen but greyed-out with an indication why it is.

Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.

Issues

• Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961

The debian package "ckan" with the two scripts "ckan-create-instance" and "ckan-instance-maintenance" depends against "postgresql". But "ckan-create-instance" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs.

Please add a flag "--without-local-db" to "ckan-create-instance" and remove the postgres dependancy from the debain package.

I'm not sure if this was reported earlier, but I have found a minor bug in the "Add new package" form. As far as I can see this bug exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). This is a problem with adding new resources in "Resources" section. To reproduce the bug do the following:

1. Go to "Add new package", for instance ​http://ckan.net/package/new
2. Scroll down to "Resources" section.
3. Click on "Remove this row" link (red circle).
4. Say "Yes" to "Are you sure you wish to remove this row?"
5. Now there should be no rows. Only the header "URL* Format Description"
6. Click on "Add row to table"
7. All you get now is another row with table header "URL* Format Description". You will not be able to add any resources now. Solution is to reload the page.

The simplest solution to this bug will be to hide the "Remove this row" link if there is only one row.

We need an email function in CKAN to accept messages sent to users. The basic signature will be:

• mail_user(user_obj, subject, body, mime_type='text/plain', headers={})

This has a number of use cases:

• Retrieval of lost passwords
• E-Mail confirmation

Finally, the mail function should be exposed in the API for sysadmin clients. This way we can have scripts traverse CKAN for 404s, invalid data or missing fields and ping users about that automatically (requires traversal by revision, not package, to get the associated users).

Implementation

Note we have already written code like this (*and* tested it) in isitopen:

• ​https://bitbucket.org/okfn/isitopen/src/bb2cbd146fa5/isitopen/lib/mailer.py
• ​https://bitbucket.org/okfn/isitopen/src/bb2cbd146fa5/isitopen/tests/lib/test_mailer.py

• No tests.
• 500 error given when user is not found.

Upload to ckan.net: ​https://sites.google.com/site/scotlandsdata/

Got broken due to the moderated edits changes #1141.

Need to make sure preview works.

Using ckanapi

Dgu tests failed as harvesting table are not created at the correct point.

This needs to be abstracted and we still need to be able to remove items before saving.

I'm finding that if I try to take an action with insufficient credentials from a test then I often (but not always) get a 401 error, whereas in the browser I get redirected to the login page.

It's a bit worrying that the program in its test environment doesn't behave like it does in the browser.

Current write method just creates a bare bones package and does not write any of the available metadata other than name.

Background

In the Web UI, when you rate a package it simply links to something like:

​http://ckan.net/package/rate/mke-liquor-licenses?rating=3

This creates a GET request.

This is bad because:

• Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this)
• There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept? GET> Apache instruction in. But the database may still get written for these ratings.
• Best practise for web requests is for GET to be a read-only request.

This package:

​http://ckan.net/package/dbtune-audioscrobbler

was:

1. created by Richard (logged-in)
2. edited by Richard (logged-in)

(According to the logs, at this point the state was changed from 'active' to 'deleted') -- RP was it set to 'deleted' or just ?

The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests.

Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement?

We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include:

• CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
• Catalogue scraping for LOD2 experiments (yields RDF graphs)
• Atom/DCat for LOD2 production (yields RDF graphs)
• OAI-PMH for ​http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios.

In general, such a system would have the following stages:

• Source selection: find what to download/scrape/harvest/parse
• Index retrieval (i.e. package index)
• Item retrieval (i.e. package entity)
• (Optional: Serialization)
• Normalisation
• Loading/Merging? into CKAN

Exisiting harvesters are at:

• CSW: ​https://bitbucket.org/okfn/ckanext-csw/src/
• Scraper+CKAN: ​https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/

We should create a public AMI with CKAN pre-installed and configured such that users can easily create their own EC2 machine with a running CKAN to play with.

There are three phases:

1. [nils] Deploy an empty EC2 instance to become the CKAN image master instance
2. Install a CKAN and give it a standard configuration.
3. [nils] Create a AMI from the CKAN image master instance and publish it.

I am happy to do first and last. Who is installing and configuring CKAN?

Unfortunately AMIs are specific to region, architecture and storage type. We cannot maintain too many images, so a number of choices have to be made:

• Which distribution/version? Ubuntu 10.04 LTS
• Which architecture/instance-type? I suggest 64-bit/t1.micro
• Which region? I suggest us-east-1 and maybe eu-west-1
• Which storage type? EBS (way easier to make an AMI from than instance-store)
• Install CKAN from deb packages via mercurial/virtualenv? I assume the latter because the AMI is targeted to developers?

​1 ​2

There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests.

Introducing caching has to be done carefully and should be done in a standards compliant manner.

General strategy

• Where possible, cache output within the pylons app (beaker).
• Facilitate external caching in an end-user's web browser or a caching proxy
• Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes).
• When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data.

Tasks

These tasks should be broken into sub-tickets:

• caching of parts of templates that are expensive to render (package list, tag list, group list)
• caching of entire output using beaker particularly for API read operations.
• need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches.
• cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: ​http://www.mnot.net/cache_docs/#CACHE-CONTROL
  • Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well
  • Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner

Future Work

• Investigate ckanclient library maintaining a local cache as a web browser would
• Investigate using a CDN like Google Storage or Amazon for serving cached data.

This needs more analysis, but the GEMINI2 attribute "metadata point of contact" must be reconciled with the registered publisher (or agent).

This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider.

PDEU has a newer layout which could be adapted to also be used on ckan.net. For this, things should be cleaned up, copied to the ckanext-ckan.net repo as needed and re-colored the core CKAN color scheme.

Maybe we could have this coincide with a CKAN rename?

As a user I want to view a package at a given revision:

• When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
• package history page should provide links to these pages

Cost: 2h

As a user i want to see a package at "version" X (NB: not revision X).

• When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version)
• Implementation:
  • Find revision for this version (search revision history for when version field was last 0.7)
  • Show pacakge at that revision (as in ticket:103)
• On history page also shows versions in list of revisions associated with the package

Search of package name and title (and other fields) using regular expressions.

Current example use-case: Wanting to specify packages with title beginning with 'B'.

Issues:

• Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields?

Implementation:

• Postgres reg ex searching detailed here:

​http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE

Child tickets:

• #234 UI Review - Autocomplete package names & tags in search
• #193 Searching by time-related field
• #191 Searching by modification date
• #905 Unable to search with accented characters in package names
• #906 Ability to search without accents for accented words
• #924 Search box has no search button

Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two.

As an admin I want to check a CKAN instance works having just upgraded it or configured it.

When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again.

When package does not exist in ckan catalogue, ckanclient.package_entity_get should raise more specific exception, such as CkanNotFoundError? instead of generic CkanApiError?.

There's a problem with getting old versions of packages using @date. The query looks for the PackageRevision? related to the date, but ignores TagRevisions? etc.

So for example if you compare this table: ​http://ckan.net/package/history/osm with this view: ​http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830 you actually see the 2010-01-13 11:13 revision - the wrong revision.

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

• Testing for dataproxy
  • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
• Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
• Google docs style row/column selections
• Use the swiss library - ​https://bitbucket.org/okfn/swiss
  • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
• Handle redirects for content-length?
• Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

• Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}

Characters outside of ASCII range are not supported within data previews.

Steps to reproduce:

1. Visit ​http://ckan.net/package/kivele2010
2. Click on [preview] for any of the resources

All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: ​http://ckan.net/package

Currently have this script section put in for i18n. It shouldn't be.

<script type="text/javascript">
//<![CDATA[
(function($){
    $.fn.ajaxCreateSlug = function(name, url) {
        var title = this;
        var updater = {
            init: function(title, name) {
                // Add a new element where the validity of the package name can be displayed
                this.name_field = name;
                this.title_field = title;
                this.name_field.parent().append('<div id="package_name_valid_msg"></div>');
                this.title_field.blur(this.title_change_handler())
                this.title_field.keyup(this.title_change_handler())
                this.name_field.keyup(this.name_change_handler());
                this.name_field.blur(this.name_blur_handler());
                this.url = url;
            },
....

As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

• An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install.
• A Reference Manual which is the primary source of reference for CKAN software - this includes API docs.

The current chapters of the Sphinx docs should be moved as follows:

• index.rst - copy some stuff from README.txt, keep short
• README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst
• [NEW] install.rst - new file on installation
• [NEW] theming.rst - move over from wiki
• api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
• i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
• design.rst - REMOVE (can copy some over if you can be bothered to ​http://wiki.ckan.net/Vision (all philosophical stuff should go!))
• loaders.rst: move to wiki.ckan.net/Using_Loaders
• feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
• importer.rst: remove (don't even copy)
• forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) ​http://wiki.ckan.net/Customizing_Forms
• forms-integration.rst: remove
• model.rst: leave but move to reference (only for core devs)
• load-testing.rst -> move to ​http://wiki.ckan.net/Load_Testing
• distributed.rst -> remove
• admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
• deb.html -> go into reference (query james gardner on list about moving this to wiki?)

It seems that searching for extras_foo:value works with solr, but extras_foo.bar:value doesn't. No theory on why.

For example, I will document the following:

• A specific guide on using mercurial when working with our branching and merging policy
  • DONE - ​http://wiki.ckan.net/Becoming_a_CKAN_Developer
• Policy document on how we use tickets, plan sprints and make releases
  • DONE - ​http://wiki.ckan.net/Becoming_a_CKAN_Developer
• How to package software as .deb file for project deployment

I'd love if someone else would write:

• An authorisation tutorial covering the core model, the command line tools and examples of every possible way of using the system
• A HOWTO guide with screenshots for adding a package

See reports on list at:

• ​http://lists.okfn.org/pipermail/ckan-dev/2011-July/001103.html
• ​http://lists.okfn.org/pipermail/ckan-dev/2011-July/001107.html

Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix.

We get exceptions on these occasions:

• user/edit if not logged in
• user/edit/bob if bob is not a known user

These are a problem only on 1.4.3b.

Introduced in #1229 - merge of branch feature-1229-db-out-of-controllers

Occasionally see this problem when producing an error page. Here's an example

WebApp Error: <type 'exceptions.TypeError'>: You cannot set the body to a unicode value without a charset
URL: http://at.ckan.net/authorizationgroup/new
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:117 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:284 in __call__
<<              if log_debug:
                       log.debug("Calling Response object to return WSGI data")
                   return response(environ, self.start_response)
               
               if log_debug:
>>  return response(environ, self.start_response)
Module webob.exc:248 in __call__
<<              return []
               if not self.body and not self.empty_body:
                   return self.generate_response(environ, start_response)
               return Response.__call__(self, environ, start_response)
>>  return self.generate_response(environ, start_response)
Module webob.exc:239 in generate_response
<<              status=self.status,
                   headerlist=headerlist,
                   content_type=content_type
               )
               return resp(environ, start_response)
>>  content_type=content_type
Module webob.response:94 in __init__
<<                  if charset is None:
                           raise TypeError(
                               "You cannot set the body to a unicode value without a charset")
                       body = body.encode(charset)
                   self._body = body
>>  "You cannot set the body to a unicode value without a charset")
TypeError: You cannot set the body to a unicode value without a charset

The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters:

In [1]: import ckanclient

In [2]: c = ckanclient.CkanClient()

In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api')

In [4]: c.package_search('', search_options={'groups':'openspending'})
Out[4]: 
{u'count': 2102,
 u'results': <generator object _result_generator at 0x10168b1e0>}

In [5]: c2.package_search('', search_options={'groups':'openspending'})
Out[5]: {u'count': 29, u'results': <generator object _result_generator at 0x10168b410>}

Thanks to borior for finding this and raising it.

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

I've installed the Moderated Edits extension (ckanext-moderatededits) and am editing a package imported from IATIregistry.org, with an extra field which contains a bit of HTML.

The editor indicates the field has changed, although the content hasn't (see screenshot). All I can find so far is a minor difference: in the field content, there is a code &#8212 and in the rendered table that is an &mdash;

The CSW harvesters implemented at the moment were developed with the DGU project in mind, and they assume all remote CSW servers to implement the Gemini 2 specification. Gemini 2 is the profile defined in the UK for INSPIRE complying metadata, so obviously catalogs from other countries or non-INSPIRE complying ones won't be able to be harvested.

The changes needed to support generic CSW servers (i.e. those implementing the ISO 19139 profile) are:

• Handling the validators (right now are hardcoded in the harvester

code). This probably involves issues discussed in the CREP 3 (ticket #1134)

• Changes in the model to adapt the specification to ISO 19139

• Renaming objects and classes which are now Gemini-centric

List of CSW servers tested:

​https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&hl=en_US&authkey=CNu4hsEB#gid=0

There are some functions that still use the Auhtorizer().authorized_query method:

./ckan/controllers/authorization_group.py:24:        query = ckan.authz.Authorizer().authorized_query(c.user, model.AuthorizationGroup)
./ckan/lib/base.py:237:        groups = ckan.authz.Authorizer.authorized_query(c.user, model.Group, 
./ckan/lib/search/sql.py:55:        q = authz.Authorizer().authorized_query(username, model.Group)
./ckan/lib/search/sql.py:118:        q = authz.Authorizer().authorized_query(self.options.get('username'), model.Package)
./ckan/logic/action/get.py:154:    query = Authorizer().authorized_query(user, model.Group, model.Action.EDIT)

./ckan/tests/test_authz.py:158:        q = self.authorizer.authorized_query(self.notadmin.name, model.Package)
./ckan/tests/test_authz.py:353:        q = self.authorizer.authorized_query(self.notmember.name, model.Package)
./ckan/tests/test_authz.py:357:        q = self.authorizer.authorized_query(self.member.name, model.Package)
./ckan/tests/functional/test_authorization_group.py:44:        group_count = Authorizer.authorized_query(u'russianfan', model.AuthorizationGroup).count()

CORS - ​http://www.w3.org/TR/cors/ - support.

This is what you do in Apache. Should do this in lib/base.py or similar.

    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Methods "POST, PUT, GET, OPTIONS"
    Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type"

    # Respond to all OPTIONS requests with 200 OK
    # This could be done in the webapp
    # This is need for pre-flighted requests (POSTs/PUTs)
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} OPTIONS
    RewriteRule ^(.*)$ $1 [R=200,L]