CKAN: Ticket Query

#182: Improve openness icons

rgrp — Fri, 06 Nov 2009 19:09:50 GMT

Tick and cross aren't clear enough. Propose:

Icon for downloadability (nothing if absent ...(?))
Icon for open license? (Need something if closed ...)

Cost: 2h (?)

#333: CKAN front end requirements for package notifications

dread — Mon, 31 May 2010 16:40:42 GMT

Use case: new package

An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package.

User: clicks 'new package'.

CKAN presents the package/new form to the user.

(After a couple of previews) User: clicks 'commit'.

Notification message goes from CKAN to the front-end detailing the new package.

The user is redirected back to the front-end web page displaying the list of packages, which contains the new one.

The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6).

In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages?

Use case: CKAN imports packages

CKAN administrator runs a script that adds 100 new packages into CKAN.

CKAN sends notification message to front-end to report the new packages/revisions.

Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages.

The front-end queries CKAN for each new package one-by-one.

A new user request to the front-end will include the info about the new packages.

The package addition could be achieved in 1 revision, 100 revisions or some compromise:

If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'.

If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN.

This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages.

#339: Remove download_url from package data format in API Version 2, but maintain the old data format in the old version of the interface (API Version 1)

johnbywater — Mon, 07 Jun 2010 09:04:59 GMT

#341: Web UI accepts package IDs in URLs

dread — Thu, 10 Jun 2010 09:33:20 GMT

(in the same way that we can refer to packages in the API by ID as well as name.)

As a

client of CKAN

I want to

link to a package page in the UI, referring to it by ID

#344: REST Create package with incorrect format gives 500 error

dread — Sat, 12 Jun 2010 11:12:52 GMT

PUT to /rest/api/package of {"name": "name", "resources": ["someurl.com/data"] } (i.e. resource is a string, not a dictionary) gives 500 error, when it should give a 400 error and helpful error message.

Exception:

Module ckan.controllers.rest:154 in create
<<              if register == 'package' and not subregister:
                       fs = ckan.forms.get_standard_fieldset()
                       request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data)
                       fs = fs.bind(model.Package, data=request_fa_dict, session=model.Session)
                   elif register == 'package' and subregister in model.PackageRelationship.get_all_types():
>>  request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data)
Module ckan.forms.package_dict:88 in edit_package_dict
<<                      for res_dict in value:
                               res_dict_str = {}
                               for key, value in res_dict.items():
                                   res_dict_str[str(key)] = value
                               resources.append(res_dict_str)
>>  for key, value in res_dict.items():
AttributeError: 'unicode' object has no attribute 'items'

#353: SOLR search indexing

dread — Mon, 21 Jun 2010 12:31:20 GMT

As a

SOLR instance

I want to

keep my search index of CKAN packages up-to-date

Implementation

Using asynchronous event notifications
Running in a separate process to CKAN

#356: Search box in at top of page (UI)

rgrp — Tue, 22 Jun 2010 19:36:51 GMT

A small but useful ui improvement would be to have a search box at top right on every page.

As an example see the one here on trac or on github.com or bitbucket.org.

It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration).

#364: Search for 'statistic' returns nothing

dread — Mon, 12 Jul 2010 15:47:00 GMT

On ckan.net there are plenty of packages (and indeed their tags) with the word 'statistic' in them, but no packages turn up when you search for it:

http://ckan.net/package/search?q=statistic&search=Search+Packages+%C2%BB

(Using Postgres full text search)

#368: 500 Server error when creating package

anonymous — Tue, 20 Jul 2010 10:41:02 GMT

I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response:

"publish: New Zealand Coastline (new-zealand-coastline)" opening connection to nz.ckan.net... opened <- "POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n" <- "{\"name\":\"new-zealand-coastline\",\"title\":\"New Zealand Coastline\"}" -> "HTTP/1.1 500 Internal Server Error\r\n" -> "Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n" -> "Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n" -> "Vary: Accept-Encoding\r\n" -> "Content-Type: text/html; charset=iso-8859-1\r\n" -> "Cache-Control: proxy-revalidate\r\n" -> "Content-Length: 640\r\n" -> "Connection: close\r\n" -> "\r\n" reading 640 bytes... -> "<!DOCTYPE HTML PUBLIC \"-IETFDTD HTML 2.0EN\">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n<hr>\n<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80</address>\n</body></html>\n" read 640 bytes Conn close

The server should not respond with a 500 error if the package was created.

#377: API Should return JSON in all cases

glen@… — Tue, 27 Jul 2010 22:02:38 GMT

When the API returns any response it should always retunr JSON. For example a 404 response should return something like

{'status': 404, 'message': 'package somename not found'}

When errors are encountered when creating a package:

{'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be
blank', 'description':'license code must be an integer']}

Not authorised:

{'status': 503, 'message': 'You are not authorised to create this package.'}

The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created.

#381: Test defect

johnbywater — Wed, 28 Jul 2010 19:38:57 GMT

please ignore

#395: Set up profiling to analyze performance issues

pudo — Tue, 03 Aug 2010 08:38:59 GMT

At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls.

This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance.

NB: We should also agree on a maximum request latency.

TODO: Read up on all those QoS tickets to avoid overlapping efforts.

#403: Create web service for OFS storage

pudo — Mon, 09 Aug 2010 09:28:30 GMT

Create a thin web service that will allow users to upload files to OFS on S3 without knowing S3 credentials. This could also provide static urls for all stored resources.

relates to #307

#407: Link to Scraperwiki

dread — Mon, 09 Aug 2010 10:06:08 GMT

As a

user browsing a CKAN package which has a resource in a bad format

I want to

be alerted to the possibility of scraping it on scraperwiki

Implementation

This should be done in a CKAN extension. It should appear in a side-bar on the package read page.

If the package does not have a resource with scraperwiki.com in the url, then it displays this message:

Title: Scrape it
Image: (Scraperwiki dumper truck)
Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can request this data is scraped or even write the scraper yourself.

#412: Data format for package importing shall be documented

johnbywater — Tue, 10 Aug 2010 09:00:09 GMT

Requirements:

cater naturally for multiple packages with multiple resources
international encoding
large number of packages

#415: Chase Talis about loading RDF from CKAN.

dread — Tue, 10 Aug 2010 09:14:16 GMT

#418: With COI, get SOLR working

dread — Tue, 10 Aug 2010 09:14:51 GMT

#422: Administrator imports meta-data into CKAN

dread — Tue, 10 Aug 2010 09:44:50 GMT

Requirements:

cater naturally for multiple packages with multiple resources
international encoding
large number of packages

#428: Daily dump of DGU CKAN available

dread — Fri, 13 Aug 2010 12:40:51 GMT

End users want to get hold of a JSON & CSV dump of the DGU data.

#429: ONS data is imported daily to DGU

dread — Fri, 13 Aug 2010 12:41:34 GMT

#433: Data package metadata in the Egg

wwaites — Tue, 17 Aug 2010 20:57:40 GMT

Still not sure if we shouldn't use the existing setuptools machinery to manage this -- there is already a way to get at the metadata. In any event, I've just made an addition to datapkg that makes it possible to put datapkg_sources as a dictionary in your package's setup.py. Afterwards it is possible to pull the metadata out of the egg. Of course this could easily be changed to save the information in whatever form, indeed if you pass it a string instead of a dictionary it will just write whatever you gave it into the datapkg_sources.spec. The point is, I think that the egg is a good place to stuff this information.

For non-python users, it is always possible to simply put up the datapkg_sources.spec somewhere on the web so they can directly retrieve the data files.

From the docstring::

    This is the implementation for an [egg_info.writers] entrypoint.
    Datapkg adds an argument to setuptools's setup() function called
    datapkg_sources. The argument should be a dictionary of the form:
    .. code-block:: python
        setup(
            ...,
            datapkg_sources = {
                "cra2009" : "http://www.hm-treasury.gov.uk/d/cra_2009_db.csv"
            }
        )
    The result of this is that there will be a file in the egg called
    datapkg_sources.spec that looks like this::
        [sources]
        cra2009=http://www.hm-treasury.gov.uk/d/cra_2009_db.csv
    How do you get at this data? Simple::
    .. code-block:: python
        import pkg_resources
        dist = pkg_resources.get_distribution("ukgov_treasury_cra")
        spec = dist.get_metadata("datapkg_sources.spec")
    and 'spec' will be the contents of the file as a string.

#434: Logged in user to si.ckan.net causes genshi exception

dread — Wed, 18 Aug 2010 11:21:55 GMT

Here's the trace:

[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] Error - <type 'exceptions.IndexError'>: list index out of range
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] mod_wsgi (pid=6195): Exception occurred processing WSGI script '/home/okfn/var/srvc/si.ckan.net/py
env/bin/si.ckan.net.py'.
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] Traceback (most recent call last):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/paste/cascade.py", line
130, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return self.apps[-1](environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/paste/registry.py", line
 375, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     app_iter = self.application(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/repoze/who/middleware.py
", line 107, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     app_iter = app(environ, wrapper.wrap_start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/middleware.py", l
ine 214, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     self.app, new_environ, catch_exc_info=True)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/util.py", line 94
, in call_wsgi_application
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     app_iter = application(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/weberror/errormiddleware
.py", line 156, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return self.application(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/beaker/middleware.py", l
ine 73, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return self.app(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/beaker/middleware.py", l
ine 152, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return self.wrap_app(environ, session_start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/routes/middleware.py", l
ine 130, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     response = self.app(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/wsgiapp.py", line
 125, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     response = self.dispatch(controller, environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/wsgiapp.py", line
 324, in dispatch
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return controller(environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py", line 73, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return WSGIController.__call__(self, environ, start_response)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core.
py", line 221, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     response = self._dispatch_call()
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core.
py", line 172, in _dispatch_call
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     response = self._inspect_call(func)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core.
py", line 107, in _inspect_call
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     result = self._perform_call(func, args)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core.
py", line 60, in _perform_call
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return func(**args)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/controllers/error.py", line 32, in doc
ument
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     ckan_template = render('error_document_template.html')
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py", line 38, in render
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return render_template()
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py", line 36, in render_templ
ate
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return literal(stream.render(method=method, encoding=None))
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/core.py", line 18
3, in render
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return encode(generator, method=method, encoding=encoding, out=out)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line
57, in encode
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     return _encode(''.join(list(iterator)))
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 339, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for kind, data, pos in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 670, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for kind, data, pos in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 771, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for kind, data, pos in chain(stream, [(None, None, None)]):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 586, in __call__
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for ev in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/core.py", line 288, in _ensure
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py", line 618, in _include
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in tmpl.generate(ctxt, **vars):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py", line 618, in _include
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in tmpl.generate(ctxt, **vars):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py", line 605, in _include
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/markup.py", line 327, in _match
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py", line 545, in _flatten
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for kind, data, pos in stream:
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py", line 177, in _generate
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     for event in msgbuf.translate(gettext(msgbuf.format())):
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py", line 1029, in translate
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     parts = parse_msg(string)
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]   File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py", line 1143, in parse_msg
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197]     parts.append((stack[-1], string))
[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] IndexError: list index out of range

ckan: 1e403927f2bb (metastable) genshi: 0.6 lang: slovenian

#436: Investigate exception: resource search JSON

dread — Thu, 19 Aug 2010 08:35:59 GMT

Here's the dump from 22:10 last night:

URL: http://ckan.net/api/search/resource?all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class= Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:152 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:130 in call << environSCRIPT_NAME? = environSCRIPT_NAME?[:-1]

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:73 in call << # available in environpylons.routes_dict?

try:

return WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

return WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.rest:400 in search << response.status_int = 400

return gettext('Blank qjson parameter')

params = json.loads(request.paramsqjson?)

elif request.params.values() and request.params.values() != [u] and request.params.values() != [u'1']:

params = request.params

params = json.loads(request.paramsqjson?)

Module simplejson:384 in loads << parse_constant is None and object_pairs_hook is None

and not use_decimal and not kw):

return _default_decoder.decode(s)

if cls is None:

cls = JSONDecoder

return _default_decoder.decode(s)

Module simplejson.decoder:402 in decode << """

obj, end = self.raw_decode(s, idx=_w(s, 0).end()) end = _w(s, end).end() if end != len(s):

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

Module simplejson.decoder:420 in raw_decode << obj, end = self.scan_once(s, idx)

except StopIteration?:

raise JSONDecodeError("No JSON object could be decoded", s, idx)

return obj, end

raise JSONDecodeError("No JSON object could be decoded", s, idx)

JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, application/json' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_ACCEPT_LANGUAGE 'en-us,en;q=0.5' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'utma=27730403.1245320310.1281386803.1281386803.1282164955.2; utmz=27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892; utmb=27730403.3.10.1282164955; utmc=27730403' HTTP_HOST 'ckan.net' HTTP_KEEP_ALIVE '300' HTTP_REFERER 'http://jira/browse/PLATFORM-892' HTTP_USER_AGENT 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/api/search/resource' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/search/resource' QUERY_STRING 'all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=' REMOTE_ADDR '64.235.97.218' REMOTE_PORT '20720' REQUEST_METHOD 'GET' REQUEST_URI '/api/search/resource?all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/api/search/resource' SCRIPT_URL '/api/search/resource' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0xa1c13ec> beaker.cache <beaker.cache.CacheManager? object at 0xa1c142c> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0xa1c12ac>> beaker.session {'_accessed_time': 1282165818.0880959, '_creation_time': 1282165818.0880959} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host mod_wsgi.listener_port '80' mod_wsgi.process_group mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (<SimpleCookie: __utma='27730403.1245320310.1281386803.1281386803.1282164955.2' __utmb='27730403.3.10.1282164955' __utmc='27730403' __utmz='27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892'>, 'utma=27730403.1245320310.1281386803.1281386803.1282164955.2; utmz=27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892; utmb=27730403.3.10.1282164955; utmc=27730403') paste.parsed_querystring ([('all_fields', '1'), ('offset', '0'), ('limit', '20'), ('qjson', '<span class=')], 'all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=') paste.registry <paste.registry.Registry object at 0x130ed84c> paste.throw_errors True pylons.action_method <bound method RestController?.search of <ckan.controllers.rest.RestController? object at 0xe9bbe0c>> pylons.controller <ckan.controllers.rest.RestController? object at 0xe9bbe0c> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0xe9bbe8c> pylons.routes_dict {'action': u'search', 'controller': u'rest', 'register': u'resource'} repoze.who.logger <logging.Logger instance at 0xa3cb0cc> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 170067148>, 'auth_tkt': <AuthTktCookiePlugin? 171739788>} routes.route <routes.route.Route object at 0xa102fac> routes.url <routes.util.URLGenerator object at 0x13a5a3cc> webob._parsed_query_vars (GET([('all_fields', '1'), ('offset', '0'), ('limit', '20'), ('qjson', '<span class=')]), 'all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x12f53530> wsgiorg.routing_args (<routes.util.URLGenerator object at 0x13a5a3cc>, {'action': u'search', 'controller': u'rest', 'register': u'resource'})

#437: Buildbot test failures - ascii codec

dread — Thu, 19 Aug 2010 13:14:00 GMT

On today's buildbot: http://buildbot.okfn.org/builders/buildbot-test/builds/201

2 failures about ascii (ignore other 2)

#439: Plan server requirements for DGU

dread — Thu, 19 Aug 2010 13:27:32 GMT

Using use cases for security etc. Draw diagram to satisfy use cases. Communicate to Paul and therefore TSO.

#444: Discuss package relationships ideas with JF

dread — Fri, 20 Aug 2010 10:13:58 GMT

Create test data on visible ckan
Discuss with JF

#446: Server configuration query (?)

dread — Fri, 20 Aug 2010 10:16:25 GMT

Ensure use case is in design - "an admin writes to the CKAN API" (and see JB notes on paper)
Communicate how this is planned to be achieved in design to JF

#450: Assist with proxying of static media for forms

dread — Fri, 20 Aug 2010 10:19:33 GMT

John did this a while ago.

#452: Assist with using package update error form

dread — Fri, 20 Aug 2010 10:20:47 GMT

#455: SOLR - suggest 1 pager about how system would work

johnbywater — Fri, 20 Aug 2010 10:25:13 GMT

Either CKAN writes to SOLR and Drupal reads from SOLR, or CKAN writes to SOLR and Drupal reads SOLR via CKAN API (so search resource locations are unaffected).

#459: Versions on branches are broken

johnbywater — Fri, 20 Aug 2010 10:26:13 GMT

#463: package diff exception

dread — Fri, 20 Aug 2010 12:07:17 GMT

Investigate exception:

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               response = self.app(environ, start_response)
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
AUTH_TYPE	'cookie'
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'en-US,en;q=0.8'
HTTP_CACHE_CONTROL	'max-age=0'
HTTP_COOKIE	'__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173'
HTTP_HOST	'ckan.net'
HTTP_REFERER	'http://ckan.net/package/history/dbtune-audioscrobbler'
HTTP_USER_AGENT	'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'92.226.62.168'
REMOTE_PORT	'55102'
REMOTE_USER	'http://pudo.org/'
REMOTE_USER_TOKENS	['']
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0xbb528cc>
beaker.cache	<beaker.cache.CacheManager object at 0xbb528ec>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0xbb5286c>>
beaker.session	{'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: __utma='27730403.1381387877.1281962773.1282248244.1282250173.11' __utmb='27730403.6.10.1282250173' __utmc='27730403' __utmz='27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan' auth_tkt='e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!'>, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0xf702aec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xf6eaf0c>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xf6eaf0c>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0xf6eaa8c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.identity	<repoze.who identity (hidden, dict-like) at 280815340>
repoze.who.logger	<logging.Logger instance at 0xbcdfccc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 196612236>, 'auth_tkt': <AuthTktCookiePlugin 198048140>}
routes.route	<routes.route.Route object at 0xbbc44cc>
routes.url	<routes.util.URLGenerator object at 0xf5e55ec>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0xfe0b0b0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xf5e55ec>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})

#465: Require API key for all API requests.

johnbywater — Fri, 20 Aug 2010 12:55:04 GMT

#467: Admin configures CKAN to expect API key in named HTTP header

johnbywater — Fri, 20 Aug 2010 13:22:42 GMT

#469: Unit test to check API key in correctly named HTTP header is accepted by server

johnbywater — Fri, 20 Aug 2010 13:26:00 GMT

#471: API user sends API key in correctly named header

johnbywater — Fri, 20 Aug 2010 13:48:08 GMT

#472: API user sends API key in incorrectly named header

johnbywater — Fri, 20 Aug 2010 13:48:28 GMT

#473: API user discovers correct header for sending API key

johnbywater — Fri, 20 Aug 2010 13:50:03 GMT

#475: Unit test to check API key in incorrectly named HTTP header is not accepted by server

johnbywater — Fri, 20 Aug 2010 14:00:02 GMT

#476: Daily database dumps shall be discoverable and downloadable

johnbywater — Fri, 20 Aug 2010 14:10:33 GMT

#477: Discover location of the daily database dumps

johnbywater — Fri, 20 Aug 2010 14:16:28 GMT

#478: Unit test for "interface resource" in API

johnbywater — Fri, 20 Aug 2010 14:37:47 GMT

We don't want /api to be an HTML page, but rather provide information about the API in a JSON structure.

#481: Unrestricted total read-only access to catalogue API

johnbywater — Sat, 21 Aug 2010 20:31:37 GMT

#483: Restricted total read-write access to catalogue Web UI

johnbywater — Sat, 21 Aug 2010 20:33:21 GMT

#484: Restricted partial read-write access to catalogue Web UI

johnbywater — Sat, 21 Aug 2010 20:34:18 GMT

#485: Performance beats QoS criteria

johnbywater — Sat, 21 Aug 2010 21:13:39 GMT

#486: Catalogue service shall notify and query SOLR service

johnbywater — Sat, 21 Aug 2010 21:23:10 GMT

#487: Notify SOLR service of model events

johnbywater — Sat, 21 Aug 2010 21:25:10 GMT

#488: Catalogue service shall notify RDF service

johnbywater — Sat, 21 Aug 2010 21:27:01 GMT

#490: The system shall support registering a harvest source via the CKAN API

johnbywater — Sat, 21 Aug 2010 21:41:15 GMT

#491: Get form for creating harvest source entity

johnbywater — Sat, 21 Aug 2010 21:43:28 GMT

#492: Submit harvest source create form response to the API

johnbywater — Sat, 21 Aug 2010 21:44:02 GMT

#493: Get harvest source entity

johnbywater — Sat, 21 Aug 2010 21:44:43 GMT

#494: Get form for updating remote metadata entity

johnbywater — Sat, 21 Aug 2010 21:45:06 GMT

#495: Put form for updating remote metadata entity

johnbywater — Sat, 21 Aug 2010 21:45:50 GMT

#498: Search packages within location "bounding box"

johnbywater — Sat, 21 Aug 2010 21:50:12 GMT

#500: Exception from diff

dread — Mon, 23 Aug 2010 08:43:53 GMT

Investigate exception occured occasionally in last couple of days on ckan.net:

WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'key'
URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               response = self.app(environ, start_response)
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'zh-cn,zh-tw'
HTTP_CONNECTION	'close'
HTTP_HOST	'ckan.net'
HTTP_USER_AGENT	'Baiduspider+(+http://www.baidu.com/search/spider.htm)'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'123.125.66.32'
REMOTE_PORT	'63767'
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x9f603ec>
beaker.cache	<beaker.cache.CacheManager object at 0x9f6042c>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x9f602ac>>
beaker.session	{'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: >, '')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0x104552ec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xfb17aec>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xfb17aec>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x10286d4c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.logger	<logging.Logger instance at 0xa16e0cc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 167584972>, 'auth_tkt': <AuthTktCookiePlugin 169253516>}
routes.route	<routes.route.Route object at 0x9f3690c>
routes.url	<routes.util.URLGenerator object at 0xfd8d7cc>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x103a5bf0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xfd8d7cc>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})

#501: Read-only maintenance mode

pudo — Mon, 23 Aug 2010 09:10:17 GMT

CKAN should have a read-only maintenance mode with a nice little banner on all pages, appropriate REST messages etc. Bonus points if this is triggered via an environment variable and thus can be triggered by the surrounding apache.

#507: Upgrade a CKAN service

johnbywater — Tue, 24 Aug 2010 14:54:06 GMT

#511: Better warnings and errors when using API

dread — Wed, 25 Aug 2010 16:44:37 GMT

#512: User creates package via API with incorrect core fields specified

dread — Wed, 25 Aug 2010 16:45:50 GMT

#513: Creating a package with API should warn if there are incorrect fields

dread — Wed, 25 Aug 2010 16:46:42 GMT

When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK.

#514: Inconsistent use of 'location' header in API

dread — Wed, 25 Aug 2010 17:29:17 GMT

When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all.

I've removed the info about the header in the docs in the meantime.

#516: Double-escaped text visible - revision page

dread — Thu, 26 Aug 2010 13:32:19 GMT

At: http://ckan.net/revision you can see: "Next »" on pager This doesn't seem to be a problem with earlier (1.0.1) version.

#533: The performance of CKAN shall scale

johnbywater — Fri, 27 Aug 2010 12:01:39 GMT

CKAN uses a database. We can put the database on another machine and then clone the CKAN machine. Then there are two machines calling the database and converting SQL results into HTTP responses.

How many machines can you have before the database becomes the bottleneck? At that point, what QoS can be obtained with normal hardware?

Afterwards, can CKAN be made to scale further than this with a little development? Can its Postgres database be mirrored in realtime? Or can we change SQL writes (create/updates) to write to many databases?

Are there any other options for scaling the persistence mechanism?

#535: genshi error when logged into sl.ckan.net

dread — Mon, 30 Aug 2010 10:56:14 GMT

Genshi exception when rendering the page whilst logged in to sl.ckan.net.

#544: Backport facet browsing to CKAN 1.2

pudo — Tue, 31 Aug 2010 15:08:12 GMT

This is in IATI, would be nice to have in generic CKAN.

http://bitbucket.org/okfn/iati/changeset/378431974c76

#557: Remove siblings from link_to/from relationship display

pudo — Fri, 03 Sep 2010 19:36:35 GMT

Displaying siblings is not needed for link_to/from.

#558: Sort extras, relationships by key/object alphabetically

pudo — Fri, 03 Sep 2010 19:38:00 GMT

#561: Deleted packages are returned in the API

pudo — Mon, 06 Sep 2010 12:19:38 GMT

Anja is reporting this, severe bug, I think.

#562: OpenID Login fails using Yahoo and Google

pudo — Wed, 08 Sep 2010 10:18:39 GMT

reported by LOD folks.

subsidary bug: error messages from OpenID providers are not shown on the form return page.

#564: The system shall support location based search

johnbywater — Thu, 09 Sep 2010 12:03:49 GMT

#567: Post new harvest job for given harvest source

johnbywater — Thu, 09 Sep 2010 13:45:43 GMT

#573: Add metadata entity to harvesting queue

johnbywater — Thu, 09 Sep 2010 15:15:53 GMT

#574: Create UKLII package with attributes from remote metadata record

johnbywater — Thu, 09 Sep 2010 15:23:25 GMT

#575: The system shall notify remote metadata harvesting errors

johnbywater — Thu, 09 Sep 2010 15:59:19 GMT

#577: Get remote metadata harvest job errors

johnbywater — Sat, 11 Sep 2010 12:49:41 GMT

#578: Get remote metadata harvest job

johnbywater — Sat, 11 Sep 2010 12:50:07 GMT

#579: Delete remote metadata harvest job

johnbywater — Sat, 11 Sep 2010 12:50:40 GMT

#580: Write (create or update) CKAN package for metadata document

johnbywater — Sat, 11 Sep 2010 13:12:10 GMT

#582: Domain object class for harvest source

johnbywater — Sat, 11 Sep 2010 13:18:24 GMT

#583: Define fieldset object for harvest source forms

johnbywater — Sat, 11 Sep 2010 13:19:29 GMT

#584: Migration script to add harvest source table

johnbywater — Sat, 11 Sep 2010 13:20:04 GMT

#585: Method on API controller to return harvest source create form

johnbywater — Sat, 11 Sep 2010 13:21:42 GMT

#586: Routing configuration for harvest source create form API resource

johnbywater — Sat, 11 Sep 2010 13:22:16 GMT

#587: Documentation for getting harvest source create form from API

johnbywater — Sat, 11 Sep 2010 13:22:44 GMT

#588: Method on API controller to accept valid harvest source create form submission

johnbywater — Sat, 11 Sep 2010 13:26:15 GMT

#589: Method on API controller to reject invalid harvest source create form submission

johnbywater — Sat, 11 Sep 2010 13:27:00 GMT

#590: Documentation for submitting harvest source create form submission to API

johnbywater — Sat, 11 Sep 2010 13:27:39 GMT

#591: Routing configuration for harvest source entity API resource

johnbywater — Sat, 11 Sep 2010 13:34:42 GMT

#592: Methods to return harvest source entity as JSON

johnbywater — Sat, 11 Sep 2010 13:38:58 GMT

#593: Method on API controller to return remote metadata edit form

johnbywater — Sat, 11 Sep 2010 13:41:38 GMT

#594: Routing configuration for remote metadata edit form API resource

johnbywater — Sat, 11 Sep 2010 13:42:12 GMT

#595: Documentation for getting remote metadata edit form from API

johnbywater — Sat, 11 Sep 2010 13:42:52 GMT

#596: Method on API controller to accept valid remote metadata edit form submission

johnbywater — Sat, 11 Sep 2010 13:44:22 GMT

#597: Method on API controller to reject invalid remote metadata edit form submission

johnbywater — Sat, 11 Sep 2010 13:44:49 GMT

#598: List remote metadata entities for given publisher

johnbywater — Sat, 11 Sep 2010 14:02:10 GMT

#599: Routing configuration for listing remote metadata entities for a given publisher

johnbywater — Sat, 11 Sep 2010 14:03:34 GMT

#600: Method on API controller for listing remote metadata entities for a given publisher

johnbywater — Sat, 11 Sep 2010 14:04:14 GMT

#601: Documentation for listing remote metadata entities for a given publisher via API

johnbywater — Sat, 11 Sep 2010 14:05:28 GMT