{22} Trac tickets (2647 matches)

Results (1001 - 1100 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#961 enhancement kindly rgrp ckan-v1.5 closed fixed [super] Refactoring of forms, validation and model synchronization

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

  1. Generating (or just filling) a form template with 'form data' (and errors)
  2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
  3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

  • #926 - Pick a simpler form framework
  • #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
    • #1079 Refactor API to use new logic layer and dictization
    • #1078 Refactor WUI controllers and forms to use logic layer
    • cf existing dumper and importer code
    • This will fix #662
  • [not ticketed yet] - validation layer (should work on serialized objects?)
  • #662 - Can't put entity that is returned by posting to package register (Defect)
  • #972 - Merge 'extras' into main package dict rather than having separate key
  • #1035 - Form impressions are given IDs
  • #810 - Move "add packages" field up in group form (easier to do this once forms are done)
1297069849000000 1310126100000000
#1053 defect dread dread ckan-v1.5 closed fixed Deletion in Model API

Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted.

There is no way to delete objects at the moment - changes to 'state' are ignored in the API.

Do we need an alternative way to purge objects in the API?

1300790039000000 1310126546000000
#559 defect pudo pudo ckan-v1.5 closed invalid Fix package lookup on group edit form

Lookup is needed to add a package but won't show up at some times.

Originally reported by Anja Jentzsch, re questions.

1283542776000000 1310127694000000
#810 enhancement pudo pudo ckan-v1.5 closed wontfix Move "add packages" field up in group form

Subticket of forms super ticket #961

1289992287000000 1310128477000000
#910 enhancement pudo pudo ckan-v1.5 closed invalid Refactor CKANrdf to support extras

Extra fields should be represented in some manner.

1295266026000000 1310128544000000
#1009 enhancement pudo rgrp ckan-backlog new Improvements to user accounts sytem
  • Forgot password (email a new password)
  • Confirm email
  • Do not show register page if you are logged in (redirect to home page)
  • ticket:1010 - listing of users.
    • Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)
1298635991000000 1310128574000000
#358 enhancement rgrp dread ckan-v1.5 closed duplicate Resources in REST API

(spun out of ticket:336)

Resource added to model API at:


Example model request

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028


 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2


  1. Have it generic (ie. not per resource) and use an action/role on system
  2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
  3. Introduce Resource in authorization system (requires migration)

Mixed model

Create / Edit:

if resource associated to package:
    check_permissions(package, update)
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)
1277483282000000 1310128782000000
#764 enhancement thejimmyg dread ckan-v1.5 closed duplicate Read-only CKAN Web UI

Whilse using CKAN web interface, you are not tempted to edit stuff:

  • You know at all times this CKAN is read-only
  • All editing facilities are still seen but greyed-out with an indication why it is.
1288091619000000 1310133334000000
#1163 enhancement rgrp rgrp ckan-backlog new Improvements to Storage Extension

Storage is now working but there are

  • Integrate with Resources (e.g. create a resource for each file upload and give option to associate with a package)
    • Should we introduce rule that files *not* associated with a Resource are periodically deleted?
  • Allow setting of a file name/path before upload
  • Allow for file overwriting/deleting etc (how should this work -- do we want to allow this sort of thing)
  • Integrate local file upload stuff in api/auth/*

Different Backend Issues

Local file store is rather different from 'remote' storage in various ways:

  • For remote you don't want to use many buckets as there are bucket limits while for local you want to. Should we there have a single path that users provide which we then partition differently for different backends.
1306408778000000 1310133808000000
#972 enhancement sebbacon rgrp ckan-v1.5 closed wontfix Merge 'extras' into main package dict rather than having separate key

Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.


  • Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961

1297332282000000 1310134129000000
#941 enhancement thejimmyg rgrp ckan-v1.5 closed wontfix Submit apps or ideas for apps related to datasets (extension)

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

1296403099000000 1310134339000000
#1125 enhancement dread nils.toedtmann closed fixed Debian package "ckan" should not depend on "postgresql"

The debian package "ckan" with the two scripts "ckan-create-instance" and "ckan-instance-maintenance" depends against "postgresql". But "ckan-create-instance" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs.

Please add a flag "--without-local-db" to "ckan-create-instance" and remove the postgres dependancy from the debain package.

1304538095000000 1310134813000000
#1095 defect kindly kindly closed fixed add way to pass in schema to logic layer.

We need a way to pass in schemas to the logic layer to deal with edge cases.

1303221854000000 1310134959000000
#1215 defect dread closed fixed Edit Resources table - removing all rows prevents adding them again

I'm not sure if this was reported earlier, but I have found a minor bug in the "Add new package" form. As far as I can see this bug exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). This is a problem with adding new resources in "Resources" section. To reproduce the bug do the following:

  1. Go to "Add new package", for instance http://ckan.net/package/new
  2. Scroll down to "Resources" section.
  3. Click on "Remove this row" link (red circle).
  4. Say "Yes" to "Are you sure you wish to remove this row?"
  5. Now there should be no rows. Only the header "URL* Format Description"
  6. Click on "Add row to table"
  7. All you get now is another row with table header "URL* Format Description". You will not be able to add any resources now. Solution is to reload the page.

The simplest solution to this bug will be to hide the "Remove this row" link if there is only one row.

1310133224000000 1310385485000000
#1218 enhancement dread [email protected] ckan-sprint-2011-10-28 closed fixed Colour the History tab icon

Change the color of the "History" tab icon to yellowish, like the rest of the icons in the other tabs ("View" and "Edit", "Authorization").


When you view a data package, for instance http://ckan.net/package/thesaurus-w, you will see "View", "Edit", "History" tabs on the top. "History" tab has a black-and-white icon which makes it look like an inactive/disabled tab (since the text is also grayed out when the tab is not selected). This is not a major issue, but it is a little bit confusing for the users. This icon exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net).

The simple solution would be to change the color of the "History" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. "View" and "Edit").

1310375768000000 1310389390000000
#1199 enhancement pudo pudo closed fixed Implement an email function for users.

We need an email function in CKAN to accept messages sent to users. The basic signature will be:

  • mail_user(user_obj, subject, body, mime_type='text/plain', headers={})

This has a number of use cases:

  • Retrieval of lost passwords
  • E-Mail confirmation

Finally, the mail function should be exposed in the API for sysadmin clients. This way we can have scripts traverse CKAN for 404s, invalid data or missing fields and ping users about that automatically (requires traversal by revision, not package, to get the associated users).


Note we have already written code like this (*and* tested it) in isitopen:

1308821986000000 1310555589000000
#1186 enhancement dread closed fixed Password reset facility

You can register a user with password and (optional) email address. But if you forget the password you can't then log in again. We need a password reset facility that sends and email with a new password.

1308142766000000 1310556519000000
#1221 defect dread dread ckan-sprint-2011-10-28 closed fixed Password reset tidy up
  • No tests.
  • 500 error given when user is not found.
1310475661000000 1310556607000000
#1222 defect dread dread ckan-sprint-2011-10-28 closed fixed Mailer has no tests 1310479136000000 1310556618000000
#1128 task dread dread ckan-sprint-2011-10-28 closed fixed Upload Scotland gov data

Upload to ckan.net: https://sites.google.com/site/scotlandsdata/

1304712293000000 1310568028000000
#1205 defect kindly dread closed fixed db upgrade problem on cz.ckan.net

When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below).

cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan.

In the meantime I've reverted the db to the old version (pre-upgrade).

cz.ckan.net is not working properly until this is fixed.

[email protected]:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://<user>:<pass>@<host>/<dbname>
  module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects
2011-06-29 09:07:41,294 INFO  [migrate.versioning.api] 28 -> 29... 
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname
  ret = fn(self, con, *args, **kw)
Traceback (most recent call last):
  File "/usr/bin/paster", line 18, in <module>
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 218, in run
    result = self.command()
  File "/usr/lib/pymodules/python2.6/ckan/lib/cli.py", line 84, in command
  File "/usr/lib/pymodules/python2.6/ckan/model/__init__.py", line 163, in upgrade_db
    mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 185, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 364, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py", line 83, in runchange
    change.run(self.engine, step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py", line 140, in run
  File "/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py", line 152, in upgrade
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1788, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1191, in execute
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
    return self.__execute_context(context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1302, in __execute_context
    context.parameters[0], context=context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py", line 299, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "package_group_revision" violates foreign key constraint "package_group_revision_continuity_id_fkey"
DETAIL:  Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table "package_group".
 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'}
1309339700000000 1310568631000000
#1209 defect kindly closed fixed sort out rendering of formalchemy package preview

Got broken due to the moderated edits changes #1141.

Need to make sure preview works.

1309817240000000 1310571715000000
#1223 enhancement pudo pudo closed fixed Caching of static files

StaticURLParser can have caching - use it

1310573854000000 1310573893000000
#1225 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Tool for mass tag changes

Using ckanapi

1310642161000000 1310642407000000
#1219 defect timmcnamara closed fixed CSS issues on IE7

As reported on ckan-dev:

items in the footer of CKAN ("Packages", "Groups & Tags", "About", "Language", etc.) are shown vertically instead of horizontally in IE7. This works fine in later browsers like IE8, IE9, FF4, and latest Opera and Chrome.

This seems to exist in all recent CKAN versions up to 1.4.1a.

1310423688000000 1310740534000000
#1230 enhancement kindly ckan-sprint-2011-10-28 closed fixed Make sure plugins tables work in tests.

Dgu tests failed as harvesting table are not created at the correct point.

1311100811000000 1311154142000000
#861 defect wwaites wwaites closed invalid unicode errors creating rdf output


1291726125000000 1311168845000000
#651 requirement pudo closed fixed Improve javascript helper for package/user addition on groups/authzgroups

This needs to be abstracted and we still need to be able to remove items before saving.

1284972211000000 1311171104000000
#1094 enhancement thejimmyg thejimmyg ckan-v1.5 closed duplicate [super] Refactor the Auth System

Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787.

Two man reasons for the changes are:

  • We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer.
  • Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able.

The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package).

  1. The first thing the is_authorized method is a hook to a plugin

which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!)

Reason: authorization can be completely delegated to another system (or partially)

  1. is_authorized method currently takes (username, action, object)

but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model.

  1. Rename these two classes to better reflect what they are
  1. Rename the Editor role to PriveledgeUser? since Editors sometimes can't edit.

Although this sounds a bit radical we already have auth extensions.

Read-only CKAN Web UI

(Additional requirement from #764)

Whilse using CKAN web interface, you are not tempted to edit stuff:

  • You know at all times this CKAN is read-only
  • All editing facilities are still seen but greyed-out with an indication why it is.
1303117973000000 1311173649000000
#1118 defect johnlawrenceaspden closed invalid tests are testing something other than the behaviour seen in the browser

I'm finding that if I try to take an action with insufficient credentials from a test then I often (but not always) get a 401 error, whereas in the browser I get redirected to the login page.

It's a bit worrying that the program in its test environment doesn't behave like it does in the browser.

1304093017000000 1311174062000000
#96 defect rgrp rgrp closed invalid info command must give up to date information for python-type distributions

(2008-09-10) at present when setup.py is edited info command may not give up to date information if pkg-info (in egg.info) is not rebuilt.

Suggest: info command needs to rebuilds pkg-info in python-type distributions.

1249982410000000 1311176063000000
#200 defect rgrp rgrp closed invalid PythonDistribution write method should write all available metadata

Current write method just creates a bare bones package and does not write any of the available metadata other than name.

1259229583000000 1311176118000000
#312 defect rgrp dread closed invalid Race condition creating PackageRating

On ckan.net there are a few packages which cause a 403 exception when you try to rate them: http://www.ckan.net/package/rate/coins-data?rating=3

The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() InvalidRequestError?: Multiple rows returned for one()

It looks like this package got rated twice in quick succession, creating two similar PackageRating? objects. This race condition needs to be solved and these particular rating objects repaired.

1273482785000000 1311176173000000
#362 defect dread dread closed invalid Ratings should not be created with a GET


In the Web UI, when you rate a package it simply links to something like:


This creates a GET request.

This is bad because:

  • Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this)
  • There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept? GET> Apache instruction in. But the database may still get written for these ratings.
  • Best practise for web requests is for GET to be a read-only request.
1278925451000000 1311176564000000
#668 defect thejimmyg Colin Calnan closed invalid Caching issues on API v1

It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues.

Steps to test:

  1. Modify a dataset on datadotgc.ca, redirects to CKAN
  1. On save, redirects to http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100
  1. You can see that the Dataset has not updated correctly. Run a check on the API v1 - http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present
  1. Check the v2 of the API - http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present.
  1. Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either.
1285953542000000 1311176649000000
#460 defect pudo johnbywater closed fixed State field changed by non-sysadmin

This package:



  1. created by Richard (logged-in)
  2. edited by Richard (logged-in)

(According to the logs, at this point the state was changed from 'active' to 'deleted') -- RP was it set to 'deleted' or just ?

  1. pudo changed the state back to active

Similarly an incident with bibbase package where field set to (see http://ckan.net/revision/diff/bibbase?diff=702bb0a3-03b7-49ac-87ad-e489c414962f&oldid=5447842d-b6ed-41d9-9cfd-8bb73b85c409)

Need to investigate how this got changed, fix if necessary and report back to Richard. Note that package 'admins' as well as sysadmins can change the state of a package (though note that bibbase did not appear to have an owner).

Suggested solution (for setting to ):

  • Ensure in ckan/forms.py that there is a validator for state field that ensures only set to valid values.
  • Check that we do not allow state to be changed in the api except by package owner or sysadmin
1282300002000000 1311176868000000
#812 defect cygri ckan-backlog new Package edit form only allows three extra fields


The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview).


A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

1289995010000000 1311176917000000
#855 defect pudo dread closed invalid Authenticator tests

The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests.

1291723967000000 1311176988000000
#856 defect wwaites dread closed invalid Tests for caching

Test coverage for ckan.lib.cache is only 40%. Can that be usefully improved?

1291724064000000 1311177085000000
#859 defect pudo dread closed invalid Tests for User model

Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement?

1291724348000000 1311177461000000
#355 defect rgrp rgrp closed fixed Dashes versus underscores in package names

Sort out how we deal with dashes versus underscores in package names.

1277221996000000 1311177552000000
#987 defect pudo pudo closed duplicate Common harvesting framework

We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include:

  • CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
  • Catalogue scraping for LOD2 experiments (yields RDF graphs)
  • Atom/DCat for LOD2 production (yields RDF graphs)
  • OAI-PMH for http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios.

In general, such a system would have the following stages:

  • Source selection: find what to download/scrape/harvest/parse
  • Index retrieval (i.e. package index)
  • Item retrieval (i.e. package entity)
  • (Optional: Serialization)
  • Normalisation
  • Loading/Merging? into CKAN

Exisiting harvesters are at:

1297684756000000 1311177705000000
#225 enhancement rgrp dread closed invalid UI Review - swap URL order

Not so keen on URLS such as /package/edit/mypackagename. These seem to be task-oriented rather than resource oriented (which would be /package/mypackagename/edit), and unstable: there is a common root for /package/edit/mypackage and /package/history/mypackage which is not reflected in the URL structure.

1260880487000000 1311178276000000
#1167 task nils.toedtmann closed fixed Create a standard CKAN system image for Amazon EC2 (AMI)

We should create a public AMI with CKAN pre-installed and configured such that users can easily create their own EC2 machine with a running CKAN to play with.

There are three phases:

  1. [nils] Deploy an empty EC2 instance to become the CKAN image master instance
  2. Install a CKAN and give it a standard configuration.
  3. [nils] Create a AMI from the CKAN image master instance and publish it.

I am happy to do first and last. Who is installing and configuring CKAN?

Unfortunately AMIs are specific to region, architecture and storage type. We cannot maintain too many images, so a number of choices have to be made:

  • Which distribution/version? Ubuntu 10.04 LTS
  • Which architecture/instance-type? I suggest 64-bit/t1.micro
  • Which region? I suggest us-east-1 and maybe eu-west-1
  • Which storage type? EBS (way easier to make an AMI from than instance-store)
  • Install CKAN from deb packages via mercurial/virtualenv? I assume the latter because the AMI is targeted to developers?

1 2

1306423016000000 1311178516000000
#543 task wwaites rgrp closed duplicate Investigate partial page caching and edge-side includes

Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page.

Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish.

Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat.

1283244784000000 1311178918000000
#537 task wwaites wwaites closed duplicate Caching and Performance improvement

There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests.

Introducing caching has to be done carefully and should be done in a standards compliant manner.

General strategy

  • Where possible, cache output within the pylons app (beaker).
  • Facilitate external caching in an end-user's web browser or a caching proxy
  • Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes).
  • When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data.


These tasks should be broken into sub-tickets:

  • caching of parts of templates that are expensive to render (package list, tag list, group list)
  • caching of entire output using beaker particularly for API read operations.
  • need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches.
  • cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: http://www.mnot.net/cache_docs/#CACHE-CONTROL
    • Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well
    • Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner

Future Work

  • Investigate ckanclient library maintaining a local cache as a web browser would
  • Investigate using a CDN like Google Storage or Amazon for serving cached data.
1283184362000000 1311178929000000
#995 defect kindly kindly closed wontfix consolidate caching

Try and minimise the amount of different caching methodologies, especially for api calls.

1298283499000000 1311179009000000
#794 requirement amercader johnbywater ckan-backlog assigned Investigate reconciling UKLP Publisher and Provider with DGU

This needs more analysis, but the GEMINI2 attribute "metadata point of contact" must be reconciled with the registered publisher (or agent).

This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider.

1289227811000000 1311179581000000
#1204 enhancement timmcnamara closed invalid renaming packages seems to result in a server error

When renaming a package, and doing other changes at the same time an error is reported. However, the changes seem to go through fine.


  • datacatalogs.org as of 29 June 2011
1309303816000000 1311179980000000
#1200 enhancement pudo closed duplicate Port PDEU theme to CKAN.net

PDEU has a newer layout which could be adapted to also be used on ckan.net. For this, things should be cleaned up, copied to the ckanext-ckan.net repo as needed and re-colored the core CKAN color scheme.

Maybe we could have this coincide with a CKAN rename?

1308824017000000 1311180218000000
#830 enhancement rgrp closed fixed Themes for CKAN

Already have the basic capacity using *_extras paths for templates and public directory (see http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: http://packages.python.org/Flask-Themes/

1290760612000000 1311180263000000
#103 enhancement kindly rgrp ckan-v1.5 closed fixed View a package at a given revision

As a user I want to view a package at a given revision:

  • When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
  • package history page should provide links to these pages

Cost: 2h

1251289897000000 1311180850000000
#990 defect rene.kapusta closed fixed umlauts in title / text

It's not possible to use umlauts (like "CKAN Österreich") in the title / text.

1297801036000000 1311180850000000
#104 enhancement rgrp rgrp closed fixed View a package at a given version

As a user i want to see a package at "version" X (NB: not revision X).

  • When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version)
  • Implementation:
    • Find revision for this version (search revision history for when version field was last 0.7)
    • Show pacakge at that revision (as in ticket:103)
  • On history page also shows versions in list of revisions associated with the package
1251292672000000 1311181303000000
#165 enhancement rgrp dread closed invalid Use 'tag:' instead of 'tags:' for wui package search

Because people might try and do "tags:gov sweden" when "tag:gov sweden" makes more sense (sweden is searched for in all fields in these cases).

But in REST i/f search allow "tag"="gov" OR "tags"="gov sweden" in parameters.

1256200526000000 1311181391000000
#249 enhancement rgrp dread closed invalid Regex search

Search of package name and title (and other fields) using regular expressions.

Current example use-case: Wanting to specify packages with title beginning with 'B'.


  • Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields?


  • Postgres reg ex searching detailed here:


1265994509000000 1311182450000000
#857 defect dread closed invalid Tests for CLI

Test coverage for ckan.lib.cli is 34%. Need to improve that.

1291724131000000 1311182581000000
#1235 enhancement thejimmyg ckan-future new [super] Search Improvements

Child tickets:

  • #234 UI Review - Autocomplete package names & tags in search
  • #193 Searching by time-related field
  • #191 Searching by modification date
  • #905 Unable to search with accented characters in package names
  • #906 Ability to search without accents for accented words
  • #924 Search box has no search button

Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two.

1311182641000000 1311182641000000
#447 task memespring johnbywater closed fixed Presentation on Package edit form (labels IE7, horiz alignment in chrome)

Follow up with NS and RP.

1282299423000000 1311182895000000
#449 task wwaites johnbywater closed invalid Start email thread about RDF in CKAN with JF and WW 1282299482000000 1311182945000000
#659 enhancement nils.toedtmann dread closed fixed Check CKAN instance works automatically

Auto way to check web and API interface of a CKAN instance basically works. Several gotchas can be quickly determined, such as logging in, search not working. Needs to be configurable per site basis.

1285348333000000 1311183031000000
#660 requirement dread closed invalid Check CKAN instance works

As an admin I want to check a CKAN instance works having just upgraded it or configured it.

1285348463000000 1311183115000000
#1234 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Setting locale error conditions handled badly

Needs tests for error conditions for bad locale and redirect parameters.

1311179871000000 1311183328000000
#1228 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Display OpenID login errors

When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again.

1311086562000000 1311325161000000
#820 defect dread closed fixed Exception browsing /user/apikey when not logged in

Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in.

WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'apikey' 						

URL: http://test-hmg.ckan.net/user/apikey
1290006079000000 1311325226000000
#1207 enhancement dread dread closed fixed ckanclient.package_entity_get should raise more specific exception

When package does not exist in ckan catalogue, ckanclient.package_entity_get should raise more specific exception, such as CkanNotFoundError? instead of generic CkanApiError?.

1309515582000000 1311325343000000
#283 enhancement rgrp dread closed wontfix Manage deletions of unwanted packages

Use case

As a user I want to notify the CKAN admins of a spammed or unsuitable package for deletion.

Suggested solution

In the package view side-bar, there is a note: "To have this package completely removed, contact the [ca.ckan.net administrators [email protected]…]."

Other solutions

A more complicated solution would be to allow packages to be tagged for deletion, which would auto-alert administrators, and allow easier administration of this. But this might be overkill.

c.f. http://en.wikipedia.org/wiki/Deleting_an_article

1270660210000000 1311325526000000
#1238 defect kindly dread ckan-sprint-2011-10-28 closed invalid Viewing a package @date misses non-PackageRevisions

There's a problem with getting old versions of packages using @date. The query looks for the PackageRevision? related to the date, but ignores TagRevisions? etc.

So for example if you compare this table: http://ckan.net/package/history/osm with this view: http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830 you actually see the 2010-01-13 11:13 revision - the wrong revision.

1311351303000000 1311759784000000
#1183 defect johnglover nickstenning ckan-sprint-2011-10-28 closed fixed Downloads "Preview" button doesn't preview.

The "Preview" button is a nice idea, but it doesn't seem to actually "preview" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file.

This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of application/x-some-junk-here in order to force a download.

1307701669000000 1311771069000000
#888 enhancement johnglover rgrp ckan-sprint-2011-10-28 closed fixed Improvements to the dataproxy and the data API

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

  • Testing for dataproxy
    • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
  • Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
  • Google docs style row/column selections
  • Use the swiss library - https://bitbucket.org/okfn/swiss
    • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
  • Handle redirects for content-length?
  • Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

  • Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}
1293649783000000 1311773103000000
#1062 defect johnglover sebbacon ckan-backlog assigned Data preview encoding error

The preview of "Species Misc Turtle Download" at http://ckan.net/package/taxonconcept results in the following error:

Unable to Preview - Had an error from dataproxy: Data Transformation Error (Data transformation failed. Reason: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte

1301396143000000 1311773731000000
#1150 defect johnglover timmcnamara ckan-sprint-2011-10-28 closed fixed Non-ASCII chars prevent data preview

Characters outside of ASCII range are not supported within data previews.

Steps to reproduce:

  1. Visit http://ckan.net/package/kivele2010
  2. Click on [preview] for any of the resources
1306019914000000 1311774141000000
#1123 requirement dread nils.toedtmann closed fixed Please re-package CKAN packages as "noarch"

... or, if the CKAN packages do contain architecture-specific binary code, build packages for i386 too.

Currently, http://apt-alpha.ckan.org/debian only offers packages for amd64, but e.g. "m1.small" EC2 instances are i386.

We would need this in order to migrate the community instances to a packaged based CKAN.

Rufus, pls prioritise.

1304530050000000 1311863806000000
#1246 defect pudo [email protected] ckan-sprint-2011-10-28 closed fixed Search results on ckan.net are mistakenly all 'open'

All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: http://ckan.net/package

1311863353000000 1311892816000000
#1247 task rgrp dread ckan-sprint-2011-10-28 closed fixed Put docs on ReadTheDocs

Put docs on ReadTheDocs? instead of eu13.

1311941948000000 1311955185000000
#1249 enhancement pudo rgrp ckan-v1.5 closed fixed Exclude script tag from extraction for i18n

Currently have this script section put in for i18n. It shouldn't be.

<script type="text/javascript">
    $.fn.ajaxCreateSlug = function(name, url) {
        var title = this;
        var updater = {
            init: function(title, name) {
                // Add a new element where the validity of the package name can be displayed
                this.name_field = name;
                this.title_field = title;
                this.name_field.parent().append('<div id="package_name_valid_msg"></div>');
                this.url = url;
1311958536000000 1311959356000000
#1195 defect rgrp dread ckan-sprint-2011-10-28 closed fixed Storage extension to handle incorrect routes better

We've seen exceptions caused by people getting the file storage link wrong, adding a slash to the end.

e.g. http://ckan.net/storage/f/file/e696d857-e997-41c8-be08-408697168ca8/

We should aim to handle all requests better than causing exception. 404 (or even 302 in this case) is better than 500.

1308576769000000 1311964374000000
#1192 task annapowellsmith annapowellsmith ckan-sprint-2011-10-28 closed fixed Convert CKAN Sphinx docs into admin/reference manual

As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

  • An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install.
  • A Reference Manual which is the primary source of reference for CKAN software - this includes API docs.

The current chapters of the Sphinx docs should be moved as follows:

  • index.rst - copy some stuff from README.txt, keep short
  • README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst
  • [NEW] install.rst - new file on installation
  • [NEW] theming.rst - move over from wiki
  • api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
  • i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
  • design.rst - REMOVE (can copy some over if you can be bothered to http://wiki.ckan.net/Vision (all philosophical stuff should go!))
  • loaders.rst: move to wiki.ckan.net/Using_Loaders
  • feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
  • importer.rst: remove (don't even copy)
  • forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) http://wiki.ckan.net/Customizing_Forms
  • forms-integration.rst: remove
  • model.rst: leave but move to reference (only for core devs)
  • load-testing.rst -> move to http://wiki.ckan.net/Load_Testing
  • distributed.rst -> remove
  • admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
  • deb.html -> go into reference (query james gardner on list about moving this to wiki?)
1308324593000000 1312191462000000
#1208 enhancement rgrp rgrp ckan-sprint-2011-10-28 closed fixed Webstore: a datastore with web API
  • SQLite based
  • Use Cyclone for async http server on top with auth
  • Designed to not be specific to CKAN or anything else

Current code is here: https://github.com/okfn/webstore.

API Spec


Two basic ways to query::

    GET: /{owner}/{db-name}/?sql=...
    GET: /{owner}/{db-name}/?table=...&attr=value&attr=value&limit=...


      u'keys': [u'id', u'name'],
      u'data': [
          [1, u'jones'],
          [u'aaa', u'jones']


POST to::


Payload is json data structured as follows::


unique_keys: [list of key attributes] data: {dict of values}


Authentication and Authorization

Authentication: use basic auth header.


  • Default: all read, owner can write
  • Restricted: owner can read and write, everyone can do nothing

Possible future: config file can specify a python method (TODO: method signature)

Integration with Other Systems

TODO: Specify how to delegate authenatication to user database in some other system.

1309804460000000 1312191646000000
#1261 defect pudo ckan-backlog new Investigate dots in extras search

It seems that searching for extras_foo:value works with solr, but extras_foo.bar:value doesn't. No theory on why.

1312366768000000 1312366768000000
#1250 enhancement pudo rgrp ckan-v1.5 closed fixed Search results should be sorted by score rather than alphabetical

At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases.

Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard).


  • Default this in solr (no need to touch code) but fragile and affects everything ...
  • Do it in code and default to score
  • Do it in code and have alphabetical (on name or title?) when no criteria otherwise score

Aside: may also wish to support search in query api but that is for later!

1311964752000000 1312370201000000
#1041 enhancement thejimmyg thejimmyg ckan-backlog assigned Start Using the CKAN Wiki for Tutorial-style documentation

For example, I will document the following:

I'd love if someone else would write:

  • An authorisation tutorial covering the core model, the command line tools and examples of every possible way of using the system
  • A HOWTO guide with screenshots for adding a package
1300284715000000 1312372367000000
#143 enhancement thejimmyg dread ckan-backlog assigned Most active users listed on homepage

Display league of users' recent activity on homepage.

1255010373000000 1312372381000000
#1239 defect rgrp rgrp ckan-sprint-2011-10-28 closed fixed Remove and re-add tag on package not working

See reports on list at:

Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix.

1311441020000000 1312453156000000
#1140 defect dread ckan-v1.5 closed fixed Adding the package to the group is not search indexed

To reproduce:

  1. paster db init
  2. paster create-test-data
  3. In Web UI: create new group 'tilo', which includes package 'annakarenina'
  4. curl http://localhost:5000/api/search/package?groups=tilo results in 0 results (WRONG)
  5. paster search-index rebuild
  6. curl http://localhost:5000/api/search/package?groups=tilo results in 1 result
1305718290000000 1312537257000000
#1265 defect dread dread ckan-sprint-2011-10-28 closed fixed user/edit error conditions cause exceptions

We get exceptions on these occasions:

  • user/edit if not logged in
  • user/edit/bob if bob is not a known user

These are a problem only on 1.4.3b.

Introduced in #1229 - merge of branch feature-1229-db-out-of-controllers

1312565788000000 1312566052000000
#1266 defect dread dread ckan-sprint-2011-10-28 closed fixed Timestamps without microseconds causing exception

I'm not sure why some timestamps are missing microseconds in the ckan.net data, but plenty seem to. This causes this exception when viewing the package rss feed:

URL: http://ckan.net/package/history/sfk_publishing?format=atom&days=7
In Module ckan.lib.helpers:227 in date_str_to_datetime
WebApp Error: <type 'exceptions.ValueError'>: time data '2008-04-13T20:40:20' does not match format '%Y-%m-%dT%H:%M:%S.%f'

Only seen with ckan 1.4.3b

1312794831000000 1312797477000000
#1267 defect dread dread closed fixed Random exception: cannot set the body to a unicode value

Occasionally see this problem when producing an error page. Here's an example

WebApp Error: <type 'exceptions.TypeError'>: You cannot set the body to a unicode value without a charset
URL: http://at.ckan.net/authorizationgroup/new
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               response = self.app(environ, start_response)
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:117 in __call__
<<          # available in environ['pylons.routes_dict']    
                   return WSGIController.__call__(self, environ, start_response)
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:284 in __call__
<<              if log_debug:
                       log.debug("Calling Response object to return WSGI data")
                   return response(environ, self.start_response)
               if log_debug:
>>  return response(environ, self.start_response)
Module webob.exc:248 in __call__
<<              return []
               if not self.body and not self.empty_body:
                   return self.generate_response(environ, start_response)
               return Response.__call__(self, environ, start_response)
>>  return self.generate_response(environ, start_response)
Module webob.exc:239 in generate_response
<<              status=self.status,
               return resp(environ, start_response)
>>  content_type=content_type
Module webob.response:94 in __init__
<<                  if charset is None:
                           raise TypeError(
                               "You cannot set the body to a unicode value without a charset")
                       body = body.encode(charset)
                   self._body = body
>>  "You cannot set the body to a unicode value without a charset")
TypeError: You cannot set the body to a unicode value without a charset
1312889084000000 1312897724000000
#1259 enhancement johnglover pudo ckan-backlog new "Add a row" for Extras on Package form

The default package form offers 4 empty extras fields. Like the resource section, it should have an "add more" button to add another row.

1312302693000000 1312907056000000
#1270 defect dread dread ckan-sprint-2011-10-28 closed fixed Default site www.ckan.net is a redirect and loses POSTs

The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters:

In [1]: import ckanclient

In [2]: c = ckanclient.CkanClient()

In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api')

In [4]: c.package_search('', search_options={'groups':'openspending'})
{u'count': 2102,
 u'results': <generator object _result_generator at 0x10168b1e0>}

In [5]: c2.package_search('', search_options={'groups':'openspending'})
Out[5]: {u'count': 29, u'results': <generator object _result_generator at 0x10168b410>}

Thanks to borior for finding this and raising it.

1312911847000000 1312912188000000
#1255 enhancement kindly kindly ckan-backlog assigned Drupal consistancy checks.

Make a robust way to make sure the drupal database is consistent with the ckan data.

1312219968000000 1313400054000000
#1253 enhancement amercader amercader ckan-sprint-2011-10-28 closed fixed Authz refactor

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

1312199491000000 1313401367000000
#1260 enhancement pudo ckan-backlog new Remove duplicate functions from _util.html

There seems to be both a list view for dictized and non dictized data structures for all entities in _util.html at the moment. Probably in the back of someone's mind already, but cleanup here would be nice.

1312366652000000 1313401499000000
#1203 defect johnglover rolf ckan-backlog new Moderated edits: html code shows as "changed" although it is not

I've installed the Moderated Edits extension (ckanext-moderatededits) and am editing a package imported from IATIregistry.org, with an extra field which contains a bit of HTML.

The editor indicates the field has changed, although the content hasn't (see screenshot). All I can find so far is a minor difference: in the field content, there is a code &#8212 and in the rendered table that is an &mdash;

1309274970000000 1313401579000000
#1254 task dread dread ckan-sprint-2011-10-28 closed fixed Feature sheet

Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers.

1312211925000000 1313401629000000
#1152 enhancement amercader amercader ckan-backlog new True support for generic CSW servers

The CSW harvesters implemented at the moment were developed with the DGU project in mind, and they assume all remote CSW servers to implement the Gemini 2 specification. Gemini 2 is the profile defined in the UK for INSPIRE complying metadata, so obviously catalogs from other countries or non-INSPIRE complying ones won't be able to be harvested.

The changes needed to support generic CSW servers (i.e. those implementing the ISO 19139 profile) are:

  • Handling the validators (right now are hardcoded in the harvester

code). This probably involves issues discussed in the CREP 3 (ticket #1134)

  • Changes in the model to adapt the specification to ISO 19139
  • Renaming objects and classes which are now Gemini-centric

List of CSW servers tested:


1306141334000000 1313411822000000
#1273 requirement amercader ckan-backlog new Create docs for API v3 1313412083000000 1313412083000000
#1278 enhancement amercader ckan-backlog new Refactor authorized_query calls

There are some functions that still use the Auhtorizer().authorized_query method:

./ckan/controllers/authorization_group.py:24:        query = ckan.authz.Authorizer().authorized_query(c.user, model.AuthorizationGroup)
./ckan/lib/base.py:237:        groups = ckan.authz.Authorizer.authorized_query(c.user, model.Group, 
./ckan/lib/search/sql.py:55:        q = authz.Authorizer().authorized_query(username, model.Group)
./ckan/lib/search/sql.py:118:        q = authz.Authorizer().authorized_query(self.options.get('username'), model.Package)
./ckan/logic/action/get.py:154:    query = Authorizer().authorized_query(user, model.Group, model.Action.EDIT)

./ckan/tests/test_authz.py:158:        q = self.authorizer.authorized_query(self.notadmin.name, model.Package)
./ckan/tests/test_authz.py:353:        q = self.authorizer.authorized_query(self.notmember.name, model.Package)
./ckan/tests/test_authz.py:357:        q = self.authorizer.authorized_query(self.member.name, model.Package)
./ckan/tests/functional/test_authorization_group.py:44:        group_count = Authorizer.authorized_query(u'russianfan', model.AuthorizationGroup).count()
1313415177000000 1313415177000000
#1274 enhancement johnglover johnglover ckan-sprint-2011-10-28 closed fixed Testing solr search

Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend.

1313413202000000 1313429049000000
#1271 enhancement rgrp rgrp ckan-sprint-2011-10-28 closed fixed CORS support

CORS - http://www.w3.org/TR/cors/ - support.

This is what you do in Apache. Should do this in lib/base.py or similar.

    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Methods "POST, PUT, GET, OPTIONS"
    Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type"

    # Respond to all OPTIONS requests with 200 OK
    # This could be done in the webapp
    # This is need for pre-flighted requests (POSTs/PUTs)
    RewriteEngine On
    RewriteRule ^(.*)$ $1 [R=200,L]
1313241839000000 1313753663000000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Note: See TracReports for help on using and creating reports.