{22} Trac tickets (2647 matches)

Results (901 - 1000 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#311 defect dread rgrp v1.1 closed fixed Reordering of package resources can lead to integri

Created a new package resource line and then moved it up above existing one and hit save. Result: 500 error. In logs have:

[Sat May 08 21:55:41 2010] [error] [client] Error - <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "package_resource_revision_pkey", referer: http://ckan.net/package/edit/cofog
[Sat May 08 21:55:41 2010] [error] [client]  'INSERT INTO package_resource_revision (id, package_id, url, format, description, hash, position, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(url)s, %(format)s, %(description)s, %(hash)s, %(position)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'hash': '', 'description': 'The Treasury record of COFOG functions. ', 'format': 'XLS', 'url': 'http://www.hm-treasury.gov.uk/d/cofog_definitions_coins250609.xls', 'package_id': '8482334d-fe2e-4285-9114-5243130f80c0', 'state': 'active', 'continuity_id': '8bf302db-8a80-47d3-b5dc-bc07512a3928', 'position': 3, 'revision_id': 'e4e2cb2d-4bd5-414e-b646-e484f174d9ab', 'id': '8bf302db-8a80-47d3-b5dc-bc07512a3928'}, referer: http://ckan.net/package/edit/cofog
1273348968000000 1274282065000000
#315 defect dread rgrp v1.1 closed fixed Improvements and fixes to csv dump
  1. Issues with quote in fields: http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html
  1. Issues with package resource serialization into csv table.

<quote> In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent. </quote>

1274100887000000 1275846764000000
#319 defect dread dread closed fixed Local Authority license misnamed in migration script 18

Migration script 18 converts license "OKD Compliant::Local Authority Copyright with data.gov.uk rights" into "uklocalauthority-withrights" when it should be "localauth-withrights", according to the license service SoS 2. This causes clients difficulties looking it up.

1274349714000000 1274366882000000
#320 enhancement dread rgrp v1.1 closed fixed site_title configuration variable which is used in template

As a sysadmin I want to configure basic site title information for use in the site templates.


  • ckan.site_title config variable
  • set this on g in app_globals.py e.g.
    • from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
  • use in head title and in main site title/logo section (use it as alt on logo image)
  • Also all other pages (e.g. index, about) which talk about CKAN
    • Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?


  • Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
  • Probably yes, but not part of this ticket.
1274378967000000 1279130535000000
#322 enhancement dread dread v1.1 closed fixed Client interface for Notification Service

Use cases

  • Register for package changes
  • Register for all revisions
  • Notified of a package change
  • Notified of a revision
  • Deregistration
  • Configuration of port in pylons config


  • Default port: 5672 (standard for AMQP)
  • Exchange name: 'ckan'
  • Exchange type: topic exchange (most flexible)
  • Routing keys: (see below)

Routing detail

Routing key format: "OBJ_TYPE" (NB tags should be identified by their name, not ID)

Example routing keys

  • 'package' - Package edited/created
  • 'resource' - Resource edited/created
  • 'revision' - Any change
  • 'db.clean'
  • 'db.rebuild'

Example queue bindings that clients may use:

  • * - no filtering - client receives all notifications
  • package - only changes to packages
  • revision - all revisions
  • db - all database operations


Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps?


  • How to use
  • simple example of an external client?
1274720042000000 1277722821000000
#323 enhancement dread dread v1.1 closed fixed Notification message

Which events to notify on

Listed by domain object, these are the notification message 'change types' that will be sent:

  • Package
  • PackageResource

Also it is clear that it could be useful to know when db-wide maintenance is carried out:

  • db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration)

Ignored domain objects

These parts of the domain model will not carry notifications as no use case has been identified for them:

  • Revision
  • Group
  • Tag
  • Rating
  • User - list of users is sensitive info
  • Relationships - complicated
  • Authz - complicated and sensitive info
  • License - change of a license's metadata is a question for the 'license service'

Message format

A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity.

For the 'db' notifications there shall be no payload.

1274723333000000 1278578841000000
#324 enhancement dread dread v1.1 closed fixed Search indexing using notifications

Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead.

The indexing shall run in a separate shell/process, managed by supervisord.

1274723483000000 1278599927000000
#325 enhancement dread dread v1.1 closed fixed Event push notification

As a

CKAN client program

I want to

be notified when changes to the CKAN metadata occur.

Examples of use

  1. An external search engine needing to (re)index a package. (interest: Package)
  2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333.
  3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: PackageResource)
  4. Do some processing on resource (e.g. extract sample data for display) (interest: PackageResource)


The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling.


Split-off into two tickets:

  • Notification message - ticket:323
    • Which events to notify on
    • Message format
  • Interface for Notifier Service - ticket:322


To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine.

1274723512000000 1278599979000000
#326 task dread dread v1.1 closed fixed Centralise importation of json library

Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers.

1274784223000000 1274789296000000
#328 enhancement dread rgrp v1.1 closed fixed Mention code libraries in API documentation

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

  • Python: ckanclient
  • Perl: luke closs wrote something
  • PHP: drupal library?
  • ...?
1274988106000000 1275318745000000
#330 defect dread dread closed fixed getdata/ons timezone not recognised on non-British servers

The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error:

DateConvertError?: Could not read date as ISO format "%a, %d %b %Y %H:%M:%S %Z". Date provided: "Mon, 04 Jan 2010 09:30:00 BST"

It turns out that it only recognises local names of timezones.

The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems.

1275300271000000 1275303122000000
#335 enhancement dread dread v1.1 closed fixed Post-package-edit redirect to configurable URL

As a

third-party interface to a CKAN instance

I want to

link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return.


  1. The 'return URL' is passed as a parameter to CKAN.
  2. CKAN substitutes the package name into the return URL.


Front-end links to: http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/<NAME> (but with the parameter URL-encoded)

When finished editing and the user commits, CKAN redirects the user to: http://datadotgc.ca/dataset/pollution_data

1275408834000000 1276179605000000
#336 defect dread donovanhide v1.1 closed fixed Resource Search API

As a

CKAN client such as ScraperWiki

I want to

search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL.

Proposed implementation

Add resource search API at:


AND resource added to model API at:


(see ticket:358)

Functional differences from the ScraperWiki suggested patch:

  • URL is not normalised

  • URLs are not grouped
  • All fields of the resource object are returned, not just the URL
  • Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend 'http://ckan.net/package/' to the package ID)

This is to make our API more general, simple and consistent. It means that the ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok?

Example search


{"url": "scraperwiki.com/", "all_fields": 1}

to: /api/2/search/resource

returns JSON:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2

Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done ticket:341 then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe

Search Parameters

Key:  q
Description: Search all resource fields for the value

Key: url / description / format / 
Description: Search particular field for the value

Key: all_fields
Value: 0 or 1 (0 is default)
Description: If 1 (true), the full record of the package resource
(and it's package reference) are returned, rather than just the
PackageResource ID.

May also choose to introduce 'offset' and 'limit' to page through a large number of results.

JSONP achieved through API-wide parameter - see ticket:342

Search is case insensitive.

Original request

Hi, have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of #388.

Could also add a search version. Not sure what the best url would be for that though.

Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere?


1275411765000000 1279373842000000
#340 enhancement dread dread v1.1 closed fixed Web UI theme easier to configure

An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in.


1276009729000000 1328807317000000
#342 enhancement dread dread closed fixed JSONP parameter in API

As a

CKAN client using JQuery

I want to

call the CKAN API and instead of receiving back JSON I get JSONP. i.e. "%s(%s)" % (callback, json_content)

Suggested implementation

All API calls allow the JSONP 'callback' parameter to be specified in the request and this wraps the JSON response. See suggested patch to rest.py by Donovan Hide:



import re import unittest

def test_jsonp_callback():

response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com&callback=jsoncallback') match = re.match('jsoncallback\(.*\);',response) self.assertTrue(match)

response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com') match = re.match('jsoncallback\(.*\);',response) self.assertFalse(match)

I think the point needs to be made that JSONP only works for GET requests and not POST/PUT/DELETE, so there needs to be a check for that in the _finish_ok method.

(thanks to Donovan Hide for test)

1276166426000000 1276278485000000
#345 requirement dread johnbywater closed invalid metastable, stable & ultrastable branches shall be automatically built and tested 1276523083000000 1291831615000000
#346 defect dread johnbywater ckan-v1.3 closed wontfix Revision search API (response data format and documentation issue)

Whilst going through the API docs for the revision search API, it was noticed that the "Gdu" SoS doc doesn't match up. It returns revision IDs (perhaps this is useful to note in the spec?) so the format is probably not 'limitedstring'. Also, they appear to be ordered youngest first, not oldest as stated.

And in the revision model, it refers to 'simplestring' which it doesn't define - I guess the names should be 'limitedstring'?

Could this be checked out?

1276523517000000 1296477510000000
#352 enhancement dread dread closed wontfix Package notification worker - sends XML-RPC

As an

external front-end

I want to

be notified (by XML-RPC) about package creations and updates.


  1. A message queue worker waits for package update notifications
  2. On reception, it constructs XML detailing the changes and PUSHes it to a configured URI.

Rather than turning the package fields into XML fields, the JSON dump of the list of package dictionaries will become a single XML parameter.

Config - in the CKAN config will be:

  • URI to callback to
  • API version to use (version 2 gives packages referred by ID not name)
1276597996000000 1286375870000000
#362 defect dread dread closed invalid Ratings should not be created with a GET


In the Web UI, when you rate a package it simply links to something like:


This creates a GET request.

This is bad because:

  • Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this)
  • There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept? GET> Apache instruction in. But the database may still get written for these ratings.
  • Best practise for web requests is for GET to be a read-only request.
1278925451000000 1311176564000000
#365 enhancement dread dread closed fixed ResourceNotifications

If you change a resource then you not only get a PackageNotification?, but also a ResourceNofication?.

1279037411000000 1279300621000000
#367 enhancement dread dread closed fixed Notfication monitor

Runs on the commandline and prints out notifications.

1279303310000000 1279303693000000
#373 requirement dread johnbywater closed fixed Sort out gov daily script

Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally.

Requested by DGU.

1279885887000000 1286376176000000
#390 task dread johnbywater ckan-v1.2 closed fixed Remove a package for SB 1280418625000000 1282214629000000
#399 task dread johnbywater ckan-v1.2 closed fixed Import BIS data for SD 1281000890000000 1282294177000000
#420 task dread dread ckan-v1.2 closed fixed Release ckan 1.1 1281432518000000 1281522091000000
#421 task dread dread ckan-v1.2 closed fixed Upgrade test-hmg.ckan.net to latest ckan 1.1 1281432541000000 1282909772000000
#423 task dread johnbywater closed fixed Add examples to documentation about uploading data 1281435260000000 1282812768000000
#424 task dread dread ckan-v1.2 closed fixed Tell JF the format spec & examples 1281436534000000 1282919429000000
#426 task dread dread ckan-v1.2 closed fixed Ask list for suggestions improving API docs 1281436574000000 1282925305000000
#432 defect dread dread ckan-v1.2 closed fixed Creating package over REST gives 500 error

This occurs when CKAN is run with mod_wsgi (not under paster).

This is because of a unicode header being creating in rest.py.

1282065101000000 1282552819000000
#438 requirement dread dread ckan-v1.3 closed invalid Data importers use minimal processing

e.g. ONS importer

1282224383000000 1294661079000000
#440 task dread dread closed duplicate Write and pass comprehensive performance tests

Run latest ckan on eu0. Automate some queries and searches. Check load and database connections / processes.

1282226932000000 1294417436000000
#441 requirement dread dread ckan-v1.3 closed duplicate CKAN read-only state

When performing maintenance on CKAN it may be necessary to make CKAN obviously read-only, telling the users and restricting access to 'edit' pages.

Examples of use:

  • Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done.
  • A CKAN is used just for distributing metadata and so is always read-only. Updates may still arrive through direct db manipulation, e.g.:
    • another (but writable) CKAN instance is connected to the same db
    • restoring database dumps from another CKAN db
  • Should a security be breached, all editing could be stopped
1282227314000000 1292586309000000
#442 requirement dread dread ckan-v1.3 closed worksforme Importing data

Review what chunks of code we have and how to move forward with requirements.


  • Bulk parsing is done away from the CKAN server

Possible future requirements:

  • /importer for importing spreadsheets in WUI
  • validation-light - 'scoring' a package

Current importers:


  • pkg_dict --form--> fs & validation
  • fs --PackageSaver?--> package
1282228977000000 1294661028000000
#443 task dread dread ckan-v1.2 closed fixed Get DGU daily dump working (local)
  • Producing daily dump file
  • Publishing at ckan.net/dumps/ or better
1282299046000000 1282735977000000
#451 task dread dread ckan-v1.3 closed invalid Show/Allow only departments associated with user

Use drupal api

1282299641000000 1292587233000000
#454 story dread dread ckan-v1.2 closed fixed Import ONS data via API 1282299851000000 1286376044000000
#457 story dread dread ckan-v1.2 closed fixed Dump database on a daily basis 1282299921000000 1282909051000000
#461 task dread johnbywater ckan-v1.2 closed fixed Change ONS data importing to work via API
  • Move script out to ckandgu repo
  • Change script to convert xml into package dicts
  • Test (against test.ckan.net, hmg.test.ckan.net)
  • Deploy
1282303411000000 1283250478000000
#462 story dread johnbywater ckan-v1.2 closed fixed Visitor downloads a daily database dump 1282305305000000 1285757238000000
#466 requirement dread johnbywater ckan-v1.3 closed wontfix Name of HTTP header for API key shall be configurable 1282308995000000 1294835610000000
#470 task dread johnbywater ckan-v1.3 closed fixed Adjust API key's HTTP header name on DGU "live" and "test" 1282310854000000 1292587187000000
#474 task dread dread closed wontfix Questions related to fi.ckan.net

Several questions to answer (and see file in email). To respond to the Finnish team and copy all Q&A to the ckan-discuss list.

1 ) I have played with two different datasets to get the metadata from those imported to CKAN. The other dataset is in PCAXIS format and other one is in Excel format. Please find the scripts + sample files attached in a zip file that contains instructions too to run the scripts. I looked at import scripts written in object oriented manner and getdata samples in CKAN but it was easier for me too write these from scratch (copied the unicrap function from CKAN samples). If I run the PCAXIS import script I get Unicode decode error (see errors file). When I use the unicrap function then the error does not appear in logs. I assume this is some Python environment setting issue. Of course I would like to run the script without unicrap function to get scandic letter imported correctly to ckan(and displayed in WUI). In Excel case there is no scandic letter issue which is kind of strange but this probably relates how the Excel library handles encoding.

To be honest the whole unicode/encoding/decoding issue has been quite difficult to deal with and debug. I hope I would get proper environment configuration to handle scandic letters.

2 ) It seem in CKAN by default the localisation file contains 462 strings. The transifex link Rufus sent contains 400 strings. Is transifex the one we should use as localisation file ? Where are the rest 62 strings then fetched from or are they needed at all. I have found that as I have been using the .po file coming in CKAN source (which has API translations too) this also causes similar unicode problems when an error in API occurs (the API error messages containing scandic letters provide the similar UnicodeDecoding? errors as mentioned above). So, it seems the unicode issues appear in various cases. Maybe a proper environment settings will be able to prevent also these problems... If transifex is the one to use I can prepare it and maybe then you could upload it to fi.ckan.net as there will be also other people checking the translations. To run the test and see the API translation errors especially in pcaxis case I would recommend to install attached FI translations to your environment and include lang = fi configuration. It looks like the API errors does not come when running default lang = en.

3 ) Is it possible to change the license selection/options available in license field drop-down ? Where are these located and can these be modified locally ?

4) It looks like the values stored in package extra fields are not searchable. Is this intentional ? It looks more or less that all other fields can be searched using free-text search in CKAN.

5) How package dependencies are defined and managed ? For instance we want to import some packages in a manner that the other package is a child package of another. Does API support this and/or is this possible to do in WUI ?

Best Regards, Henri

1282312255000000 1294916760000000
#479 task dread johnbywater ckan-v1.2 closed fixed Suggest DGU presents database dumps through front end 1282324365000000 1288004211000000
#502 requirement dread rgrp ckan-v1.3 closed wontfix Data4nr uploading script shall remove deleted packages

Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed.

1282659304000000 1292586466000000
#503 task dread rgrp closed fixed Provide exemplar package relationships for some packages
  1. Find some relationships.
  2. Add them via API
  3. Notify DGU of completion and point to exemplars
1282659631000000 1314031851000000
#504 requirement dread rgrp ckan-v1.3 closed invalid The DGU fieldset shall conform to v3 of DGU metadata specification

No need for requirement - closing in favour of #505

1282660246000000 1292586686000000
#505 enhancement dread rgrp closed fixed DGU fieldset supports v3 of DGU metadata specification 1282660416000000 1298368280000000
#506 task dread rgrp closed fixed Plan transition to DGU metadata-fields v3 1282660490000000 1287080989000000
#508 task dread dread ckan-v1.2 closed fixed Create branch ultrastable
  • Create a branch named ultrastable that represents latest version of 1.0 as found on dgu
    • Going forward ultrastable will act as stable - 1 (and either ultrastable or stable will be on the rock-solid production systems such as dgu)
  • Do not deprecate the stable or metastable branch for the present
  • Ask for recommendations on mailing list about best practice for branching (e.g. versioned based versus a simple HEAD versus stable) and how this affects standard operations such as applying bugfixes.
    • For useful discussion of how to do backporting and applying bugfix patches see: http://www.python.org/dev/peps/pep-0374/#backport
    • NB: in general do not always have to apply in stable/ultrastable and merge upwards for bugfixes -- can use transplant extension as well.
1282663663000000 1282908795000000
#509 story dread dread closed invalid ONS data imported daily 1282665870000000 1291734435000000
#510 enhancement dread dread closed fixed Setup cron for ons import 1282665939000000 1294138332000000
#536 task dread dread ckan-v1.2 closed fixed Extend base importer merging in resources

Multiple ONS records can go into one package e.g. Stats for Jan, Stats for Feb, Stats for Mar etc.

We already identify an existing package by 'external_id'. Now need to find package by title and an extra field (department in ONS case) and allow it to merge in resource and tags. Alert if rest of meta data is different.

1283173313000000 1286376029000000
#677 task dread dread ckan-v1.3 closed fixed License change script
  • Uses API to change licenses of all packages in a CKAN instance to a particular one.
  • Run it on DGU.
1286384756000000 1292587315000000
#678 task dread dread ckan-v1.2 closed fixed Form for DGU metadata v3
  • Create form
  • Deploy on test
  • Deploy on live
1286385712000000 1286979133000000
#680 task dread dread closed fixed Prepare for DGU tech meeting 1286386074000000 1287141809000000
#681 task dread dread ckan-v1.3 closed fixed Department field bug


1286386685000000 1290507180000000
#682 enhancement dread dread ckan-v1.4-sprint-1 closed fixed Add ckanclient tests to buildbot

Some work done on this to make tests run independently.

1286386769000000 1297358266000000
#683 story dread dread closed fixed metastable, stable & ultrastable smoketests run by buildbot 1286388134000000 1287142289000000
#684 task dread dread closed fixed metastable tested on buildbot 1286388181000000 1286452710000000
#685 task dread dread closed fixed stable tested on buildbot 1286388206000000 1287997008000000
#686 task dread dread closed wontfix ultrastable tested on buildbot 1286388225000000 1287997047000000
#687 task dread dread ckan-v1.2 closed fixed Release ckanclient 0.4

Done. Added release process here: wiki:ReleaseProcess

1286452098000000 1286452650000000
#731 bug dread dread ckan-v1.3 closed worksforme Geo coverage field losses in Form API

Sometimes editing a package via api on dgu results in some countries being lost in geo coverage.

1287738539000000 1288038226000000
#732 task dread dread ckan-v1.3 closed fixed Reproduce and investigate (geo coverage issue) 1287738761000000 1288038335000000
#733 task dread dread ckan-v1.3 closed invalid Fix problem (geo coverage issue) 1287738764000000 1288038341000000
#734 task dread dread ckan-v1.3 closed fixed Investigate department issue (ONS import)

Creation of duplicate packages named with trailing underscore

ww 22/9/10:

Need to ... merge some packages (the "find packages by attributes" is not 100% reliable because of inconsistent data already in there).

The "find by tags" algorithm uses (title, department). Unfortunately some of the data already there has no department. This causes it to make a new package (with trailing underscore).

dread 24/9/10:

I've done a test for the missing department issue and I can't reproduce the problem. This may be due to my general tidy up and getting tests passing using the latest ckanclient. Anyway, can you check over my test to see if this is what you mean?: dgu/ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadMissingDept

ww 20/10/10:

you should uncover it if you run against (a copy of) the dgu database.

1287747273000000 1292587603000000
#735 task dread dread closed fixed Investigate 'deduping' (ONS import)

ww 22/9/10:

(discussion of ticket:509) This leads us to deduping which should be the next order of business oin this thread...

ww 20/10/10:

if a dataset is missing a department and the ons script makes a new version of it with a department we now have two reccords for the same dataset. these should be manually merged - need a merge tool where you can tell it that two datasets are the same.

1287748786000000 1292957248000000
#742 task dread dread ckan-v1.3 closed fixed Release ckan 1.2 to metastable

Need to make sure #700 is fixed.

1288002397000000 1289824163000000
#777 task dread dread ckan-v1.3 closed fixed Put 503-returning middleware into a module

Also configure hmgckan, hmgapi, test-hmg and dev-hmg machines.

1288608324000000 1292586843000000
#798 task dread dread ckan-v1.3 closed fixed cospread importer using API

Move existing getdata/cospread.py into dgu repo and use the importer/loader mechanism.

1289475362000000 1294916973000000
#805 task dread dread ckan-v1.4-sprint-2 closed fixed Migration scripts on buildbot 1289816755000000 1298379084000000
#823 defect dread dread closed fixed Etags cache doesn't update when only non-core package fields changed.
  1. Create a package.
  2. Edit package and add a resource.
  3. View package in browser - no sign of the resource - ERROR.
  4. View package in API - resource is there.

Reason: hash doesn't change because package.revision only takes account of core fields.

1290505078000000 1290506116000000
#824 enhancement dread dread closed fixed 'Getdata' moved out of CKAN

Data importers should use API instead of object model directly:

  • saves load on server
  • much better error checking and authentication
  • eating our own dogfood (tests the API well)
1290595507000000 1290595559000000
#825 enhancement dread dread closed fixed Document fabric tool 1290609160000000 1290624449000000
#840 enhancement dread dread ckan-v1.4-sprint-6 closed fixed On/off switch for ETags cache
  • Use config variable to switch ETags caching on or off. Consider joining in with 'cache_enabled'.
  • Default setting for (all) caching should be off.
  • Needs documenting in configuration.rst
1291308727000000 1302694133000000
#858 defect dread dread closed fixed Tests for diff

Test coverage for ckan.lib.diff is 24%. Need to improve

1291724171000000 1291734297000000
#867 enhancement dread dread ckan-v1.3-sprint-1 closed fixed ckanclient raises exceptions

To be more pythonic, raise exceptions when ckanclient gets status which isn't 200.

1291915162000000 1299866685000000
#873 defect dread dread closed fixed ONS loader assigns packages to same package even when dept name added manually

e.g. national_child_measurement_programme had no department but "Information Centre for Health and Social Care" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_.

1292437253000000 1292939274000000
#897 defect dread dread closed duplicate "New package" highlighted in Navigator when searching

On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted.

1294648815000000 1294914333000000
#919 enhancement dread dread ckan-v1.4-sprint-6 closed fixed Package preview contains API & datapkg instructions
  1. Edit a package
  2. Hit 'preview'.

The preview contains the section "CKAN API / datapkg" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the "CKAN API / datapkg" section too?

1295346730000000 1303202627000000
#931 enhancement dread dread ckan-v1.4-sprint-1 closed fixed Search results generator hides paging functionality

ckanclient's search results list only packages up to the 'limit'. It would be good to return a generator instead of a list. When the limit is reached on the generator then another 'page' is loaded automatically.

1296147360000000 1298379187000000
#932 enhancement dread dread ckan-v1.3 closed fixed Move to SQLAlchemy-Migrate 0.6

Basic steps from kindly:

  • changing the setup.py dependencies for sqlalchemy-migrate and sqlalchemy to their latest versions.
  • running the migrate script on the repository (or changing the repository scripts manually).
  • checking in both these changes at once.
1296156660000000 1297076710000000
#950 enhancement dread dread ckan-v1.3 closed fixed Move importer controller to ckanext-importer

ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN.

In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext.

1296666038000000 1297079743000000
#955 defect dread dread ckan-v1.3 closed fixed Tidy up tests
  • Removal of init_db from every setup etc.
  • Get them working
1296819300000000 1297342534000000
#982 enhancement dread rgrp ckan-v1.4-sprint-2 closed fixed Remove various pip-requirements versions

We can just use the branch name to pull down specific versions of pip requirements so we don't need all the different versions in head.

Remove them and update any client systems.

Cost: 20m

1297516060000000 1298887980000000
#983 defect dread dread ckan-v1.3 closed fixed 'db upgrade' creates system priviledges

(should leave priviledges alone)

1297518265000000 1297773407000000
#1002 enhancement dread rgrp ckan-sprint-2011-12-05 closed fixed Remove changeset model code

This code is unused and tests are disabled i belive. I suggest stripping this out (can always be re-added in an extension later). If we remove we should probably delete the relevant tables (and so a migration is needed).

1298489773000000 1322515696000000
#1005 defect dread dread closed duplicate Editing in ckan missing publishers

Package edit form at hmg.ckan.net has no publishers any more.

1298561348000000 1300100085000000
#1018 task dread dread ckan-v1.4-sprint-3 closed fixed Remove gov form

(It has been moved into ckanext-dgu)

1299072516000000 1299073340000000
#1025 enhancement dread dread ckan-v1.4-sprint-3 closed fixed Default authz can be set in config

Currently the default authz for a package is hard-coded to:

 <PackageRole user="visitor" role="editor" context="Package">,
 <PackageRole user="logged_in" role="editor" context="Package">,

This should be configurable in the config, so that you can have a more locked down instance etc.

1299596110000000 1299751045000000
#1026 enhancement dread dread ckan-v1.4-sprint-3 closed fixed cli for creating users

It's handy to be able to create users using the cli (e.g. dgu migration)

1299604652000000 1299605128000000
#1028 defect dread dread ckan-v1.4-sprint-3 closed fixed Open redirect in locale setting 1299759883000000 1299760360000000
#1033 defect dread dread closed fixed Register user with blank password causes 500

Go to http://ckan.net/user/register and fill in all the fields apart from password. On submit you get 500 error.

1299796274000000 1308310446000000
#1038 enhancement dread dread ckan-v1.4 closed fixed Authz tool - operate on all packages at once

Add 'package:all' to authz tool to allow mass changes of authz.

1300212788000000 1300212841000000
#1039 enhancement dread dread ckan-v1.4 closed fixed Default user roles read from config

(instead of being hard coded)

1300212822000000 1300212856000000
#1042 task dread dread closed fixed 'Ckanext' split-up

It's not good to have ckanext doing lots of different things with different dependencies. Split it off into:

  • ckanext-importlib


And then deprecate the ckanext repo itself.

1300293907000000 1300969865000000
#1045 defect dread dread ckan-v1.4-sprint-4 closed fixed Group identified by ID in API

returns group IDs but I can only reach a group by name:


when I also want to get a group by ID:

1300360642000000 1300793261000000
#1048 enhancement dread dread ckan-v1.4-sprint-4 closed fixed Complete making groups versioned
  • Deleting a group changes state to 'deleted' rather than purging it
  • Adding authz tests for deleted groups
1300387655000000 1300702752000000
#1052 defect dread dread ckan-v1.4-sprint-4 closed fixed Authz holes

No authz on:

  • Group creation/edit/listing
  • Package relationship create/edit/delete
1300709144000000 1300895410000000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Note: See TracReports for help on using and creating reports.