{22} Trac tickets (2647 matches)

Results (801 - 900 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#802 story johnbywater johnbywater closed duplicate Save last harvested time on source 1289484226000000 1294233294000000
#803 story thejimmyg johnbywater closed invalid Auto-generate next migration version unit and test stub files

As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually.

1289486678000000 1314031451000000
#804 task johnbywater johnbywater ckan-v1.3 closed fixed Rework analysis for publisher/provider in UKLP

We need an incremental plan that connects with current state of DGU and reflects what is actually required by UKLP.

1289816054000000 1294233156000000
#805 task dread dread ckan-v1.4-sprint-2 closed fixed Migration scripts on buildbot 1289816755000000 1298379084000000
#806 enhancement rgrp rgrp ckan-v1.3 closed fixed Metadata created and last modified timestamps for packages

Packages should provide create and last modified attributes which are included in package dict exposed via api.

These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc)

1289854894000000 1324034356000000
#807 enhancement rgrp ckan-v1.3 closed wontfix Handle robot traffic better

We get a lot of robot traffic. Notice that robots are visiting package edit pages. Suggest:

  • Add rel=nofollow to various links
  • Throttle some robots (not all seem to respect robots.txt)
1289899826000000 1297075372000000
#808 enhancement pudo rgrp ckan-v1.4-sprint-2 closed fixed Allow i18n language choice on site (and possibly set automatically)

Allow choice of language on ckan instance.

  • Configuration variable listing language choices (could try doing this automatically but recommend against this)
  • Switcher option in UI (perhaps on user profile page)
  • Default it automatically based on browser settings (?)

Think this is high value (and cost is low)

Cost: low (4h)

1289944394000000 1297783658000000
#809 requirement pudo ckan-v1.3 closed wontfix Link to the Wiki pages in package form

This would allow for a community-driven definition of field values, conventions, etc.

Plus: create wiki.ckan.net

1289992231000000 1297075561000000
#810 enhancement pudo pudo ckan-v1.5 closed wontfix Move "add packages" field up in group form

Subticket of forms super ticket #961

1289992287000000 1310128477000000
#811 defect cygri ckan-backlog new Extra field editing form layout breaks when there are long field names

The layout of the editing section for extra fields breaks when a field name is slightly too long. Field names jump over to the right. See http://ckan.net/package/edit/dbpedia for examples.

1289994812000000 1323170289000000
#812 defect cygri ckan-backlog new Package edit form only allows three extra fields


The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview).


A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

1289995010000000 1311176917000000
#813 requirement cygri ckan-v1.3 closed fixed Add link to “Register new package” to the homepage

A link to the “register new package” page would be handy somewhere on the front page. As a frequent editor, I often go to http://ckan.net in order to create a new package. I don't want to hunt around for the link to the “new package” form.

The link could go into the bar next to “About / Statistics”. Or into the “More information” sidebar. Neither of these truly make sense, but it's a frequently accessed link and that's reason enough to put it somewhere onto the frontpage.

1289995253000000 1294411047000000
#814 requirement cygri ckan-v1.3 closed fixed Have an “About CKAN” link on every page

A link “About CKAN” pointing to http://ckan.net/about should be visible on every page. This is important because people often land on subpages and may have trouble figuring out what CKAN is.

I would put this link into the main navigation bar, on the very right next to “Revision History”. Then I would also remove the “Home” item from the main navigation bar because it is redundant. The CKAN logo is already a link to the homepage. The only other subpage in the “Home” section is “Statistics” and that's already linked from the homepage sidebar.

But anywhere else would be fine as well.

1289995821000000 1294411109000000
#815 requirement cygri ckan-v1.3 closed duplicate Autocomplete for the search field

The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page.

1289995951000000 1294410951000000
#816 enhancement johnglover cygri ckan-sprint-2011-10-28 closed fixed Autocomplete for the resource format field

The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource.

This would help improve data quality and consistency.

1289996574000000 1319812324000000
#817 requirement cygri ckan-v1.3 closed worksforme Proposed redesign of Resources table on package pages

There are some problems with the Resources table on package pages:

  1. The column labelled “URL” doesn't contain URLs, but just the word “Download”.
  2. The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links).

Attached is a screenshot for a proposed redesign. The changes are:

  1. Move the Format column to the right of the Description column (description is more important)
  2. Make the description a clickable link
  3. Drop the “URL/Download” column because it's now redundant
  4. Rename “Description” to “Resource” (not that important)
1289997384000000 1297073724000000
#818 requirement cygri ckan-backlog new Rethinking the author and maintainer fields

The semantics of the Author and Maintainer fields are really unclear at the moment. This leads to very inconsistent usage. Also, perhaps Name and Email are not the only fields that are needed for a contact.

Here is a table that shows the current usage of these fields in CKAN: http://richard.cyganiak.de/2010/ckan/ckan-ppl.html

We note several problems:

  • Author and Maintainer are often the same
  • Author and Maintainer are often used interchangeably
  • People really want to specify URLs for the contacts and stick them into random places because there is no field for it
  • Multiple comma-separated names in a single field

I'm not sure what to do about this, but a redesign is necessary in my opinion.

Some ideas:

  • Remove the maintainer field?
  • Make really clear that Author doesn't refer to the metadata on CKAN, but to the original data
  • Add an “author URL” field?
1290003524000000 1339774621000000
#819 defect cygri ckan-backlog closed fixed Tag autocompletion widget broken

The widget for tag autocompletion is broken in various ways.

For example, if I edit a package that is tagged "music", and just tab through the form fields to get to the Author field, then the widget changes to "industrial-music" as I tab through it.

Or if the tag is "foo bar" and I hit alt-right to jump to the end of the text field (with the intent of adding a third tag), then the contents change to "foo barbecue".

1290004121000000 1323171435000000
#820 defect dread closed fixed Exception browsing /user/apikey when not logged in

Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in.

WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'apikey' 						

URL: http://test-hmg.ckan.net/user/apikey
1290006079000000 1311325226000000
#821 defect dread ckan-v1.4-sprint-2 closed worksforme Some resources have disappeared in CKAN

Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end.

Here are the details:

  • in CKAN's package view (in web interface, API and dump) no resources display
  • but they WERE created in CKAN, as shown by the revision diffs.
  • they are in the Drupal front-end

So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal.

Packages affected: anti-social-behaviour-orders-1999-2007 asylum-applications-jan-mar-2009 control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december coroners-statistics-england-and-wales courts-statistics-user-survey-england-and-wales court-statistics-company-insolvency-and-bankruptcy-england-and-wales court-statistics-england-and-wales court-statistics-mortages-and-landlord-possession-england-and-wales crime-in-england-and-wales crime-statistics-local-reoffending-england-and-wales crime-statistics-prison-and-probation-england-and-wales crime-statistics-reoffending-of-adults-england-and-wales crime-statistics-reoffending-of-juvenilles-england-and-wales data_gov_uk-datasets digest-uk-energy-statistics-2008 directgov-central-hottest-pages-monthly directgov-central-internal-search-terms-monthly directgov-section-visits-monthly electricity-consumption-2007 electricity-gas-consumption-2007 energy-consumption-uk-2008 final-energy-consumption-2007 foi-statistics-uk-central-government fuel-poverty-statistics-2007 gas-consumption-2007 gb-reported-bicycling-accidents gb-road-traffic-counts gb-traffic-matrix greenhouse-gas-emissions-2008 high-level-indicators-energy-use-2006 judicial-and-court-statistics-england-and-wales laboratory-tests-and-prices local-authority-carbon-dioxide-emissions-2007 magistrates-courts-statistics-survey-england-and-wales monthly-energy-prices monthly-energy-trends ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig ni_013_migrants_english_language_skills_and_knowledge ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so ni_026_specialist_support_to_victims_of_a_serious_sexual_offence ni_029_gun_crime_rate ni_031_re-offending_rate_of_registered_sex_offenders ni_032_repeat_incidents_of_domestic_violence ni_034_domestic_violence_-_murder ni_036_protection_against_terrorist_attack ni_038_drug_related_class_a_offending_rate ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an ni_109_delivery_of_sure_start_childrens_centres ni_126_early_access_for_women_to_maternity_services ni_127_self_reported_experience_of_social_care_users ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law ni_185_co2_reduction_from_local_authority_operations ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an other-fuels-consumption-2006 police-use-firearms-england-wales-2007-2008 prison-end-of-custody-licence-releases-and-recalls-england-and-wales prison-population-england-and-wales probation-offender-management-caseload-statistics-england-and-wales probation-statistics-quarterly-brief-england-and-wales quality-indicators-energy-data-2007 quarterly-energy-prices quarterly-energy-trends road-transport-energy-consumption-2007 sentencing-statistics-england-and-wales statistics-terrorism-arrests-outcomes-2001-2008 ukba-control-of-immigration-statistics-2008 ukba-control-of-immigration-statistics-2008-supplementary-tables uk-energy-in-brief-2008 uk-energy-sector-indicators-background-2008 uk-energy-sector-indicators-key-supporting-2008 uk-exportcontrollists uk-exportcontrol-sanctions uk-export-control-statistics uk-glossary-exportcontrol uk-ipo-offences weekly-fuel-prices

1290015776000000 1298486642000000
#822 enhancement Javier de la Cueva <[email protected]…> closed fixed Docs: Configuration as site-available instead of site-enabled for apache

The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net

1290107103000000 1290506354000000
#823 defect dread dread closed fixed Etags cache doesn't update when only non-core package fields changed.
  1. Create a package.
  2. Edit package and add a resource.
  3. View package in browser - no sign of the resource - ERROR.
  4. View package in API - resource is there.

Reason: hash doesn't change because package.revision only takes account of core fields.

1290505078000000 1290506116000000
#824 enhancement dread dread closed fixed 'Getdata' moved out of CKAN

Data importers should use API instead of object model directly:

  • saves load on server
  • much better error checking and authentication
  • eating our own dogfood (tests the API well)
1290595507000000 1290595559000000
#825 enhancement dread dread closed fixed Document fabric tool 1290609160000000 1290624449000000
#826 enhancement kindly dread ckan-v1.4-sprint-1 closed fixed Resource 'extra' fields

There are a number of extra properties we want to store in a resource:

i.e. this is a much needed enhancement

Need to be able to read/write these values in web interface and API.

(At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.)


I suggest these are arbitrary key/value pairs, just like package extras, reusing PackageExtra? code as much as possible. (Other much more radical alternative is to move model to RDF.)

1290611905000000 1306766057000000
#827 defect rgrp dread closed fixed metadata_created_and_modified error

Intermittent failure on ckan default branch. Looks like brittle test.

FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified
Traceback (most recent call last):
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py", line 250, in test_02_metadata_created_and_modified
    assert out == exp, (out, exp)
AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938))


1290678624000000 1290766239000000
#828 enhancement rgrp rgrp ckan-v1.3 closed fixed Update cache support in home controller to use cache config in ini file

Currently cache usage in home controller is inconsistent with elsewhere and, in particular, does not respect cache_disabled and is not configurable using path notation (e.g. ckan.controllers.home. ...).

This is especially frustrating in a development environment where is currently running into fact front page is being cached.

Fix this.

1290693154000000 1290696586000000
#829 defect rgrp dread ckan-v1.4-sprint-6 closed wontfix Admin CRUD broken

Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception.

1290695043000000 1303838115000000
#830 enhancement rgrp closed fixed Themes for CKAN

Already have the basic capacity using *_extras paths for templates and public directory (see http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: http://packages.python.org/Flask-Themes/

1290760612000000 1311180263000000
#831 task rgrp rgrp ckan-v1.3 closed fixed Migrate ckan trac to ckan.org

Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc.

1290760820000000 1290760864000000
#832 enhancement rgrp rgrp ckan-v1.3 closed fixed Convert stats to a plugin

Convert stats section to a ckan plugin.

Stats need not be in core and making it a plugin would make it easier to extend and expand.

1290765915000000 1296334980000000
#833 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-6 closed fixed [super] Administrative dashboard extension

Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:

  • Purge revisions (or sets of revisions) and purge objects #1076
  • Set roles for users #1075
  • Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
  • Overview of activity
    • WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.




Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of 
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system
1290765921000000 1303236364000000
#834 task Alexander ckan-v1.3 closed worksforme Searching in CKAN


I've installed stable CKAN v1.1 from PyPI.

I can't find any docs about using CKAN API in order to query packages.

Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible?

Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&open_only=1&downloadable_only=1 and Ckanclient:

result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1})

As result nothing found.


1290769564000000 1291633657000000
#835 enhancement pudo pudo ckan-v1.3 closed fixed Authorization CLI and Documentation

As a sysadmin I want to take CKAN out of wiki-mode into normal mode. In normal mode visitor and logged in users cannot create or add packages.

Suggest put this as an internal command in authz system and add to cli as paster command.

1291029108000000 1291644820000000
#836 enhancement rgrp ckan-v1.3-sprint-1 closed fixed Use site_url config option in templates

A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html

Instead we should use the site_url config option where necessary.

Cost: 1h

1291132434000000 1302882808000000
#837 enhancement rgrp ckan-backlog new CKAN integration with freebase gridworks / google refine

Thread: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

Scenario 1

  1. User installs Refine and CKAN extension for refine
  2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
  3. They edit the dataset on Refine
  4. On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)

NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required)

Scenario 2

  1. User visits a package on CKAN.net (or another CKAN instance)
  2. There is a button on the page "View and edit this dataset in Google Refine"
  3. Click button -- ask them if they have Google refine installed
    • Yes: instructions for loading dataset into refine
    • No: load dataset in hosted version of google refine (we could run this)
  4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset.
1291140609000000 1339774605000000
#838 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed UI improvements December 2010

This is a meta ticket for a bunch of UI changes:

  • #839 Package page redesign
  • #846 Merge css files
  • #850 Show welcome message
  • #863 Prompt users to enter missing info
  • #864 Search results changes
  • #866 Add download formats to search results
1291296811000000 1295259773000000
#839 enhancement memespring anonymous ckan-v1.3-sprint-1 closed fixed Package page redesign

as per http://ckan.org/wiki/UIRedesignPackage

1291299579000000 1292239918000000
#840 enhancement dread dread ckan-v1.4-sprint-6 closed fixed On/off switch for ETags cache
  • Use config variable to switch ETags caching on or off. Consider joining in with 'cache_enabled'.
  • Default setting for (all) caching should be off.
  • Needs documenting in configuration.rst
1291308727000000 1302694133000000
#841 enhancement kindly dread ckan-v1.4-sprint-4 closed duplicate Caching docs (as a whole)

Documentation article on caching / improving performance. (To complement configuration docs.)

  • Different sorts of cache - beaker style, etags, package_dict in search results(?)
  • How each one affects performance
  • How to turn them on/off and configure them
  • Is it possible to bypass each of them in the browser or with wget/curl?
1291308879000000 1300364333000000
#842 enhancement johnglover rgrp ckan-v1.4-sprint-7 closed fixed Todo list CKAN extension

An extension that provides a todo list feature on CKAN so that people can register and find things to do.

Extension name: ckanext-todo

User Story

Package page

As a user I come to a package:

  • Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
  • At the bottom is a section of the package display titled "ToDo?" where I see a list of all toDos for the package most recent at the top
    • If I am logged in
      • See a form for "Add to do" at the top of the todo section and can add one straight away
      • I see a "now resolved" button next to each which goes green when you hover.

When clicked the todo fades away.

  • Not logged in: I see a button that says "login to add todo"

Todo list page

When a user comes to todo overview at /todo

At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category.

Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo.


The Todo form

  • One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed)
  • Add a description
  • Submit, the todo gets added via AJAX to the list at the top as the most recent todo


todo table

  id (autoincrement integer)
  todo_category_id (required)
  description (required)
  resolved=null (unresolved) or a datetime (datetime of resolution)

todo_category table

Prepopulate with: broken-resource-link, no-author, bad-format, add-description

API at /api/2/todo

  • GET / POST / PUT ...


  • support limit (?)

/api/2/todo/category -> return list of todo categories

  • No GET / PUT / POST (these are auto-created by creation of todo)

Optional Extras (Will not be done atm)

  • Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ...
1291467708000000 1305646487000000
#843 enhancement wwaites closed fixed user list ui for group curation
  • perhaps we should show the openid as well to distinguish between users with the same name.
  • perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name.
  • also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it.
1291484245000000 1319721601000000
#844 defect wwaites wwaites closed fixed bad mail config ckan.net/ckan.org

ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example:

B30E6D9F       2873 Sat Dec  4 14:05:40  MAILER-DAEMON
(delivery temporarily suspended: connect to ckan.net[]:25: Operation timed out)
                                         [email protected]

both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null

1291556753000000 1296340486000000
#845 enhancement dread ckan-v1.3-sprint-1 closed fixed Required fields

Allow a form to specify fields that are required. Currently name and resource url are required.

  • Show required fields with asterisks by the input box.
  • Resource fields to be included in this.
1291634995000000 1291723492000000
#846 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed merge style.css and ckan.css

no need for separate files

1291635478000000 1291719074000000
#847 enhancement pudo rgrp ckan-v1.3-sprint-2 closed fixed CKAN search using SOLR backend

This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010).

3 key aspects of this:

  • Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN)
    • Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable
    • Asynchroously: via worker with queue
      • #324 Search indexing using notifications
      • #669: Refactor Solr (indexer) to become a generic worker
      • #874: Extract Solr search backend into an extension
  • In core support using SOLR backend as well as postgres for queries
    • #317: Make search pluggable (?)
  • Integrate into WUI (if changes needed)
    • #672 - Facets in CKAN search results


  • Solr index testing tool #431
1291639273000000 1295259902000000
#848 enhancement pudo ckan-v1.3-sprint-1 closed fixed Ability to hide extras fields on package read

for helsinki regional infoshare.

1291714316000000 1291751275000000
#849 defect pudo closed duplicate Import extra CSS from main template

The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import.

1291714923000000 1291715179000000
#850 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed Show welcome message on first visit to package page

Control using cookies.

1291721923000000 1291721964000000
#851 defect wwaites wwaites ckan-v1.3-sprint-1 closed fixed Link Checking

revisit link checker from http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py

revisit ollyc's parallel work

look at how this gets used in practice (maybe deprecate in favour of curate tool (suitably wrapped to hide details from the user)

1291722860000000 1293025112000000
#852 enhancement rgrp wwaites ckan-v1.5 closed fixed [super] Dataset upload and archiving

3 (4) use cases: (With and without access to auth keys.)

  1. upload form on ckan (by default local system has auth key for storage)
  2. datapkg upload (client) (by default without key) (plumbing done)
  3. background job archiving downloads (by queue client)

Most important are:

  1. Upload form on ckan.net (with access to keys)
  2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net


Required infrastructure

  • #826 - Resource 'extra' fields
  • #358 - Resources in REST API
  • #235 - Resource format normalization and detection

User upload via Web UI

  • #877 - File upload in WUI
  • #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

  • #853 - plumbing for credential-less rest upload
  • #879 - Storage auth API

Also related:

  • #405 - Retrieval options for package resources

More Info

1291722993000000 1315821628000000
#853 enhancement wwaites wwaites ckan-v1.3-sprint-1 closed fixed Client upload to storage without having primary storage keys

Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys.

This would lead to an extension to OFS.

This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN).

1291723063000000 1294594581000000
#854 defect johnlawrenceaspden dread ckan-v1.4-sprint-6 closed fixed Tests for authorization_group

The coverage tool (run by buildbot in the ckan build) reports that only 24% of lines of ckan.controllers.authorization_group are run in tests and 38% of ckan.forms.authorization_group. This suggests a need for more tests.

1291723815000000 1304351843000000
#855 defect pudo dread closed invalid Authenticator tests

The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests.

1291723967000000 1311176988000000
#856 defect wwaites dread closed invalid Tests for caching

Test coverage for ckan.lib.cache is only 40%. Can that be usefully improved?

1291724064000000 1311177085000000
#857 defect dread closed invalid Tests for CLI

Test coverage for ckan.lib.cli is 34%. Need to improve that.

1291724131000000 1311182581000000
#858 defect dread dread closed fixed Tests for diff

Test coverage for ckan.lib.diff is 24%. Need to improve

1291724171000000 1291734297000000
#859 defect pudo dread closed invalid Tests for User model

Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement?

1291724348000000 1311177461000000
#860 defect wwaites wwaites ckan-v1.3 closed fixed full text indexing for semantic.ckan.net


1291726029000000 1291726067000000
#861 defect wwaites wwaites closed invalid unicode errors creating rdf output


1291726125000000 1311168845000000
#862 defect wwaites wwaites ckan-v1.3 closed fixed deploy rdf on catalogue.data.gov.uk

handle various dgu extras, configure apache for autonegotiation, etc.

1291726237000000 1291726261000000
#863 enhancement memespring memespring closed wontfix prompt users for missing package information

e.g. license, owner etc

1291729787000000 1338206455000000
#864 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed Search results UI changes

as per http://ckan.org/wiki/UIRedesignSearch

1291736441000000 1291741028000000
#865 defect pudo pudo ckan-v1.3-sprint-1 closed fixed Support external strings in JS
  • Provide and include a "language.js" file.
1291801938000000 1295259773000000
#866 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed Include download formats in search results

as above

1291812213000000 1292240191000000
#867 enhancement dread dread ckan-v1.3-sprint-1 closed fixed ckanclient raises exceptions

To be more pythonic, raise exceptions when ckanclient gets status which isn't 200.

1291915162000000 1299866685000000
#868 enhancement sebbacon thejimmyg closed fixed Test improvements

The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach.

We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!)

Some suggestions for achieving this include:

  • Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database
  • Not setting up and tearing down the database so frequently.
1291987811000000 1294753596000000
#869 enhancement rgrp rgrp ckan-v1.3-sprint-1 closed fixed Add plugin interface to allow editing and extending of config by extensions

Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)

1291989395000000 1292235262000000
#870 defect memespring ckan-v1.3-sprint-1 closed fixed Menu items not selecting correctly

Since the main menu items have changed (part of the UI redesign) the selected items are not functioning correctly.

Specifically if Home / About or Add Package / Search are selected both menu items are highlighted.

This is because the code assumes only one top-level menu item per controller.

1292243583000000 1294914243000000
#871 defect nils.toedtmann closed invalid Check whether localhost-only exim installtions need upgrading too

The infamous exim bug only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers.

We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not.

BTW did we already run a rootkit checker like Rootkit hunter on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark? has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could.

1292264117000000 1296340558000000
#872 enhancement johnbywater dread closed fixed ONS loader assigns packages to same package even when dept name changes

Solutions: ResourceSeriesLoader? allows synonyms for values.

1292424112000000 1292957110000000
#873 defect dread dread closed fixed ONS loader assigns packages to same package even when dept name added manually

e.g. national_child_measurement_programme had no department but "Information Centre for Health and Social Care" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_.

1292437253000000 1292939274000000
#874 enhancement pudo ckan-v1.3-sprint-2 closed fixed Extract solr search backend into an extension

Move the solr SearchBackend? to a seperate code base, remove any mentions of the CKAN model.

1292844422000000 1293057151000000
#875 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Search backend supports solr interface and query API mimics solr

Consolidate search API interface (and backend) on solr (solrpy) type interface.

  • Support for standard query structure
  • Support for facet options

Do not need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

  1. Disable (happens automatically)
  2. Implement - suggest using group by etc


  • Front-page tag cloud: change this to use facets
    • Accept this means that if facets not functional in backend we have no tag cloud
1292844957000000 1297085261000000
#876 enhancement sebbacon rgrp ckan-v1.3-sprint-2 closed fixed Support sqlite as a database backend for CKAN

Among other things this will allow the tests to run much quicker.

1292858711000000 1294753889000000
#877 enhancement rgrp rgrp ckan-v1.4-sprint-2 closed fixed File upload in WUI (extension)

Support uploading files to a storage system from WUI with a simple upload form.


  • Implement as an Extension
  • Locate at /upload
  • Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
  • Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
  • This wants to be tied in to general package workflow. See parent ticket:852.
1292868146000000 1298624165000000
#878 enhancement rgrp rgrp ckan-sprint-2011-09-12 closed fixed Integrate file upload with workflow around package resources

Create/edit a package -> add a resource -> Oh, i haven't uploaded yet, 'upload here' -> /upload -> standard upload form

Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata?

If yes suggests (need to ticket):

  • 'ResourceExtras?' for extra metadata on resources
  • Makes 'Resource' into first class object (and make available in WUI)?


  • #877 (Upload in WUI) - should be part of that extension
1292868269000000 1315820838000000
#879 enhancement wwaites rgrp ckan-v1.3 closed fixed Storage auth API

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

1292868509000000 1316030482000000
#880 defect dread ckan-v1.3-sprint-2 closed fixed ONS loader tripping up on deleted packages

Deleted packages with the same title and extra fields as an active package trip up the loader. It finds them in the search, but due to ckanclient not using the apikey when you 'get' the package to examine it further, it causes an exception.

1292939379000000 1292945137000000
#881 defect thejimmyg fccoelho closed invalid http authentication requirement when installing

I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code.

1293021889000000 1296335072000000
#882 defect pudo pudo closed wontfix Update i18n user docs and backport genshi i18n:domain fixes from WDMMG 1293192044000000 1340626094000000
#883 enhancement wwaites wwaites ckan-v1.4 closed fixed uklii harvesting refactor

master ticket

  • #884 implement harvest job delete command for an easier life
  • #885 move csw code into dedicated ckanext plugin
  • #886 move forms api into dgu extension module
  • #887 move harvesting code to use generic harvesting module

1293277713000000 1300196622000000
#884 enhancement thejimmyg wwaites ckan-v1.3 closed fixed harvest job delete operation

implement paster subcommand to delete harvest jobs

1293277800000000 1296592140000000
#885 defect wwaites wwaites ckan-v1.3 closed fixed reimplement csw module to use owslib 1293278099000000 1294253508000000
#886 defect wwaites wwaites ckan-v1.3 closed duplicate move uklii forms api to dgu module

doesn't belong in ckan core

1293278145000000 1294916538000000
#887 enhancement wwaites wwaites ckan-v1.4 closed fixed move harvesting to generic module

in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat

1293278215000000 1300196600000000
#888 enhancement johnglover rgrp ckan-sprint-2011-10-28 closed fixed Improvements to the dataproxy and the data API

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

  • Testing for dataproxy
    • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
  • Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
  • Google docs style row/column selections
  • Use the swiss library - https://bitbucket.org/okfn/swiss
    • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
  • Handle redirects for content-length?
  • Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

  • Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}
1293649783000000 1311773103000000
#889 enhancement rgrp rgrp ckan-v1.3-sprint-2 closed fixed Support extra footer material in config option (e.g. for google analytics)

Add a config option 'template_footer_end' which is inserted in layout_template just before </body>.

This allows sysadmins to add extra items, especially, scripts directly into site without having to do any theming and is especially useful for things like google analytics.

Aside: going forward may want to turn this into a extension.

Cost: 1h

1293713189000000 1293715109000000
#890 enhancement kindly pudo ckan-v1.6 closed invalid Introduce timed actions into ckanext-queue

The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows:

  • routing_key: Package
  • operation: daily
  • payload: pkg.as_dict()

Where operation is one of daily, weekly, monthly or any other interval.

1294050769000000 1318599247000000
#891 task johnglover pudo ckan-sprint-2011-11-07 closed fixed Resource download worker daemon

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository.


  • Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
  • Should we have clever ways to dump APIs? ANS: no.
  • Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
  • Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
    • Complete support for ETags
    • Expires, Max-Age etc.
  • Check


  • Download files via HTTP, HTTPS (will not do FTP)


  1. [Archiver.Update checks queue (automated as part of celery)]
  2. Open url and get any info from resource on cache / content-length etc
    1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
    2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get content-length and content-length unchanged GOTO step 4
  3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
    • Add cache url to resource and updated date
    • Add other relevant info to resource such as md5, content-type etc
  4. Update task_status

Optional functionality

  • If result object is HTML, search for references to "proper data" (CSV download pages etc.)
  • Download from POST forms (accepting licenses or weird proprietary systems)
  • Support running on Google Apps Engine to save traffic costs.

Existing work

1294052979000000 1320149841000000
#892 enhancement johnglover pudo ckan-sprint-2012-01-09 closed fixed Make stored data available in WUI - 0.5d

Once we have storage, make the data available in the following ways:

  • Now have a cached_url field can show in the frontend ...
  • Add a [<a href="${cached_url}">cached</a>] link to right of real url on resource listing on dataset page.
  • On resource page: will not add it yet.
    • At the moment no clear place to pu this given nice big download button (could put in list of items on left but that does not seem right and note that it will turn up in big list of info at bottom)
  • Add test (?)
  • Deploy
1294053293000000 1324402480000000
#893 defect pudo ckan-v1.4 closed wontfix ExtrasField should not overwrite more specific extras

At the moment, ExtrasField? cannot be used in conjunction with more specific extra fields, such as TextExtraField? or SuggestTextExtraField?.

1294135783000000 1298293527000000
#894 defect wwaites wwaites ckan-v1.4 closed invalid harvest from http://locationmetadataeditor.data.gov.uk/geonetwork/srv/csw 1294253876000000 1300196388000000
#895 defect memespring ckan-backlog new Add version number (or simular) to css/js includes query string

Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g.


1294343382000000 1339774593000000
#896 story thejimmyg closed duplicate Distributed Data and Syncing Between CKAN Instances

Based on the ideas in:

Merging the tickets #296, #297, #298, #299 into this one.

Related ideas include:

  • Using dcat to exchange data
  • Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance

This should be considered a wishlist item for the timebeing.

1294406999000000 1300217375000000
#897 defect dread dread closed duplicate "New package" highlighted in Navigator when searching

On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted.

1294648815000000 1294914333000000
#898 defect rgrp dread closed fixed Changes stored indefinitely

Every change to every object is being stored in memory, which could add up to quite a lot of memory.

This fixes it by making sure the objects are in a weakref. https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f

1294659490000000 1294662408000000
#899 enhancement rgrp dread closed fixed Optimise check_real_change

There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e

From David Raznick

1294659583000000 1294916615000000
#900 defect dread closed fixed Intermittent error with new revision not having id

There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision. We add the uuid earlier to fix this. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e.

From David Raznick

1294659696000000 1294916587000000
#901 enhancement rgrp dread closed fixed vdm support for sqla 0.6 and sqlite 1294660144000000 1294662466000000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Note: See TracReports for help on using and creating reports.