As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually.

We get a lot of robot traffic. Notice that robots are visiting package edit pages. Suggest:

• Add rel=nofollow to various links
• Throttle some robots (not all seem to respect robots.txt)

This would allow for a community-driven definition of field values, conventions, etc.

Plus: create wiki.ckan.net

The layout of the editing section for extra fields breaks when a field name is slightly too long. Field names jump over to the right. See ​http://ckan.net/package/edit/dbpedia for examples.

A link to the “register new package” page would be handy somewhere on the front page. As a frequent editor, I often go to ​http://ckan.net in order to create a new package. I don't want to hunt around for the link to the “new package” form.

The link could go into the bar next to “About / Statistics”. Or into the “More information” sidebar. Neither of these truly make sense, but it's a frequently accessed link and that's reason enough to put it somewhere onto the frontpage.

The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page.

There are some problems with the Resources table on package pages:

1. The column labelled “URL” doesn't contain URLs, but just the word “Download”.
2. The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links).

Attached is a screenshot for a proposed redesign. The changes are:

1. Move the Format column to the right of the Description column (description is more important)
2. Make the description a clickable link
3. Drop the “URL/Download” column because it's now redundant
4. Rename “Description” to “Resource” (not that important)

The widget for tag autocompletion is broken in various ways.

For example, if I edit a package that is tagged "music", and just tab through the form fields to get to the Author field, then the widget changes to "industrial-music" as I tab through it.

Or if the tag is "foo bar" and I hit alt-right to jump to the end of the text field (with the intent of adding a third tag), then the contents change to "foo barbecue".

Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end.

Here are the details:

• in CKAN's package view (in web interface, API and dump) no resources display
• but they WERE created in CKAN, as shown by the revision diffs.
• they are in the Drupal front-end

So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal.

Packages affected: anti-social-behaviour-orders-1999-2007 asylum-applications-jan-mar-2009 control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december coroners-statistics-england-and-wales courts-statistics-user-survey-england-and-wales court-statistics-company-insolvency-and-bankruptcy-england-and-wales court-statistics-england-and-wales court-statistics-mortages-and-landlord-possession-england-and-wales crime-in-england-and-wales crime-statistics-local-reoffending-england-and-wales crime-statistics-prison-and-probation-england-and-wales crime-statistics-reoffending-of-adults-england-and-wales crime-statistics-reoffending-of-juvenilles-england-and-wales data_gov_uk-datasets digest-uk-energy-statistics-2008 directgov-central-hottest-pages-monthly directgov-central-internal-search-terms-monthly directgov-section-visits-monthly electricity-consumption-2007 electricity-gas-consumption-2007 energy-consumption-uk-2008 final-energy-consumption-2007 foi-statistics-uk-central-government fuel-poverty-statistics-2007 gas-consumption-2007 gb-reported-bicycling-accidents gb-road-traffic-counts gb-traffic-matrix greenhouse-gas-emissions-2008 high-level-indicators-energy-use-2006 judicial-and-court-statistics-england-and-wales laboratory-tests-and-prices local-authority-carbon-dioxide-emissions-2007 magistrates-courts-statistics-survey-england-and-wales monthly-energy-prices monthly-energy-trends ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig ni_013_migrants_english_language_skills_and_knowledge ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so ni_026_specialist_support_to_victims_of_a_serious_sexual_offence ni_029_gun_crime_rate ni_031_re-offending_rate_of_registered_sex_offenders ni_032_repeat_incidents_of_domestic_violence ni_034_domestic_violence_-_murder ni_036_protection_against_terrorist_attack ni_038_drug_related_class_a_offending_rate ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an ni_109_delivery_of_sure_start_childrens_centres ni_126_early_access_for_women_to_maternity_services ni_127_self_reported_experience_of_social_care_users ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law ni_185_co2_reduction_from_local_authority_operations ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an other-fuels-consumption-2006 police-use-firearms-england-wales-2007-2008 prison-end-of-custody-licence-releases-and-recalls-england-and-wales prison-population-england-and-wales probation-offender-management-caseload-statistics-england-and-wales probation-statistics-quarterly-brief-england-and-wales quality-indicators-energy-data-2007 quarterly-energy-prices quarterly-energy-trends road-transport-energy-consumption-2007 sentencing-statistics-england-and-wales statistics-terrorism-arrests-outcomes-2001-2008 ukba-control-of-immigration-statistics-2008 ukba-control-of-immigration-statistics-2008-supplementary-tables uk-energy-in-brief-2008 uk-energy-sector-indicators-background-2008 uk-energy-sector-indicators-key-supporting-2008 uk-exportcontrollists uk-exportcontrol-sanctions uk-export-control-statistics uk-glossary-exportcontrol uk-ipo-offences weekly-fuel-prices

1. Create a package.
2. Edit package and add a resource.
3. View package in browser - no sign of the resource - ERROR.
4. View package in API - resource is there.

Reason: hash doesn't change because package.revision only takes account of core fields.

Intermittent failure on ckan default branch. Looks like brittle test.

======================================================================
FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py", line 250, in test_02_metadata_created_and_modified
    assert out == exp, (out, exp)
AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938))

​http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio

Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception.

Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc.

Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:

• Purge revisions (or sets of revisions) and purge objects #1076
• Set roles for users #1075
• Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
  • WONTFIX
• Overview of activity
  • WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.

​https://bitbucket.org/okfn/ckanext-admin

Tickets

• ticket:1031 - user autocomplete api in ckan

Notes

Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of 
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system

As a sysadmin I want to take CKAN out of wiki-mode into normal mode. In normal mode visitor and logged in users cannot create or add packages.

Suggest put this as an internal command in authz system and add to cli as paster command.

Thread: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

Scenario 1

1. User installs Refine and CKAN extension for refine
2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
3. They edit the dataset on Refine
4. On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)

NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required)

Scenario 2

1. User visits a package on CKAN.net (or another CKAN instance)
2. There is a button on the page "View and edit this dataset in Google Refine"
3. Click button -- ask them if they have Google refine installed
   • Yes: instructions for loading dataset into refine
   • No: load dataset in hosted version of google refine (we could run this)
4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset.

as per http://ckan.org/wiki/UIRedesignPackage

Documentation article on caching / improving performance. (To complement configuration docs.)

• Different sorts of cache - beaker style, etags, package_dict in search results(?)
• How each one affects performance
• How to turn them on/off and configure them
• Is it possible to bypass each of them in the browser or with wget/curl?

• perhaps we should show the openid as well to distinguish between users with the same name.

• perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name.

• also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it.

Allow a form to specify fields that are required. Currently name and resource url are required.

• Show required fields with asterisks by the input box.
• Resource fields to be included in this.

This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010).

3 key aspects of this:

• Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN)
  • Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable
  • Asynchroously: via worker with queue
    • #324 Search indexing using notifications
    • #669: Refactor Solr (indexer) to become a generic worker
    • #874: Extract Solr search backend into an extension
• In core support using SOLR backend as well as postgres for queries
  • #317: Make search pluggable (?)
• Integrate into WUI (if changes needed)
  • #672 - Facets in CKAN search results

Extras:

• Solr index testing tool #431

The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import.

revisit link checker from ​http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py

revisit ollyc's parallel work

look at how this gets used in practice (maybe deprecate in favour of curate tool (suitably wrapped to hide details from the user)

Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys.

This would lead to an extension to OFS.

This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN).

• master ticket #852

The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests.

Test coverage for ckan.lib.cli is 34%. Need to improve that.

Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement?

​http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d

e.g. license, owner etc

• Provide and include a "language.js" file.

To be more pythonic, raise exceptions when ckanclient gets status which isn't 200.

Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)

The infamous ​exim bug only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers.

We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not.

BTW did we already run a rootkit checker like ​Rootkit hunter on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark? has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could.

e.g. national_child_measurement_programme had no department but "Information Centre for Health and Social Care" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_.

Consolidate search API interface (and backend) on solr (solrpy) type interface.

• Support for standard query structure
• Support for facet options

Do not need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

1. Disable (happens automatically)
2. Implement - suggest using group by etc

Extras

• Front-page tag cloud: change this to use facets
  • Accept this means that if facets not functional in backend we have no tag cloud

Support uploading files to a storage system from WUI with a simple upload form.

Implementation

• Implement as an Extension
• Locate at /upload
• Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
• Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
• This wants to be tied in to general package workflow. See parent ticket:852.

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code.

master ticket

• #884 implement harvest job delete command for an easier life
• #885 move csw code into dedicated ckanext plugin
• #886 move forms api into dgu extension module
• #887 move harvesting code to use generic harvesting module

in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat

Add a config option 'template_footer_end' which is inserted in layout_template just before </body>.

This allows sysadmins to add extra items, especially, scripts directly into site without having to do any theming and is especially useful for things like google analytics.

Aside: going forward may want to turn this into a extension.

Cost: 1h

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository.

Questions

• Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
• Should we have clever ways to dump APIs? ANS: no.
• Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
• Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
  • Complete support for ETags
  • Expires, Max-Age etc.
• Check

Functionality

• Download files via HTTP, HTTPS (will not do FTP)

Process:

1. [Archiver.Update checks queue (automated as part of celery)]
2. Open url and get any info from resource on cache / content-length etc
   1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
   2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get content-length and content-length unchanged GOTO step 4
3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
   • Add cache url to resource and updated date
   • Add other relevant info to resource such as md5, content-type etc
4. Update task_status

Optional functionality

• If result object is HTML, search for references to "proper data" (CSV download pages etc.)
• Download from POST forms (accepting licenses or weird proprietary systems)
• Support running on Google Apps Engine to save traffic costs.

Existing work

• ​https://bitbucket.org/okfn/ckanext-qa/overview
• out of date: ​https://bitbucket.org/pudo/ckanextarchive - Old archiver extension, largely experimental.
• out of date: ​https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65 - Openness scores by ollyc

At the moment, ExtrasField? cannot be used in conjunction with more specific extra fields, such as TextExtraField? or SuggestTextExtraField?.

Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g.

mycssfile.css?v=12345678

On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted.

There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. ​https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e

From David Raznick