Estimate: 4-5d

Only sysadmins can manage harvest sources right now, we need to allow publishers to handle their own sources.

We need to decide if there will be two different profiles on ckanext-harvest, the current one with only sysadmins allowed to manage the harvesting and a new one supporting publishers (ckan core can be configured to not use the publisher model)

Tasks include:

• Auth checks on the different controller actions (and the command line interface?)
• Modify the index page to just show sources the logged user has permissions on.

(Talk with Ross about the specific functions to use)

• Create/edit form to show a dropdown with the user available publishers.

Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones.

Estimate 2 weeks for someone to finish and test.

Essentially, the functionality should be as follows:

• Add logout link that has .js-hide attached to it within the header that isn't hidden within a dropdown

See ​http://plus.google.com/ (when logged in) with and without JS to see an example of the actual sign-out working without JS

Errors are caused when harvesting documents. This also makes the count not show up correctly on the ckan search page.

Ensure publisher edit page only allows admins to manage users. #1675 is related.

Ensure logging is working on server.

Enable the organizations and organizations_dataset plugins. Start CKAN. Login and create a new organization. Logout. Login with a different account that is a sysadmin. Go to the organization you just created with the other account. There are Edit and Join buttons. Click on the Edit button, and the Join button changes to Authorization.

It should probably be an Authorization button right from the start if you are a sysadmin.

Enable the multilingual plugins:

ckan.plugins = stats synchronous_search multilingual_dataset multilingual_group multilingual_tag

and set your default language to one not supported by the multilingual plugin, e.g.

ckan.locale_default = cs_CZ

now run CKAN and try to add a dataset:

File '/home/seanh/Projects/ckan171/ckan/ckanext/multilingual/plugin.py', line 141 in before_index

text_field_items+ default_lang?.extend(all_terms)

KeyError?: 'text_cs_CZ'

It doesn't matter what language you are viewing the site in in your browser, the default language setting in the ini file determines whether it crashed or not.

A number of supported languages are defined at the top of ckanext/multilingual/plugin.py. I think if the default language is not one of these it crashes.

I think this affects all versions of CKAN since the multilingual plugin was added so at least 1.7, 1.8 and 2.0

Either: thedatahub.org does not display 'add a dataset' top navigation link or the 'create dataset' main central link to logged in user - instead showing 'sign up' link in the central box. It does however display the 'my account' and 'logout' link in the top right correctly.

Navigating to another part of the site (e.g. search or about or my account) does bring the 'add a dataset' navigation link back and the functionality works.

Works fine on test.ckan.net so only on thedatahub.

Not obviously a caching problem, since I did try clearing cache.

OR: it displays as if you're logged in even when you log out. The links doesn't work and prompt you to log in though.

Editing groups in forms doesn't work for me, with latest code from this morning:

1. Clean db
2. paster create-test-data
3. paster sysadmin create ​http://davidread.myopenid.com/
4. paster serve development.ini
5. In browser, log in to CKAN
6. Create new package: name=abc group=Roger's books
7. [Preview] - yes group appears
8. [Save] - shows package and group hasn't appeared ERROR
9. Check ​http://127.0.0.1:5000/api/rest/package/abc and reedit package it also doesn't appear here either.

Editing a group properties on ckan.net gives an exception when you submit the form. (Is this related to db migration issues?)

Example page: ​http://ckan.net/group/edit/civil-society

Exception:

WebApp Error: <class 'sqlalchemy.exc.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "group_revision_pkey" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'description': u"A group for open data related to civil society supported by members of the interest group at: http://okfn.org/groups/civil-society\r\n\r\nCivil society is composed of the totality of voluntary civic and social organizations and institutions that form the basis of a functioning society, as distinct from the force-backed structures of a state (regardless of that state's political system) and commercial institutions of the market. \r\n\r\nhttp://en.wikipedia.org/wiki/Civil_society", 'created': datetime.datetime(2011, 2, 24, 14, 30, 53, 334842), 'title': u'Civil Society', 'state': u'active', 'continuity_id': u'f4f9f09

Edit a dataset and add an author, dataset read page still says 'author not given'. If you fill out both the author and author email fields then it works.

Each dataset should have an explicit owner that is an organisation. This is the organisation that owns the dataset* and will be used for specifying who can move a dataset out of the default organisation.

This should probably be a foreign key link from dataset to organisation (or rather group).

• Currently this won't necessarily be the organisation that the dataset is a member of, but it is likely that this will be the case initially (at least until someone moves it to another organisation).

E.g. uploading a file with spaces in it: OECD Monthly Exchange Rates.zip

Gives a 404 Not Found response to the following file: ​http://test.ckan.org/api/storage/metadata/2011-12-05T193046/OECD%20Monthly%20Exchange%20Rates.zip

The upload wheel keeps spinning and the user is not informed that the upload has failed.

Analysis: turns out that google storage (possibly s3 but not checked) replaces with ' ' in keys with '+' on upload. This breaks things because we try and look up metadata about upload using the filename/key we put in but of course that does not exist because google has changed name.

Fix is trivial: replace ' ' in keys / filenames with '-'.

E.g. ​http://s031.okserver.org:2375/en/dataset/adur_district_spending/resource/04127ad5-77e5-4a08-9f40-12d3c383e460

When there are no maintainer/contact details or extra fields we should hide the additional info box completely. Otherwise hide the last updated and licence fields which are already displayed in sidebar.

Update sidebar to provide context - so 'CSV' -> 'CSV format' + 'unknown' -> 'Last updated: unknown'

E.g. ​http://ckan.net/tag/lod contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins).

Also suggested that these are showing up in the API (courtesy of Anja)

Cost: 2h

During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box.

Dumping ground for various minor and miscellaneous items (mainly refactorings):rgrp

• controllers/tag.py: why does this not use lib/search.py (but controllers/packages.py search method does)?

done:

• lib/search.py: inheritance would be nicer than switching on entity in search (i.e. have SearchPackage?, SearchTag? etc)

Download Command (was install command) should support the following modes:

• Download only the first listed resource (current behaviour, slightly arbitrary)
• Download resources by interactive selection
• Download resources by MIME type (cf #235)
• Download all resources.

Done on branch feature-1698-tag-taxonomies

Documentation is key to the success of projects!

Move material into main docs from wiki

• ​http://wiki.ckan.org/Writing_asynchronous_tasks
• ​http://wiki.ckan.org/Domain_Model
• ​http://wiki.ckan.org/Deployment

Create a User Guide

Basic step-by-step on getting started with CKAN (See start on set of slides here e.g. ​http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/)

• Publishing data
• Authorization and workflows
  • Publisher Profile (Workflow)

Break into sections

Suggest something like:

# General
* Intro
* Conceptual Overview
* What CKAN helps you do (http://ckan.org/)
* FAQ ? (or point to the wiki)

# For Administrators

* Installation and Setup
  * Configuration options
* Customization - Theming etc
* Authorization and Workflows
* Storage

# For Users (Publishers, Data Wranglers, etc etc)
* Walkthrough of publishing a dataset
* Storage
* API (see Developers section)

# CKAN Developers

* Domain Model
* API
* Extensions
* i18n

CKAN Developers
* Buildbot

Document the ideas raised on 14/03 for CityData? and distribute to team for discussion

Document deployment on tenforce test server.

Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh?

Display league of users' recent activity on homepage.

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998513/comments

Implementation:

​http://s031.okserver.org:2375/en/stats

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments

Implementation:

​http://s031.okserver.org:2375/en/user/reset

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments

Implementation:

​http://s031.okserver.org:2375/dataset/adur_district_spending/related/new

Needs a new endpoint at /dataset/DATASET/related/new this should have a form containing the following fields.

• Title (required)
• Type of item (Application|Visualisation)
• Description
• URL (required)
• Image URL

When submitted if an item is created it should redirect (303) back to /dataset/DATASET/related with a flash message saying "Related item was successfully created".

If failed to create it should redirect (303) back to the form and populate the error messages, see add dataset for examples (​https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/package/new_package_form.html)

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62308209/comments

Implementation:

(to be added once test server is working)

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

​http://s031.okserver.org:2375/user/aron

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62234769/comments

Implementation:

​http://s031.okserver.org:2375/dataset/new ​http://s031.okserver.org:2375/dataset/new_resource

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

​http://s031.okserver.org:2375/group

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61752357/comments

Implementation:

​http://s031.okserver.org:2375/dataset/adur_district_spending

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments

Discussion between John and Peter:

Given we can identify a document, does the disappearance of a document from a registered source imply the disappearance of the metadata (such that we delete packages once the documents disappear from the registered source)?

I would expect a more explicit 'delete'. The UKLP Use Case Library describes this as "withdraw a dataset or service from publication" (part of UCD03 Maintain resources).

Details of how to go about this here <​http://www.rufuspollock.org/2009/07/27/sqlalchemy-migrate-with-pylons/>

Cost: 2h

Depends on #2639

Deleting extras in the web interface is broken

Deleted packages with the same title and extra fields as an active package trip up the loader. It finds them in the search, but due to ckanclient not using the apikey when you 'get' the package to examine it further, it causes an exception.

Definitely do:

• Local file filestore: Remove pairtree (and OFS) and do something very simple

Options:

1. Local file only. Allow for uploaders to GS / S3 in the background
   • Advantages: simpler, if local upload can do progress bars etc etc
   • Disadvantages: strain for web-app (upload a 2GB file what happens)
     • This is probably fixable ...
2. Stick with how we are

New filestore without pairtree

Each uploaded file is located on disk at:

{uuid}/{filename}

Alternative:

yyyy/mm/dd/{uuid}/{filename}

Need the uuid to avoid collisions.

Metadata: Store no metadata.

Decide which extensions are important.

Datastore_url enabled checks are useless.

We should probably change this to datastore_url = 'active' and only update the flag to when a attempt at a post is made.

This will need to be removed from the form at some point.

Datasets listed in search results have not gone through schema validation (but on the user page they do), so package_list snippet currently has to display both 'notes' and 'description'.

Dataset read page for DGU

Breakdown:

• [X] Fix missing title
• [X] Links to the social media stuff
  • "Share your app"
  • "Share your idea"
  • "Request new data"
• [ ] Presentation of additional information should be broken into sections

Content is there. Don't do anything until theming/design work is undertaken.

• [ ] "More like this": links to similar datasets.

Dropping

• [N] Stars (I think these are being removed)
• [ ] Comments

Possible integration point with Drupal. Needs it own ticket.

• [X] Give feedback to department
• [X] Tidy up the QA stars

Database for all CKAN instances upgraded to Postgres 8.4, but none of the eu machines were upgraded with the tools necessary to administer them.

Data viewer / previewer for a resource that has a KML file, or others, such as GeoRSS and GeoJSON, we should provide a map.

• Support for KML files: #1458

This is non-trivial for external files as we need a way to jsonify. For files stored locally this is more of a recline issue (and will require a bit of work to either guess columns or allow user to specify them).

DSPL, the Dataset Publishing Language, is being promoted by Google for its "Google Public Data Explorer" system. It is an XML format with metadata.

The format is described on the ​developer docs ofthe Google Code site.

Google provides a ​Python script which reads CSV data and generates DSPL

Sample from ​http://code.google.com/apis/publicdata/docs/dspl_sample.html:

<?xml version="1.0" encoding="UTF-8"?>
<dspl xmlns="http://schemas.google.com/dspl/2010"
    xmlns:geo="http://www.google.com/publicdata/dataset/google/geo"
    xmlns:geo_usa="http://www.google.com/publicdata/dataset/google/geo/us"
    xmlns:time="http://www.google.com/publicdata/dataset/google/time"
    xmlns:quantity="http://www.google.com/publicdata/dataset/google/quantity"
    xmlns:entity="http://www.google.com/publicdata/dataset/google/entity">

  <import namespace="http://www.google.com/publicdata/dataset/google/time"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/quantity"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/entity"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/geo"/>
  
  <info>
    <name>
      <value>My statistics</value>
    </name>
    <description>
      <value>Some very interesting statistics about countries</value>
    </description>
    <url>
      <value>http://www.stats-bureau.com/mystats/info.html</value>
    </url>
  </info>

  <provider>
    <name>
      <value>Bureau of Statistics</value>
    </name>
    <url>
      <value>http://www.stats-bureau.com</value>
    </url>
  </provider>

  <topics>
    <topic id="geography">
      <info>
        <name><value>Geography</value></name>
      </info>
    </topic>
    <topic id="social_indicators">
      <info>
        <name><value>Social indicators</value></name>
      </info>
      <topic id="population_indicators">
        <info>
          <name><value>Population indicators</value></name>
        </info>
      </topic>
      <topic id="poverty_and_income">
        <info>
          <name><value>Poverty & income</value></name>
        </info>
      </topic>
      <topic id="health">
        <info>
          <name><value>Health</value></name>
        </info>
      </topic>
    </topic>
  </topics>

  <concepts>
    <!-- As noted in the tutorial, this concept should extend quantity:amount.-->
    <concept id="population">
      <info>
        <name>
          <value>Population</value>
        </name>
        <description>
          <value>Size of the resident population.</value>
        </description>
      </info>
      <topic ref="population_indicators"/>
      <type ref="integer"/>
    </concept>

    <!-- This country concept is defined for educational purposes only. A country
    concept exists in the Google geo dataset. See:

    http://code.google.com/apis/publicdata/docs/canonical/geo.html --> 
    <concept id="country" extends="geo:location">
      <info>
        <name>
          <value>Country</value>
        </name>
        <description>
          <value>My list of countries</value>
        </description>
      </info>
      <type ref="string"/>
      <property id="name">
        <info>
          <name><value xml:lang="en">Country name</value></name>
          <description>
            <value xml:lang="en">The official name of the country</value>
          </description>
        </info>
        <type ref="string"/>
      </property>
      <table ref="countries_table"/>
    </concept>

    <!-- This US state concept is defined for educational purposes only. A US state
      concept exists in the Google geo US dataset. See:

      http://code.google.com/apis/publicdata/docs/canonical/geo.us.html --> 
    <concept id="state" extends="geo:location">
      <info>
        <name>
          <value>State</value>
        </name>
        <description>
          <value>US states</value>
        </description>
      </info>
      <type ref="string"/>
      <property concept="country" isParent="true"/>
      <table ref="states_table"/>
    </concept>

    <concept id="gender" extends="entity:entity">
      <info>
          <name>
          <value>Gender</value>
        </name>
        <description>
          <value>Gender, Male or Female</value>
        </description>
        <pluralName><value>Genders</value></pluralName>
        <totalName><value>Both genders</value></totalName>
      </info>
      <type ref="string"/>
      <table ref="genders_table"/>
    </concept>

    <concept id="unemployment_rate" extends="quantity:rate">
      <info>
        <name>
          <value>unemployment rate</value>
        </name>
        <description>
          <value>The percent of the labor force that is unemployed, not seasonally
            adjusted.</value>
        </description>
        <url><value>http://www.bls.gov/cps/cps_htgm.htm</value></url>
      </info>
      <topic ref="social_indicators"/>
      <type ref="float"/>
      <attribute id="is_percentage">
        <type ref="boolean"/>
        <value>true</value>
      </attribute>
    </concept>

  </concepts>

  <slices>
    <slice id="countries_slice">
      <dimension concept="country"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <table ref="countries_slice_table"/>
    </slice>

    <slice id="states_slice">
      <dimension concept="state"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <metric concept="unemployment_rate"/>
      <table ref="states_slice_table"/>
    </slice>

    <slice id="countries_gender_slice">
      <dimension concept="country"/>
      <dimension concept="gender"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <table ref="countries_gender_slice_table"/>
    </slice>

  </slices>

  <tables>
    <table id="countries_table">
      <column id="country" type="string"/>
      <column id="name" type="string"/>
      <column id="latitude" type="float"/>
      <column id="longitude" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">countries.csv</file>
      </data>
    </table>

    <table id="countries_slice_table">
      <column id="country" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <data>
        <file format="csv" encoding="utf-8">country_slice.csv</file>
      </data>
    </table>

    <table id="states_table">
      <column id="state" type="string"/>
      <column id="name" type="string"/>
      <column id="country" type="string">
        <value>US</value>
      </column>
      <column id="latitude" type="float"/>
      <column id="longitude" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">states.csv</file>
      </data>
    </table>

    <table id="states_slice_table">
      <column id="state" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <column id="unemployment_rate" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">state_slice.csv</file>
      </data>
    </table>

    <table id="genders_table">
      <column id="gender" type="string"/>
      <column id="name" type="string"/>
      <data>
        <file format="csv" encoding="utf-8">genders.csv</file>
      </data>
    </table>

    <table id="countries_gender_slice_table">
      <column id="country" type="string"/>
      <column id="gender" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <data>
        <file format="csv" encoding="utf-8">gender_country_slice.csv</file>
      </data>
    </table>
  </tables>

</dspl>

Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3).

That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's "<VirtualHost?>", or tracd). Depending on the "Host:" header in the HTTP1.1 request, it would choose which local ckan ini file to obey.

I see two ways to constitute the map hostname-to-ini-file map:

• ckan reads a set of ini files, and each ini file declares which servers names it is responsible for
• In a global ini file, there are directives mapping servernames to ini files.

In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc.

Currently, as a logged in admin, you can view deleted datasets just as if they are live datasets. Nothing distinguishes them apart from normal datasets other than they don't show up on the dataset page

This is potentially very confusing Can we add some kind of banner / label to deleted datasets to show they have been deleted?

& Do you guys need Sam to design this?

Related ticket for undeleting datasets - ​http://trac.ckan.org/ticket/2780