CKAN: Ticket Query

#2757: Write documentation for upgrading a CKAN install from an older to a newer version of CKAN

seanh — Wed, 25 Jul 2012 16:54:02 GMT

#1563: Finish Data Storage

jilly mathews — Mon, 19 Dec 2011 10:59:06 GMT

Unsure what needs to happen here. Need to list outstanding tasks and implement.

#1566: [super] Finalize Google analytics extension

jilly mathews — Mon, 19 Dec 2011 11:12:48 GMT

~~(Deploy it)~~
Integrate with stats plugin (now that is in core) - #1101
Blog about it #1581
(Other tickets re upgrading for v1.5.1 - done but where are tickets?)

#1568: Moderated Edits

jilly mathews — Mon, 19 Dec 2011 11:22:56 GMT

Can this be released as a standard CKAN feature?

#1570: Integrated file Storage

jilly mathews — Mon, 19 Dec 2011 11:29:02 GMT

Is this ready for release? What needs to be done?

#698: CKAN Data API v1

thejimmyg — Thu, 14 Oct 2010 16:23:53 GMT

This proposal is to discuss adding a new API for proxying certain spreadsheet data via JSON-P to make it possible to build simple browser apps directly off the API.

See the attached proposal for information.

#736: URL validation for CSW and WMS links

johnbywater — Fri, 22 Oct 2010 12:10:16 GMT

Robust against slightly incorrect URLs (e.g. with trailing slash, or without /csw), and non-CWS service which also return <ows:ExceptionReport ... />.

Both cases would be resolved by checking capabilities (with CswClient's check_get_capabilities() method).

#1030: Move harvesting out of the rest API

thejimmyg — Thu, 10 Mar 2011 17:00:18 GMT

#1037: More Robust Harvesting for DGU

thejimmyg — Tue, 15 Mar 2011 14:00:02 GMT

CKAN's harvesting facility is now live on DGU but there are some major improvements that could be made to make it more robust and better fit the generic CKAN harvesting framework proposed in #987.

Some of the key issues:

Error reports do not currently contain the ID or title of the document with the error.
We only have "added" and "error" logging on jobs when we really need a report of "added", "updated", "not changed" and "errors" with the items in each referencing a real metadata document for which harvesting was attempted
We need deletion and editing of sources, without deleting the harvested documents or packages
We need a more robust harvesting mechanism than a cron job or we need to deal with the case of multiple cron jobs running at once.
We need to know the last time a list of documents was scheduled for harvest and the last time each one was fetched.

#1160: Havest CSW servers on PDEU

amercader — Thu, 26 May 2011 10:54:31 GMT

It could be interesting for the publicdata.eu project to add some CSW datasources. Thanks to the INSPIRE directive, there are quite a few national catalogs of geodata that could be worth harvesting. Work on this should include:

Compose Endpoint List (1d)
Harvest on staging system and evaluate which sources have useful material (2d)

First point is being performed in this GDoc: https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&authkey=CNu4hsEB&hl=en_US#gid=0

#1162: Define a common extra for storing the geographic extent

amercader — Thu, 26 May 2011 11:18:24 GMT

We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city).

Relevant discussion: http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html

#1164: Cloropleth Map of European Data Availability for PDEU

amercader — Thu, 26 May 2011 11:20:24 GMT

A nice map in the homepage showing the availability of data across Europe

#1166: Less stringent filters for displaying the WMS preview

amercader — Thu, 26 May 2011 15:10:46 GMT

Right now, for the "View available WMS layers" link to be displayed, the package needs to have an "INSPIRE=True" extra, a "resource-type=service" extra and a resource of type "WMS".

This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough.

(The repository for this is ckanext-spatial, but it's not on the list)

#1173: Offer a DCat representation for packages in the API

amercader — Wed, 01 Jun 2011 14:47:19 GMT

/api/rest/package/foo.rdf should return a DCat representation of the package. To create it, we will use the functions in ckanext-rdf.

#1253: Authz refactor

amercader — Mon, 01 Aug 2011 11:51:31 GMT

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

#1301: Give access to extensions from the old formalchemy controllers

amercader — Thu, 01 Sep 2011 13:14:48 GMT

This was removed during the auth refactoring, but it's needed in the old controllers

#1303: User can not login again if he has edited his profile

amercader — Thu, 01 Sep 2011 15:04:32 GMT

Only happens if the user does not update the password. Most probably introduced during #1229.

#1304: Show more descriptive error messages when sending emails

amercader — Thu, 01 Sep 2011 16:06:46 GMT

Sending emails is failing in some cases, and the error message just says 'Could not send reset link'.

#1333: Move form converters to ckan core

amercader — Tue, 13 Sep 2011 10:21:08 GMT

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

#1334: "AttributeError: 'NoneType' object has no attribute 'name'" exception when logging in

amercader — Tue, 13 Sep 2011 10:25:08 GMT

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'

#1337: Group extras are JSON endoded in the REST API responses

amercader — Tue, 13 Sep 2011 14:38:47 GMT

Instead of this:

extras: {
type: ""Primary source""
license_id: ""localauth-withrights""
publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""
}

we should return this:

extras: {
type: "Primary source",
license_id: "localauth-withrights",
publisher_frequency: "2Preño € $ àdrià",
}

#1338: Exception when trying to use a custom form schema from an extension

amercader — Tue, 13 Sep 2011 14:46:57 GMT

Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'

#1354: Pass the context object to plugins implementing the IPackageController and IGroupController

amercader — Sun, 25 Sep 2011 13:01:20 GMT

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

#1375: Define new package id before sending it to the IPackageController plugins

amercader — Thu, 06 Oct 2011 14:14:01 GMT

When creating a package, the package_create logic function sends the newly created package object to all plugins implementing the IPackageController interface

https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64

At this point, the changes have not yet been commited and the id of the package is not set (It has a default value of make_uuid, which is set when commited). For this reason, plugins have no way of knowing the package id. E.g. the spatial query plugin needs to know this id to link the package extent to the actual package.

One possible solution would be to set the package id in the data dict received by package_create (if it's not defined):

from ckan.model.types import make_uuid
if not 'id' in data or not data['id']:
   data['id'] = make_uuid()

#1381: Managing groups via package entity API

pudo — Mon, 10 Oct 2011 17:11:46 GMT

It is not currently possible to assign or remove groups by adding or removing their group name from the "groups" list on the REST API. This should be allowed, as it is easier than first adding a package and then editing each group it must belong to.

#1456: Use resource description instead of name if both are present

amercader — Wed, 09 Nov 2011 18:16:59 GMT

If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those.

#1462: Multi-Instance CKAN Solr Search Issues

thejimmyg — Tue, 15 Nov 2011 16:44:30 GMT

With the 1.5 package release, CKAN instances over-write each other's indexes. Adria has a fix.

I'd like to see a new CKAN 1.5.1 release with the Solr changes merged and the Solr site ID set up as part of the install ckan.site_id = dgu-os.okfn.org. I'd also like the ability to search on ID and to have the ID directly visible on the package page.

We need to decide how to do this upgrade without affecting existing CKAN solr indexes.

#1469: Geo-spatial search on the front end

thejimmyg — Tue, 15 Nov 2011 16:58:44 GMT

Now that we have a geospatial search API, we should build a front-end that integrates with the existing search for displaying results.

#1490: Standardize output from package listings coming from the logic layer

amercader — Fri, 25 Nov 2011 12:47:19 GMT

Right now, the two logic functions that return a list of packages (package_search [1] and group_packages_list [2])use custom functions to generate the output dict. That's suboptimal because:

The template function that renders the package listing (package_list_from_dict) could end up receiving different dicts.
Not all package properties are available at the template level (e.g extras, tags...). These can be really useful to customize the listing from an extension.

In general only the functions present in lib/dictization/model_dictize.py should be used to build the output of a logic function, in that case package_dictize. If necessary, they can be modified to include missing properties, like on this particular case the "isopen" property, needed by the template renderer.

[1] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L685 [2] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L442

#1498: Support multiple SOLR schema versions

amercader — Tue, 29 Nov 2011 16:47:44 GMT

At some point the SOLR schema used by CKAN will need to be changed and the changes won't be always backward compatible, so we need a mechanism to support instances that use older versions of the schema. This process will involve:

Better documentation on how to deploy SOLR, as single instance or multicore
A convention on how to tag schema versions and SOLR endpoints to ensure that CKAN instances point to the correct SOLR URL.
(If possible) Checks in CKAN core that deal with different schema versions.

#1516: Update SOLR schema after 1.5.1 release and set up multicore SOLR instances if needed

amercader — Mon, 05 Dec 2011 18:07:11 GMT

CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like:

http://<solr-server>/solr/schema-<version>

Different scenarios:

Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine):
- Update CKAN
- Update SOLR schema (symlink to suitable version in CKAN source)
- Rebuild search index

Single SOLR instance used by multiple CKAN sites.
- Update CKAN source
- Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source)
- Update solr_url on each of the CKAN sites with the suitable core
- Rebuild search index

These are the SOLR and CKAN instances that need to be updated (Please add any missing ones):

Same machine as the CKAN site:
- ~~test.ckan.net~~
- ~~iati.test.ckan.net~~
- data.gov.uk/ catalogue.data.gov.uk (confirm)

~~s046.okserver.org~~
- ~~iatiregistry.org~~

~~s052.okserver.org~~
- ~~datacatalogs.org~~

~~s004.okserver.org / eu4.okfn.org / solr.okfn.org~~
- thedatahub.org (=www.ckan.net)
- datagm.org.uk
- ~~publicdata.eu~~
- hri.fi (dev.fvh.fi?)
- it.ckan.net
- ie.ckan.net
- cz.ckan.net
- register.data.overheid.nl (nl.ckan.net?)
- no.ckan.net / datakilder.no
- br.ckan.net
- colorado.ckan.net
- at.ckan.net

TODO: which SOLR server are using these instances?

data.norge.no
nederland.ckan.net
lt.ckan.net
pl.ckan.net
datadotmontreal.ca/
ca.ckan.net / datadotgc.com

#1540: Search API returns an error if empty parameters are provided

amercader — Thu, 08 Dec 2011 15:49:48 GMT

Both in 1.5.1b:

http://thedatahub.org/api/search/dataset?groups=lodcloud&title=

and 1.5.2a (current master):

http://test.ckan.net/api/search/dataset?groups=lodcloud&title=

Although the error message in 1.5.2a is more verbose:

"Bad request - Bad search option: HTTP code=400, reason=org.apache.lucene.queryParser.ParseException?: Cannot parse 'groups:lodcloud title:': Encountered \"<EOF>\" at line 1, column 22. Was expecting one of: \"(\" ... \"*\" ... <QUOTED> ... <TERM> ... <PREFIXTERM> ... <WILDTERM> ... \"[\" ... \"{\" ... <NUMBER> ..."

Some parameter validation before sending it to Solr should do the trick

#1547: Add search methods (before and after) to the IPackageController interface

amercader — Tue, 13 Dec 2011 12:23:16 GMT

Time estimate: 1d

Add a two search method to the IPackageController interface so extensions can:

Modify the query being sent to Solr

def before_search(search_params):
    return search_params

Modify the results returned (after_search)

def after_search(search_results, search_params):
    return search_results

Please see this pad for implementation details:

http://ckan.okfnpad.org/search

#1590: Create customized feeds for the IATI Registry

amercader — Wed, 21 Dec 2011 17:02:45 GMT

We need a way to track changes on the registry (datasets edited or updated), globally and on a per country/publisher/etc. basis. RSS and Atom feeds are really popular, and after closing #191 and #1498 crating them from the search results should be fairly easy.

The following URLs are pretty self-explanatory:

http://localhost:5000/feed/registry.rss
http://localhost:5000/feed/country/AF.rss
http://localhost:5000/feed/publisher/worldbank.rss
http://localhost:5000/feed/organisation_type/10.rss

As we need to implement custom wrappers for countries, publishers, etc, we might as well offer a fully customizable feed, e.g.:

http://localhost:5000/feed/custom.rss?q=activity_count:[* TO 100]
http://localhost:5000/feed/custom.rss?publishertype=primary_source

Apart from the actual feeds, there will be a small amount of work at the template level to add the links to the suitable pages (and maybe a generic page showing all available feeds)

#1592: Add metadata_modified and metadata_created to package_dictize output

amercader — Wed, 21 Dec 2011 17:35:09 GMT

The dict returned by package_dictize does not include metadata_modified and metadata_created. These are really useful properties, so it's worth having them on the standard package dict representation, which is used in several places, like at the template level.

#1601: [super] Well integrated Geo-spatial support including search

rgrp — Fri, 30 Dec 2011 15:43:41 GMT

Need thoughts from Adria and James. RP thoughts

Integration into WUI search: #1469
JS Editor for adding/editing spatial attribute: #1818
Theme/template hook for adding the map e.g. on dataset page (or elsewhere)
Clear Documentation
Deployment
Announce post
(?) Map page for stats section showing location of datasets

Extras (probably separate):

csw support ...

#1616: Catch exceptions when rebuilding the search index

amercader — Fri, 06 Jan 2012 10:11:09 GMT

Right now if an exception is found while reindexing, the whole process stops and the remaining datasets are left out of the index. The process should continue after logging the exception. If more than a certain number of exceptions occur in a row, the process should stop.

#1640: Setup publicdata.eu harvester for Serbian CKAN datasets

dread — Thu, 12 Jan 2012 12:13:45 GMT

Set-up publicdata.eu to harvest datasets at rs.ckan.net (Serbian community CKAN).

#1641: ckanext-archiver: Content-length header not reliable to check if resource has been modified

amercader — Thu, 12 Jan 2012 13:53:40 GMT

The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the "Content-Type" and "Content-Length" headers differ from the values stored to see if the resource needs to be updated [1].

The "Content-Length" header, although widely used, is not mandatory and some servers don't provide it, e.g.:

$ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG
HTTP/1.1 200 OK
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: text/xml
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly
X-AspNetMvc-Version: 3.0
content-disposition: attachment; filename=AFG_IATI_12012012.xml
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Jan 2012 12:36:43 GMT

Also worth noting that requests, the python library that uses ckanext-archiver, sets an "Accept-Encoding: gzip" header by default, which depending on the configuration of the remote web server, may prevent the "Content-Length" server from being sent, e.g.:

$ curl -H "Accept-Encoding: gzip" -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 12:12:46 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: application/xml
curl -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 11:56:23 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Content-Length: 2686720
Vary: Accept-Encoding
Content-Type: application/xml

All this can lead to some resources never getting updated, and of course the size property of the resource not being set.

As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it).

[1] https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72

#1654: [super] Update Publicdata.eu to the latest CKAN stable version

amercader — Mon, 16 Jan 2012 10:44:07 GMT

Tasks include:

#1813 Update ckanext-pdeu (4d)
#1814 Update harvesters (2d)
#1649 Verify ckanext-rdf works with latest CKAN (3d)
#1815 Reenable Sparql endpoint (?)
#1816 Update ckanext-apps (2-3d)

#1655: Setup issues on s025 (Publicdata.eu)

amercader — Mon, 16 Jan 2012 10:47:24 GMT

Time estimate: 2d

Fix logs (apache, ckan, harvest): rotate, set suitable levels
Fix harvesting jobs: supervisord for gather consumer, cron job
Fix backups

Also it may be worth setting up a test instance ( on s023 ?)

#1678: Make synchronous search the default behaviour

amercader — Tue, 17 Jan 2012 13:40:04 GMT

Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default. We could keep a config flag to deactivate it, but synchronous search should be the default behaviour.

#1701: Normalize character encoding for ckan search.

kindly — Tue, 24 Jan 2012 15:30:50 GMT

Make sure accented characters are normalized when indexed and when searched for.

#1702: Normalize character encoding for ckan search.

kindly — Tue, 24 Jan 2012 15:36:09 GMT

Make sure accented characters are normalized when indexed and when searched for.

#1726: Update harvesting model

amercader — Tue, 31 Jan 2012 10:52:07 GMT

We need:

"current" column in harvest_object
"harvest_source_id" column in harvest_object, foreign key to harvest_source
"title" column in harvest_source

The extension should check on startup if the db needs updating and run the necessary scripts.

#1727: Add "current" property to harvest objects

amercader — Tue, 31 Jan 2012 10:54:14 GMT

Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new "current" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes:

Model update (See #1726)
Update queries for getting harvest sources' datasets
Update ckanext-csw queries for getting the objects
Update inspire and ckan harvesters
Document new field

#1728: Update UKLP harvesters

amercader — Tue, 31 Jan 2012 11:00:10 GMT

Three small changes:

Set Resource type to None instead of 'Undefined'
Set extra 'UKLP' = True instead of 'INSPIRE' = True

These two changes will require manual update of the database.

Allow datasets to be updated if the old source is inactive, even if the metadata date hasn't changed:

    if last_harvested_object.metadata_modified_date is None \
       or last_harvested_object.metadata_modified_date < self.obj.metadata_modified_date \
       or self.force_import \
       or last_harvested_object.metadata_modified_date == self.obj.metadata_modified_date and last_harvested_object.source.active is False:

#1731: Publisher support for harvesting

amercader — Tue, 31 Jan 2012 11:05:19 GMT

Estimate: 4-5d

Only sysadmins can manage harvest sources right now, we need to allow publishers to handle their own sources.

We need to decide if there will be two different profiles on ckanext-harvest, the current one with only sysadmins allowed to manage the harvesting and a new one supporting publishers (ckan core can be configured to not use the publisher model)

Tasks include:

Auth checks on the different controller actions (and the command line interface?)
Modify the index page to just show sources the logged user has permissions on.

(Talk with Ross about the specific functions to use)

Create/edit form to show a dropdown with the user available publishers.

#1733: Delete (inactivate) harvest sources

amercader — Tue, 31 Jan 2012 11:11:27 GMT

Right now harvest sources can be inactivated (only via the cli), but we need to make sure:

Expose this on the web interface (only for sysadmins, see #1731)
Inactive sources are not shown in the web interface

Also for deleted packages:

a) Deleted packages get state="deleted" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for "Delete" for sysamdins

Done in #1425. Not sure if the button is only shown to sysadmins.

b) No undelete: If you reharvest a deleted package, it remains deleted

Publisher should create a new record with new identifier and same/different content as applicable.
If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated.
You won't get the state where a package is updated in its deleted state
If they change the ID and reharvest, a new record appears on DGU.

c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API.

#1734: [super] Updates on harvesting for UKLP

amercader — Tue, 31 Jan 2012 11:14:34 GMT

Some changes are needed in ckanext-harvest to support the new piece of work in the UKLP harvesters, mainly related to the deletion of existing sources.

#1726 Update harvesting model
#1727 Add "current" field to harvesting objects
#1728 Update UKLP harvesters
#1733 Delete (inactivate) harvest sources

At some point we also want to support a publisher model for the harvesting:

#1731 Publisher support for harvesting

#1813: Upadate ckanext-pdeu to work with CKAN 1.6

amercader — Mon, 20 Feb 2012 16:58:48 GMT

Tasks include:

Estimate 4d

Update requirements (git)
Search facets error
Error on dataset read (extras in context)
Url generator for rdf
Make templates work with latest theme
Use dicts in templates (isopen())

#1814: Check publicadata.eu harvesters

amercader — Mon, 20 Feb 2012 17:03:28 GMT

Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones.

#1816: Update ckanext-apps to work with CKAN 1.5.1 - 1.6

amercader — Mon, 20 Feb 2012 17:11:04 GMT

Estimate [2-3d]

If possible, it should work with both 1.5.1 and 1.6.

Needed for PDEU, also at some point for IATI

#2231: Base harvester broken on 1.6.1

amercader — Thu, 15 Mar 2012 12:44:59 GMT

On its current version, ckanext-harvest passed a custom schema to package_create and package_update to allow defining an id for new packages (the same as in remote ckan instances) [1]

Recent changes in the logic layer have made that the context object is no longer directly read from the context, but only get from plugins [2]. This makes the schema sent from ckanext-harvest to be ignored and the harvesting fails because the id field is not expected

ValidationError: {'Id': u'The input field id was not expected.'}

We need a way to either:

Be more flexible accepting schemas on the logic functions
Modify the harvester plugin to implement the necessary interface

[1] https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115

[2] https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38

#2232: Unicode Exception when rebuilding the search index

amercader — Thu, 15 Mar 2012 13:03:02 GMT

In some cases this exception is fired when building the search index. As explained in #1616 this makes the whole process stop.

  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 326, in wrapper
    return self._update(content, query)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 639, in _post
    return check_response_status(self.conn.getresponse())
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 1096, in check_response_status
    raise ex
SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
Traceback (most recent call last):
  File "/var/lib/ckan/pdeu/pyenv/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py", line 298, in command
    rebuild()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py", line 134, in rebuild
    {'id': pkg.id}
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 54, in insert_dict
    return self.update_dict(data)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 79, in update_dict
    self.index_package(pkg_dict)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 153, in index_package
    raise SearchIndexError(e)
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.

#2252: UKLP harvesters returning errors when importing due to schema changes

amercader — Wed, 21 Mar 2012 12:45:07 GMT

See trace below.

As in #2231, the schema passed with the context is no longer used on the logic layer (in this case the one used on the DGU form is used). This can be fixed allowing a schema to be passed with the context.

  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 139, in import_stage
    self.import_gemini_object(harvest_object.content)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 163, in import_gemini_object
    package = self.write_package_from_gemini_string(unicode_gemini_string)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 386, in write_package_from_gemin
    package = self._create_package_from_data(package_dict, package = package)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 486, in _create_package_from_dat
    package_dict = action_function(context, package_dict)
  File "/usr/lib/pymodules/python2.6/ckan/logic/action/update.py", line 169, in package_update
    data, errors = validate(data_dict, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 216, in validate
    converted_data, errors = _validate(flattened, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 289, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 179, in convert
    converter(key, converted_data, errors, context)
  File "/var/lib/ckan/std/pyenv/src/ckanext-dgu/ckanext/dgu/validators.py", line 81, in validate_license
    if data[('license_id',)]== '__extra__':
KeyError: ('access_constraints',)
2012-03-19 10:47:41,736 ERROR [ckanext.inspire.harvesters] Error importing Gemini document: ('access_constraints',)

#2263: IATI: Get "Record updated" field from the registry

amercader — Mon, 26 Mar 2012 12:49:05 GMT

Don't provide a field in the form or spreadsheet import for this, just use dataset.metadata_modified.

#2271: IATI: Some datasets not being indexed

amercader — Mon, 02 Apr 2012 14:04:29 GMT

When importing a CSV file for a certain publisher, records don't come up on the search index. After some investigation this has tunrn out to be caused by the records being deleted. We need to investigate if this was caused by the import or just made by the users.

#2332: Fixes for v1.7 release

rgrp — Sat, 28 Apr 2012 20:15:16 GMT

A place to list crucial fixes for v1.7 release:

Related extension:

All not complete now moved to #2347

~~Related Item has comments on index page (they should not show up here -- why is this happening!)~~
~~Related item should be to left of history (history could in fact move to RHS)~~
- What about losing icons since we duplicate for related item
Related item plus sign is very poorly aligned within box
- Add related items should be in drop down like for resources (?) to make it easy to add items
- Hover should show a title that gives some information about what related are (not at all clear)
Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

Search results:

#2331 - search should add not or
~~Search results should be similar across group pages and normal search results (suggest adopting group page style)~~
- Search results resource link should link to resource page rather than url for that resource
- Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?)

Data viewer:

~~Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07~~
- Ensure still using read-only (this seems to have been lost in upgrade)
Do not show failed preview with red-bar (esp when just empty datastore ...)

a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy b) DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

(? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state)
- Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...)
- Do #2322 ...
Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...)
Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
Data viewer embed url includes all the resource info

Dataset page:

Put home page link somewhere more prominent (e.g. the top of the sidebar)
Put author more prominent (at top right)
(? - new work) List CKAN owners at top right in sidebar
Make resources stand out more
Rename resources tab and heading to Data & Resources

Dataset edit and create:

Corrections to dataset creation form for v1.7 - #2317
Better auto-complete for groups on dataset edit/new page - #2319
- Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?)
Description / notes text area is not wide enough.
- Should also be larger on edit page
Problem with date instructions on add/edit resource: "Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d."
Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either
- A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar)
- Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like "checking url and gathering url info ...)

Resource view:

Do not show no description in no description (just show dataset info)

Theme:

~~Fix spacing above page title (i.e. dataset title etc)~~
~~Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer~~
~~Fix subnav width on resources pages (does not extend all the way to the right like it should)~~
~~Fix spacing of top menu and search box (do not look right)~~
Group pages:
- view page: image / logo is really poorly locate and sized, see e.g. http://datahub.io/group/wikimedia
- ~~view page: admin heading on RHS sidebar is wrongly indented~~

Miscellaneous:

Ensure that we out-of-the-box config for CKAN has (#2388):
- Only logged in users able to create
- Creation permissions have only creators able to update a dataset

Deployments (without deployment we cannot know these are working):

~~Social extension~~
~~Analytics~~

#2358: Finish off Release 1.7

amercader — Tue, 01 May 2012 15:43:50 GMT

Various stuff related with release 1.7:

Review patches sent to the release branch
Regularly update TDH with latest changes
write changelog, announce email, etc
...

#2359: Investigate pycsw integration

amercader — Tue, 01 May 2012 15:48:19 GMT

Tom Kralidis kindly told us on IRC [1] that the latest trunk of pycsw includes support for wsgi and different backends, which sounds like could be really useful for CKAN integration.

[1] http://irclogs.okfn.org/ckan/2012-04-25.txt 17:16

#2360: Finish INSPIRE support proposal

amercader — Tue, 01 May 2012 15:50:09 GMT

I've been writing a doc with different issues regarding INSPIRE support in CKAN. It is mostly done, except for some sections on CSW support. Depends on #2359

#2361: Install ckanext-googleanalytics and ckanext-stats on PDEU

amercader — Tue, 01 May 2012 15:51:16 GMT

ditto

#2388: Update "out of the box" settings for CKAN

amercader — Tue, 15 May 2012 15:21:09 GMT

Only logged in users able to create
Creation permissions have only creators able to update a dataset
Synchronous search is enabled (#1678)

#2389: More robust handling of DataExplorer errors

amercader — Tue, 15 May 2012 15:24:07 GMT

US-0a

Do not show failed preview with red-bar (esp when just empty datastore ...)

Data Viewer checks if datastore is empty nad falls back to ther options if emtpy
DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

Loading/showing error reports

#2399: Model cleanup in CKAN core has broken ckanext-spatial

seanh — Wed, 16 May 2012 13:04:30 GMT

root@lucid32:/usr/local/feature-2375-demo-theme/pyenv/src/ckan# sudo -u www-data /usr/local/feature-2375-demo-theme/pyenv/bin/paster --plugin=ckanext-spatial spatial initdb 4326 --config=/usr/local/feature-2375-demo-theme/pyenv/src/ckan/development.ini Traceback (most recent call last):

File "/usr/local/feature-2375-demo-theme/pyenv/bin/paster", line 9, in <module>

load_entry_point('PasteScript?==1.7.3', 'console_scripts', 'paster')()

File "/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run

invoke(command, command_name, options, args[1:])

File "/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke

exit_code = runner.run(args)

File "/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run

result = self.command()

File "/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/commands/spatial.py", line 38, in command

self._load_config()

File "/usr/local/feature-2375-demo-theme/pyenv/src/ckan/ckan/lib/cli.py", line 53, in _load_config

load_environment(conf.global_conf, conf.local_conf)

File "/usr/local/feature-2375-demo-theme/pyenv/src/ckan/ckan/config/environment.py", line 204, in load_environment

plugin.configure(config)

File "/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/plugin.py", line 60, in configure

setup_model()

File "/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/model.py", line 20, in setup

define_spatial_tables(srid)

File "/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/model.py", line 66, in define_spatial_tables

package_extent_table = Table('package_extent', metadata,

NameError?: global name 'metadata' is not defined

#2467: Fix stats extension frontend

amercader — Mon, 28 May 2012 13:53:50 GMT

After moving to Bootstrap the tables on the frontend look broken.

#2468: Finish off SlickGrid based Recline view

amercader — Mon, 28 May 2012 13:56:15 GMT

Please see these GitHub? issues:

https://github.com/okfn/recline/issues/130

https://github.com/okfn/recline/issues/131

#2472: clean up datastore controller to get rid of datastore_url enables checks

kindly — Tue, 29 May 2012 12:10:07 GMT

Datastore_url enabled checks are useless.

We should probably change this to datastore_url = 'active' and only update the flag to when a attempt at a post is made.

This will need to be removed from the form at some point.

#2505: Docs improvements for 1.7.1

amercader — Fri, 08 Jun 2012 14:42:13 GMT

There are some areas where the documentation could use some improvement and it would be good to have it available for 1.7.1.

Add an introduction to harvesting support
Add an introduction to CSW support
Better upgrade docs: Some users suggested improvements to the upgrade docs on the list [1]. Also we should mention how the upgrade would work for point point releases
Source install: see #2406

[1] http://lists.okfn.org/pipermail/ckan-dev/2012-May/002178.html

#2534: ckanext-spatial is broken with CKAN 1.8b

seanh — Fri, 15 Jun 2012 10:06:02 GMT

Looks like recent cleanups in CKAN broke an import in the extension:

https://gist.github.com/2935688

#2604: Add Requests as requirement for tests

amercader — Wed, 27 Jun 2012 09:43:49 GMT

Requests (http://python-requests.org) is really useful to deal with http requests and saves a lot of time.

I propose we include it as a requirement for the tests for the time being, and even for core if necessary at a later stage.

#2639: Update recline version for 1.8 release

amercader — Mon, 09 Jul 2012 14:56:49 GMT

New recline version has significant improvements like the SlickGrid? based grid view, it'd be good to have it on 1.8.

#2640: Update recline on the demo site

amercader — Mon, 09 Jul 2012 14:58:31 GMT

Depends on #2639

#3022: setup_template_variables method of IDatasetForm never called

amercader — Thu, 22 Nov 2012 16:45:43 GMT

On the package controller the package_type is not passed to the lookup function, so the setup_template_variables defined on the extensions is never called

#1142: [super] Major Overhaul and Extension of CKAN Documentation

rgrp — Wed, 18 May 2011 14:04:12 GMT

Child tickets:

#1041 Start Using the CKAN Wiki for Tutorial-style documentation
#1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927

CKAN 1-page overview (for enterprise and for data hackers)
Administrator's Guide (including install)
Extensions Guide
Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: http://wiki.ckan.net/Documentation_Plans

Minor Items

#1192: Convert CKAN Sphinx docs into admin/reference manual

annapowellsmith — Fri, 17 Jun 2011 15:29:53 GMT

As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install.
A Reference Manual which is the primary source of reference for CKAN software - this includes API docs.

The current chapters of the Sphinx docs should be moved as follows:

index.rst - copy some stuff from README.txt, keep short
README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst
[NEW] install.rst - new file on installation
[NEW] theming.rst - move over from wiki
api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
design.rst - REMOVE (can copy some over if you can be bothered to http://wiki.ckan.net/Vision (all philosophical stuff should go!))
loaders.rst: move to wiki.ckan.net/Using_Loaders
feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
importer.rst: remove (don't even copy)
forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) http://wiki.ckan.net/Customizing_Forms
forms-integration.rst: remove
model.rst: leave but move to reference (only for core devs)
load-testing.rst -> move to http://wiki.ckan.net/Load_Testing
distributed.rst -> remove
admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
deb.html -> go into reference (query james gardner on list about moving this to wiki?)

#1553: Upload handles non-logged in user badly

dread — Fri, 16 Dec 2011 17:26:45 GMT

When you install ckanext-store (Storage extension) then CKAN displays 'Upload a file' button in the dataset/resource edit tab.

If you are not logged in, then this feature doesn't work, but there is nothing to indicate this. The button should be disabled.

Furthermore, it lets you think you are starting the upload process, the spinner spins, it says 'Uploading...' and stays like that for a long while.

This is a problem CKAN 1.4.3(ish) - 1.5.1 (current version) when you install the current ckanext-storage.

#1718: Deprecated JS properties used

dread — Fri, 27 Jan 2012 12:04:02 GMT

I get this warning when viewing the home page:

event.layerX and event.layerY are broken and deprecated in WebKit. They will be removed from the engine in the near future.

CKAN: latest master Browser: Chrome Ubuntu

Minor, but worth checking?

#2225: Run jshint on our javascript and clean up as needed

rgrp — Sat, 10 Mar 2012 19:21:56 GMT

#2267: Add generator=ckan tag in head

rgrp — Thu, 29 Mar 2012 16:53:38 GMT

Add generator=ckan tag in head of templates to identify site as generated by CKAN

#2354: Front end performance improvements [super]

aron.carroll — Tue, 01 May 2012 15:10:42 GMT

Try and increase the page load times for the front end.

Now

Always include scripts at the bottom
Include CSS at the top
Load fonts with JS
Concatenate and minify JavaScript? files
Create a build script to load dependancies seperately
Enable Gzip compression on resources
Enable future cache headers on resources

Support serving assets from a CDN
Look into optimising the CSS - Remove large portions of unused CSS
Remove extreaneous libraries, can jQuery UI be replaced with bootstrap?

Tools

Webkit Inspector Audits
YSlow Firebug Plugin - http://yslow.org/
Speed Tracer - https://chrome.google.com/webstore/detail/ognampngfcbddbfemdapefohjiobgbdl

Continuous Integration

YSlow can be hooked up to Jenkins via Phantom JS

#2356: Localise JavaScript strings

aron.carroll — Tue, 01 May 2012 15:28:12 GMT

#2367: Style the analytics view counts on datasets

aron.carroll — Tue, 01 May 2012 17:19:49 GMT

Currently an eye and a number, needs to look a little nicer.

#2372: Create frontend style guide

aron.carroll — Thu, 03 May 2012 18:50:43 GMT

This should cover formatting and best practices for HTML, CSS and JavaScript?

Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch.

#2408: Only 4 'extras' can be added at a time

markw — Tue, 22 May 2012 10:47:19 GMT

The settings > extras page for a dataset allows extra fields to be added, but only four can be added a time in a single atomic update. It would be helpful to have an 'add another' button on the form that can be used any number of times.

#2418: Facets disappear once selected

aron.carroll — Wed, 23 May 2012 17:15:30 GMT

Currently facets in the new demo site disappear from the sidebar once they are selected.

http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa

Instead they should be marked with an "active" class and clicking the link should remove the filter in the same way as clicking the cross in a token below the search field works.

#2419: Add dataset form needs hooking up to action/controller

aron.carroll — Wed, 23 May 2012 17:16:41 GMT

Currently for some reason I'm not getting the errors/data dict in the package/new.html template.

#2421: demo-theme pills break in dataset search

Implementation:

http://s031.okserver.org:2375/user/edit

#2455: Create demo register page

aron.carroll — Mon, 28 May 2012 13:25:41 GMT

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998362/comments

Implementation:

http://s031.okserver.org:2375/en/user/register