CKAN: Ticket Query

#1516: Update SOLR schema after 1.5.1 release and set up multicore SOLR instances if needed

amercader — Mon, 05 Dec 2011 18:07:11 GMT

CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like:

http://<solr-server>/solr/schema-<version>

Different scenarios:

Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine):
- Update CKAN
- Update SOLR schema (symlink to suitable version in CKAN source)
- Rebuild search index

Single SOLR instance used by multiple CKAN sites.
- Update CKAN source
- Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source)
- Update solr_url on each of the CKAN sites with the suitable core
- Rebuild search index

These are the SOLR and CKAN instances that need to be updated (Please add any missing ones):

Same machine as the CKAN site:
- ~~test.ckan.net~~
- ~~iati.test.ckan.net~~
- data.gov.uk/ catalogue.data.gov.uk (confirm)

~~s046.okserver.org~~
- ~~iatiregistry.org~~

~~s052.okserver.org~~
- ~~datacatalogs.org~~

~~s004.okserver.org / eu4.okfn.org / solr.okfn.org~~
- thedatahub.org (=www.ckan.net)
- datagm.org.uk
- ~~publicdata.eu~~
- hri.fi (dev.fvh.fi?)
- it.ckan.net
- ie.ckan.net
- cz.ckan.net
- register.data.overheid.nl (nl.ckan.net?)
- no.ckan.net / datakilder.no
- br.ckan.net
- colorado.ckan.net
- at.ckan.net

TODO: which SOLR server are using these instances?

data.norge.no
nederland.ckan.net
lt.ckan.net
pl.ckan.net
datadotmontreal.ca/
ca.ckan.net / datadotgc.com

#1518: CKAN Upload fails if filename has spaces in it

markbrough — Mon, 05 Dec 2011 19:43:56 GMT

E.g. uploading a file with spaces in it: OECD Monthly Exchange Rates.zip

Gives a 404 Not Found response to the following file: http://test.ckan.org/api/storage/metadata/2011-12-05T193046/OECD%20Monthly%20Exchange%20Rates.zip

The upload wheel keeps spinning and the user is not informed that the upload has failed.

Analysis: turns out that google storage (possibly s3 but not checked) replaces with ' ' in keys with '+' on upload. This breaks things because we try and look up metadata about upload using the filename/key we put in but of course that does not exist because google has changed name.

Fix is trivial: replace ' ' in keys / filenames with '-'.

#1520: Disable name changing

dread — Tue, 06 Dec 2011 11:07:43 GMT

Because of #1514 we should just disable name changing, until #1514 is done.

#1524: New Release of Latest MI Reports

thejimmyg — Tue, 06 Dec 2011 11:58:08 GMT

#1532: Registration with OpenID has misleading error message

dread — Wed, 07 Dec 2011 16:46:32 GMT

The log-in page says "Login using Open ID" and gives instructions for signing up. YET this is only available to users who've already added openid to their account. If you have not done this and then sign-in via OpenID (which is successful from the OpenID end) then you are told "Login failed. Bad username or password." in a flash message.

Proposed solution (i don't know if this is possible):

When you log-in for the first time via OpenID, it doesn't actually log you in in CKAN. It just sends you to the 'Create User' page with the OpenID field pre-filled, and puts up a flash message "This OpenID account is not yet registered on thedatahub. Please complete your details.". This allows you to complete the registration and logs you in, and allows you to log-in directly with OpenID in the future.

Alternatives:

Just change the error message to be 'You need to register in CKAN first. Quote your OpenID in the registration form to use it in future.'
Remove OpenID altogether

#191: Searching by modification date

dread — Mon, 16 Nov 2009 16:09:38 GMT

Cost - 2 days

Search interface has new options to filter and sort the results by the date the package has been last modified in ckan. Search options are included in both Web UI and Search API.

The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package.

Example search parameters:

modification-range=5/4/09-	Exclude packages last modified earlier than 5/4/09
modification-range=5/4/09-5/12/09	Exclude packages last modified outside of 5/4/09-5/12/09
order_by=mod	Sort by metadata modification. Defaults to newest first.
order_by=mod-newest	Sort by metadata modification, newest first.
order_by=mod-oldest	Sort by metadata modification, oldest first.

#1462: Multi-Instance CKAN Solr Search Issues

thejimmyg — Tue, 15 Nov 2011 16:44:30 GMT

With the 1.5 package release, CKAN instances over-write each other's indexes. Adria has a fix.

I'd like to see a new CKAN 1.5.1 release with the Solr changes merged and the Solr site ID set up as part of the install ckan.site_id = dgu-os.okfn.org. I'd also like the ability to search on ID and to have the ID directly visible on the package page.

We need to decide how to do this upgrade without affecting existing CKAN solr indexes.

#1483: Tasks sometimes fail to update task_status table when trying to report a failure

johnglover — Wed, 23 Nov 2011 12:16:45 GMT

The error reports do not have a "value". The original task in the task status table is not made and it tries to make a new task, but can't.

Issue: why is the original task_status entry not made?

kindly: there is a chance of course that the task is quicker than the plugin to get to the point where it needs to update the table.

kindly: so we may have to find a way to update the task status before we run the task.

kindly: I think you can supply a task_id to celery. i.e we make our own guid as the task id. So we can use that one to populate the task status.

#1529: Display user name when logged in

dread — Wed, 07 Dec 2011 10:01:26 GMT

Currently when you log in it says "logged-in". Most sites show your user-name and this is helpful when you have more than one account or human using the computer.

#1536: Group deletion doesn't work

dread — Thu, 08 Dec 2011 12:19:26 GMT

In the Web interface and API, an administrator can set the 'state' field of a group to 'deleted' but this change is ignored in the save/submit and it doesn't get deleted.

#1546: Package metadata_created returns the wrong date

johnglover — Mon, 12 Dec 2011 16:28:47 GMT

metadata_created seems to be returning the date of the earliest revision (for all packages), rather than the earliest revision for a specific package.

#1548: Simplfy inheritance in functional tests

dread — Wed, 14 Dec 2011 10:48:07 GMT

It's hard to understand what's going on with the functional tests because there is so much going on in inherited classes. Would be better to be explicit in constructors, even if it involves cut & paste.

#1552: Be able to set password on the command line when creating users

dread — Fri, 16 Dec 2011 17:22:58 GMT

(instead of using the prompt) - needed for deployment

#1407: Stats extension not working

dread — Wed, 19 Oct 2011 08:56:45 GMT

Graphs don't show any more at http://thedatahub.org/stats

#1522: Add capacity to member table.

kindly — Tue, 06 Dec 2011 11:56:50 GMT

Need to add capacities to member tables.