CKAN: Ticket Query

#1521: [super] Groups UI and UX improvements in v1.6 from Cygri

shevski — Tue, 06 Dec 2011 11:12:41 GMT

Various UX and UI improvements to groups.

Etherpad: http://ckan.okfnpad.org/feature-1521-groups-ux

Improved groups

From Richard Cyganiak.

This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as http://ie.ckan.net/ could be turned into a simple group on the Data Hub without losing their separate identity.

Group homepages: Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.
Subdomains for groups: Enable subdomains such as ie.thedatahub.org as aliases for thedatahub.org/group/ireland
Search within the group: Have a big search box that searches just within the group on the group page, with a look like on http://thedatahub.org/dataset
~~Have an Add a dataset to this group link on the group page, perhaps in the sidebar.~~
Navigation within the group: Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from http://thedatahub.org/dataset
Group documentation: Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.
- ~~The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.~~
- Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines About, Frequently Asked Questions and How to get involved, we'd then get three links in the sidebar that point to datahub.org/group/ireland/about, datahub.org/group/ireland/frequently-asked-questions, datahub.org/group/ireland/how-to-get-involved
  - RP: WONTFIX this is being considered for future (generally useful) but cost/benefit here was low particularly as it wasn't clear what optimal implementation was.

Additional

~~Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)~~
Better group index page (sort by number of dataset, allow searching ...) - #1403
- Add search to groups: http://thedatahub.org/group
- Way to filter by activity, number of datasets
#1597 Tag search, filtered by group
- RP: not worth it in terms of benefit/cost IMO

Steps

~~"Add Dataset" button on action bar~~
- ~~Make it a dropdown? -> New Dataset / Existing Dataset~~
- ~~Redirect Add Existing -> Edit~~
- ~~Check permissions~~
~~Expandable "about this group" box~~
~~Implement simplest search-this-group box~~
- ~~turn the list of datasets into a dynamic list of search results~~
  - ~~pagination~~
- ~~Matching set of facets in the sidebar (matches 'real' search page)~~
  - ~~Deleteable, displayed facets~~

Nice-To-Have

Get rid of the old group_package_show action (confirm with team)
Tabbed "about this group" box (extract headers?)
~~Scrollable group members widget in the sidebar~~

#1629: permissions changed during upgrade to 1.5.1

dread — Tue, 10 Jan 2012 17:06:02 GMT

This was seen on datacatalogs. When we upgraded it 1.5 to 1.5.1 we saw some permissions being reset so that it could be spammed. Anonymous and logged in users were given anon_editor and editor permissions. I don't know what the previous config was. Permissions for sysadmins remain unaffected.

#1547: Add search methods (before and after) to the IPackageController interface

amercader — Tue, 13 Dec 2011 12:23:16 GMT

Time estimate: 1d

Add a two search method to the IPackageController interface so extensions can:

Modify the query being sent to Solr

def before_search(search_params):
    return search_params

Modify the results returned (after_search)

def after_search(search_results, search_params):
    return search_results

Please see this pad for implementation details:

http://ckan.okfnpad.org/search

#1622: Deploy QA on DGU UAT test server - 0.5d

johnglover — Mon, 09 Jan 2012 13:39:40 GMT

Update CKAN on DGU UAT to 1.5.1
Deploy Celery
Deploy QA extension

#1626: 'About CKAN' page update

dread — Tue, 10 Jan 2012 14:20:36 GMT

thedatahub.org/about contains info that is very general to the CKAN and really quite technical. The text should be changed to be both specific to thedatahub.org and provide the context in a non-technical way. It should be easy to customise the About page to be appropriate for say new-york.ckan.net - a bit of info about who runs it, plus the general stuff about CKAN powering it and it was written by OKF to further open data.

#1709: Simple_search fails with all packages

dread — Wed, 25 Jan 2012 13:32:46 GMT

The home page always shows the total number of packages as 0, because the query is '*:*' and SQL search doesn't know how to handle it.

Also, simple_search configuration is not documented.

Also, no tests for sql search.

#1401: Some links don't take account when CKAN is mounted at a non-root URL

flofokus — Fri, 14 Oct 2011 19:46:26 GMT

if you want your ckan to reside not at http://yourhost.org/ but rather e.g. at http://yourhost.org/ckan/ you can use this hint http://wiki.pylonshq.com/display/pylonsdocs/Configuration+Files and set ckan.site_url accordingly in your .ini. However, there are some instances where internal links are created without the required prefix.

#1541: Setup server for the DGU form-refactor.

icmurray — Thu, 08 Dec 2011 15:51:24 GMT

To enable us to show DGU work in progress, for feedback.

#1575: tag punctuation lost in ca.ckan.net import

dread — Mon, 19 Dec 2011 17:47:40 GMT

Last week I imported ca.ckan.net datasets into thedatahub.org, but the tags seem to have lost their dashes, underscores and dots.

#1597: Tag search - filter by group

dread — Thu, 22 Dec 2011 10:41:32 GMT

I want to browse tags, but filtered for a particular group. Currently our tag API doesn't allow for filtering by group.

This is important for improving groups as communities within a site #1521. It would be easy to do this by adding an option to filter by a group. BUT are there any other use cases that would warrant a more complete faceted tag search?

BTW I can currently draw a tag cloud for a group - I can get the top tags used in a group like this:

curl http://thedatahub.org/api/action/package_search -d '{"q":"groups:country-ca", "facet.field":"tags", "rows":"0"}'

but it only contains the top 20 tags.

#1624: Typo in dataset edit mode

pudo — Mon, 09 Jan 2012 14:59:57 GMT

Futher Information -> Further Information

#1627: favicon broken

dread — Tue, 10 Jan 2012 14:51:42 GMT

On thedatahub.org the favicon doesn't display. i.e. the CKAN logo should appear in the browser's tab.

Original ticket #48

#1641: ckanext-archiver: Content-length header not reliable to check if resource has been modified

amercader — Thu, 12 Jan 2012 13:53:40 GMT

The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the "Content-Type" and "Content-Length" headers differ from the values stored to see if the resource needs to be updated [1].

The "Content-Length" header, although widely used, is not mandatory and some servers don't provide it, e.g.:

$ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG
HTTP/1.1 200 OK
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: text/xml
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly
X-AspNetMvc-Version: 3.0
content-disposition: attachment; filename=AFG_IATI_12012012.xml
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Jan 2012 12:36:43 GMT

Also worth noting that requests, the python library that uses ckanext-archiver, sets an "Accept-Encoding: gzip" header by default, which depending on the configuration of the remote web server, may prevent the "Content-Length" server from being sent, e.g.:

$ curl -H "Accept-Encoding: gzip" -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 12:12:46 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: application/xml
curl -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 11:56:23 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Content-Length: 2686720
Vary: Accept-Encoding
Content-Type: application/xml

All this can lead to some resources never getting updated, and of course the size property of the resource not being set.

As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it).

[1] https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72

#1659: Cannot logout if CKAN mounted at non-root url

dread — Mon, 16 Jan 2012 12:18:22 GMT

If you set WSGIScriptAlias to mount CKAN at a URL other than / then you cannot logout without adjusting the OpenID logged_out_url to match in who.ini config. e.g.

[plugin:openid] ... logged_out_url = /sub/dir/user/logged_out

Note: all the other URLs in who.ini should not have the /sub/dir/ - it is just this one that doesn't take account of the mounting point.

The solution is to fix-up the repoze.who OpenID plugin to take account of the mounting point.

#1683: Dataset search results - last item out of order

dread — Thu, 19 Jan 2012 11:17:44 GMT

On each page of package search results, all the items are neatly sorted apart from the last item of the page. SOLR gets the sorting of the results incorrect.

This is a known issue: https://issues.apache.org/jira/browse/SOLR-1777 affecting SOLR 1.4 only (which comes with Ubuntu 10.04)

It is highlighted in CKAN test ckan/tests/functional/test_pagination.py:TestPaginationPackage of commit 39096ed54bda86d043521b08b2e14fc5e283a0ff which fails most of the time it is run (passes intermittently).

#1623: Dump to exclude deleted objects

dread — Mon, 09 Jan 2012 14:23:07 GMT

The database dump currently contains all Packages and their associated objects, even those that have been set to state=deleted. We should exclude these from the dump now.

Dump = paster db simple-dump-csv/json

reasoning

The dumps are designed for end-users to be able to run scripts on the mass of data. Since end-users don't see state=deleted packages then they shouldn't need them in the dump. In fact their presence in the dump probably confuses them.

Admins get the full database anyway in the backup pg_dump.

We only included them in the user dump because it was designed before use of state=deleted was established.

Time estimate: 2h

#1299: Activity streams table migration

kindly — Tue, 30 Aug 2011 09:30:35 GMT

Migrate tables for activity streams

#1475: Create documentation showing how to make an extension that use celery.

kindly — Sun, 20 Nov 2011 22:04:41 GMT

#1495: API call for getting a user's public activity stream

seanh — Mon, 28 Nov 2011 15:52:29 GMT

An API call that returns a user's public activity stream in JSON format.

#1511: Logic function for getting a user's public activity stream

seanh — Mon, 05 Dec 2011 14:10:07 GMT

Add a function to logic.action.get that returns a user's public activity stream as a list of dicts.

#1591: Upgrade CKAN 1.5 instances

dread — Wed, 21 Dec 2011 17:14:18 GMT

CKAN 1.5 installations need to be upgrade to 1.5.1.

Working methodically through the CKANs on http://datacatalogs.org/group/ckan :

At 1.5:

ckan-brazil http://br.ckan.net/ s057 - DONE
ckan_ireland http://ie.ckan.net/ s057 - DONE
cz-ckan-net http://cz.ckan.net/ s057 - DONE
datakilder-no http://no.ckan.net/ s057 - DONE
data_gov_uk http://catalogue.data.gov.uk/
datacatalogs-org http://datacatalogs.org/ s054 - DONE

Other versions:

berlin-open-data http://daten.berlin.de/ CKAN hidden
ckan-italia http://it.ckan.net/ 1.3.3b
ckan-poland http://pl.ckan.net/ Down
data-norge-no http://data.norge.no/ CKAN hidden
datadotgc-ca http://datadotgc.ca/ http://ca.ckan.net/ 1.4.2a
datagm http://www.datagm.org.uk/ 1.3.3
datagov-ru http://datagov.ru/ CKAN hidden
datospublicos-org http://datospublicos.org/ 1.4.2
iati-registry http://iatiregistry.org/ 1.5.1c
katalog-offene-daten-oesterreich http://offener.datenkatalog.at/ 1.3.3b
nosdonnees_fr http://www.nosdonnees.fr/ 1.3.1a
opengov-fi http://opengov.fi/ CKAN hidden
opengov_es http://opengov.es/ 1.2
publicdata-eu http://publicdata.eu/ 1.4.2a
the-data-hub http://thedatahub.org/ 1.5.1
helsinki_finland http://hri.fi/ CKAN hidden

Now deleted from datacatalogs.org:

ckan-belgium http://be.ckan.net/ Migrated
si-ckan-net http://si.ckan.net/ migrated
gr-ckan-net http://gr.ckan.net/ Migrated
hu-ckan-net http://hu.ckan.net/ Migrated
ckan-lithuania http://lt.ckan.net/ Migrated

#1608: Merge storage into Core

ross — Tue, 03 Jan 2012 09:14:04 GMT

As part of Storage Changes and http://ckan.okfnpad.org/newstorage it was decided to merge ckanext-storage back into core.

Other changes:

Remove /storage/upload page (we have frontend integration elsewhere now and we should also ensure that upload items relate to a resource or other domain object).
Provide a local path as well as remote path for form upload requests in API so that frontend can see url to CKAN path rather than cloud storage path ...

#1609: Celery task for ckanext-archiver to write to webstore.

ross — Tue, 03 Jan 2012 09:17:33 GMT

From super Storage changes - #1574 - and http://ckan.okfnpad.org/newstorage we determined that ckanext-archiver should have a celery task for grabbing local file uploads and writing to webstore

Analysis

When I upload a file to CKAN:

End up with file in permanent storage
IF file is ot type ... csv,xls,xlsx,sqlite,.sql
- End up with new db in webstore
  - Where? {username}/{resource-id}/...
    - If a single table: name it after the file name (appropriately slugified)
  - A resource *always* corresponds to a 'database' in webstore ...
  - In Data Explorer have "Sheets" tab ...
Resource url = /dataset/{x}/resource/{y}/link -> cached_url ...

#1625: Add activity stream events for new/changed users

seanh — Tue, 10 Jan 2012 09:29:54 GMT

This requires adding a logic function for emitting an activity stream event, and then editing the logic functions for creating or updating users and making them call the new emit event function. This same emit event function can later be used to emit activity stream events for other types of object as well.

#1631: Add activity stream events for new/changed groups

seanh — Wed, 11 Jan 2012 17:47:00 GMT

#1691: paster user create command takes password on command line

rgrp — Fri, 20 Jan 2012 16:35:14 GMT

Needed to support automated deployment more easily.

Est: 15m

#1613: Post-dataset-edit URL has #section

dread — Wed, 04 Jan 2012 13:59:15 GMT

Viewing a dataset, hit 'Edit'
Click on the "Basic Information" tab (note: URL has suffix #section-basic-information
Click 'Save'
URL still has suffix #section-basic-information

Affects 1.5, 1.5.1, 1.5.2a

#1638: Don't use JsonType in activity streams

seanh — Wed, 11 Jan 2012 18:02:15 GMT

Dump and load JSON explicitly instead.

#1646: Resource navigator options display spuriously

dread — Fri, 13 Jan 2012 11:05:00 GMT

When viewing a dataset, the "Resources" navigation button contained the Resource titles on the Resource navigator button, instead of in a drop-down mouse-hover menu.

http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f

Chrome/Linux?