CKAN: Ticket Query

#1593: [super] Create feeds from search results

amercader — Wed, 21 Dec 2011 17:38:35 GMT

Recent changes (see #191 and #1498) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy.

Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest.

Apart from the general feed with changes on the whole instance, other different "pre-set" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags:

http://<ckan_server>/feed/group/{id}.{format:rss|atom}
http://<ckan_server>/feed/tag/{id}.{format:rss|atom}

These could be shown on the group/tag page, both on the <head> section, so browsers can autodiscover them, and on the UI, with a message like "Subscribe to this group/tag".

Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on

http://<ckan_server>/feed/custom.{format:rss|atom}?res_format=api%2Fsparql
http://<ckan_server>/feed/custom.{format:rss|atom}?q=tags:animal*

On this case a message "Subscribe to these search results" would appear.

Note that a lot of work on this front needs to be done for the IATI registry project (see #1590)

Other tickets related:

#1592: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now.

#2251: Internal analytics for ckan.

kindly — Wed, 21 Mar 2012 11:57:09 GMT

Page views and Resources clicks need to be tracked.

User Stories

US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.

US1a Next to a resource or a dataset see how often it has been downloaded / viewed

US1b I want to see datasets or resources ranked by most downloaded or viewed

US1c See a trend graph for a dataset (and resources)

Adminstrative Dashboard (?)

I want to see the traffic breakdown by country to my site ...
Ditto for browser type, language, etc etc
I want to see it graphed over time ...

Implementation Details

How do we store this data in CKAN?
How do we track (and store)?
How do we display

Config option ckan.status.enabled = False (by default)

Storing Data

How does ckanext-googleanalytics do this? Current table:

package_id | count_recent | count_total

Move to a new stats_summary table

id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value

Do we store this data into the search (solr) so we can search by it?

Displaying Data

Helper functions / dictize:
- Helper function: h.stats_get(object_type, id, stats_type)
  - h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
- Change to dictize
Location in the default theme (do we show for example in search results too!)
Support for ranking by most popular in search?

Tracking Data

Our own solution (just write to site_tracking)
Google analytics (plus extension for retrieving data) <- would need a refactor
Piwiki

Own Solution

(For later: not as part of this ticket probably)

site_tracking table

id | url | timestamp | action (page_view, resource_download) |

Make javascript to make request to ckan to store clicks and page views.
Add middleware so these requests do not go through pylons and just store data quickly.

#2252: UKLP harvesters returning errors when importing due to schema changes

amercader — Wed, 21 Mar 2012 12:45:07 GMT

See trace below.

As in #2231, the schema passed with the context is no longer used on the logic layer (in this case the one used on the DGU form is used). This can be fixed allowing a schema to be passed with the context.

  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 139, in import_stage
    self.import_gemini_object(harvest_object.content)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 163, in import_gemini_object
    package = self.write_package_from_gemini_string(unicode_gemini_string)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 386, in write_package_from_gemin
    package = self._create_package_from_data(package_dict, package = package)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 486, in _create_package_from_dat
    package_dict = action_function(context, package_dict)
  File "/usr/lib/pymodules/python2.6/ckan/logic/action/update.py", line 169, in package_update
    data, errors = validate(data_dict, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 216, in validate
    converted_data, errors = _validate(flattened, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 289, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 179, in convert
    converter(key, converted_data, errors, context)
  File "/var/lib/ckan/std/pyenv/src/ckanext-dgu/ckanext/dgu/validators.py", line 81, in validate_license
    if data[('license_id',)]== '__extra__':
KeyError: ('access_constraints',)
2012-03-19 10:47:41,736 ERROR [ckanext.inspire.harvesters] Error importing Gemini document: ('access_constraints',)

#2260: Update ecportal theme to match latest changes to tenforce theme

johnglover — Mon, 26 Mar 2012 09:31:59 GMT

#2264: Do an audit of string translations in CKAN core and ckanext-ecportal

seanh — Tue, 27 Mar 2012 16:08:32 GMT

Check to see that all the strings are getting translated, i.e. that all user-visible strings are being passed to gettext. Fix any strings that are not getting translated.

#2270: DataExplorer overview

johnglover — Mon, 02 Apr 2012 13:45:39 GMT

get up to speed with DataExplorer? code (and dependencies like Backbone)

#2271: IATI: Some datasets not being indexed

amercader — Mon, 02 Apr 2012 14:04:29 GMT

When importing a CSV file for a certain publisher, records don't come up on the search index. After some investigation this has tunrn out to be caused by the records being deleted. We need to investigate if this was caused by the import or just made by the users.

#2272: Setup jenkins on the new dev server

seanh — Mon, 02 Apr 2012 14:04:40 GMT

#2275: Implement group logos

seanh — Mon, 02 Apr 2012 14:14:36 GMT

Groups should be able to specify a URL to an image file for a group logo, the image will be shown on the group page.

~~Actually uploading an image file (rather than giving a URL) and cropping and resizing the file will be done later, for now just give a URL.~~ (later ticket)

~~Further customisation of group pages (CSS, templates) also for later.~~ (not sure if we really want this, but in a later ticket if so)

#2276: Upgrade Slovakia CKAN instance

seanh — Mon, 02 Apr 2012 14:20:35 GMT

#2278: DataStore enabled is disabled by default but is needed for DataStorer to run

rgrp — Tue, 03 Apr 2012 08:17:28 GMT

Major UX gotcha reported by several people is that DataStore? is disabled by default and must be enabled for DataStorer? to run and import data.

Suggest switch to enable datastore by default for file and file.upload types (disable for API). People can always disable if they want.

#2281: Pagination styling converted to Bootstrap

dread — Mon, 09 Apr 2012 10:31:46 GMT

Make the search results pagination styled by Bootstrap.

Needs putting into a <ul> structure. Current page and '..' need become links that are styled disabled.

Wanted by DGU.

#2286: Upgrade to boostrap v2.0.2

zephod — Thu, 12 Apr 2012 17:30:18 GMT

I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from TheDataHub? core, for example.)

Switch to the latest minifed CSS + JS
QA-sweep the site, especially dataset/edit

#2295: Migrate CKAN packaging and apt.ckan.org from dgu-buildbot

icmurray — Mon, 16 Apr 2012 09:02:56 GMT

Create Jenkins task for packaging CKAN
... and publishing to s3://apt.ckan.org

#2296: Initial analysis of moderated edits

icmurray — Mon, 16 Apr 2012 09:04:37 GMT

What needs doing, and how long will it take?

#2297: Initial analysis of follow support

icmurray — Mon, 16 Apr 2012 09:07:03 GMT