CKAN: Ticket #2251: Internal analytics for ckan.

description changed

kindly — Wed, 21 Mar 2012 11:57:56 GMT

description modified (diff)

rgrp — Wed, 21 Mar 2012 12:07:30 GMT

Big +1: everyone wants to know page views. Would like detail of how this goes into interface. Downloads already being tracked. Also isn't this just an extension to ckanext-googleanalytics.

status changed

toby — Wed, 21 Mar 2012 12:19:50 GMT

status changed from new to accepted

not really part of ckanext-ga as we need to sit first in the middleware for speed reasons. I'm looking at recording the details and will do a summary page like ga extension.

As far as other uses like showing popular resources etc then we need to decide where the data will live - on the resource or separate.

Downloads tracked? where is this done in the code/data stored

toby — Wed, 21 Mar 2012 14:41:14 GMT

notes from meeting 21-3-2012

US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded. US1a Next to a resource or a dataset see how often it has been downloaded / viewed US1b I want to see datasets or resources ranked by most downloaded or viewed US1c See a trend graph for a dataset (and resources) Adminstrative Dashboard (?)

I want to see the traffic breakdown by country to my site ...
Ditto for browser type, language, etc etc
I want to see it graphed over time ...

Implementation Details

How do we store this data in CKAN?
How do we track (and store)?
How do we display

Config option ckan.status.enabled = False (by default)

Storing Data

Displaying Data

Helper functions / dictize:
- Helper function: h.stats_get(object_type, id, stats_type)
  - h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
- Change to dictize
Location in the default theme (do we show for example in search results too!)
Support for ranking by most popular in search?

Tracking Data

Our own solution (just write to site_tracking)
Google analytics (plus extension for retrieving data) <- would need a refactor
Piwiki

Own Solution

site_tracking table id | url | timestamp | action (page_view, resource_download) |

Make javascript to make request to ckan to store clicks and page views.
Add middleware so these requests do not go through pylons and just store data quickly.

toby — Wed, 21 Mar 2012 14:42:27 GMT

TODO

look at

How do we store this data in CKAN?

How do we display * Config option ckan.status.enabled = False (by default)

description changed

rgrp — Fri, 23 Mar 2012 17:35:23 GMT

description modified (diff)

Update description in great detail.

description changed

rgrp — Mon, 26 Mar 2012 08:55:39 GMT

description modified (diff)

description changed

rgrp — Mon, 26 Mar 2012 08:56:42 GMT

description modified (diff)

priority changed

toby — Mon, 26 Mar 2012 15:48:46 GMT

priority changed from awaiting triage to major

we want

resource show downloaded/viewed on resources

show on the dataset total/recent

description changed

toby — Mon, 26 Mar 2012 16:14:23 GMT

description modified (diff)

keywords, milestone changed

toby — Mon, 02 Apr 2012 15:51:48 GMT

keywords 6d added; 4d removed
milestone changed from ckan-sprint-2012-04-02 to current-ckan-sprint-2012-04-16

toby — Wed, 04 Apr 2012 10:32:52 GMT

we have various options on tracking unique users seems the sane approach.

unique daily view - needed for nice graphing

unique total views

total unique daily views - higher numbers and feels more right - easier to calculate if original data is lost/or archived

for now will collect all and then decide which to use for display as gives us flexibility if needed

display

why is group listing of datasets inconsistent with main search (TDH at least)? should we make these consistent?

group listing allows downloading of resource without tracking (this should be fixed)

if we make listings uniform do we want the download links or not?

I don't like the term download when it is just a link (maybe that's just me - what would be better?)

toby — Thu, 05 Apr 2012 16:48:58 GMT

most work completed: outstanding issues

add a prune tracking_raw data to cli

add indexes to tracking_raw

add summary page for new data format

add api counter

1) do we record the url or action etc (action seems more sensible) 2) add in call or in action?

toby — Tue, 10 Apr 2012 16:19:32 GMT

api counter moved to ticket #2282 as needs better specification

toby — Wed, 11 Apr 2012 11:49:59 GMT

For meeting today with Adria

How to add tracking to solr index. Can the data live in tracking_summary? Do we need any extra indexes on table to help indexing?

How will we add order by popularity to search?

Can we add resources too?

Will search be best way to find most popular packages/resources or should we get that pages data from tracking_summary?

status changed; resolution set

toby — Thu, 03 May 2012 11:55:55 GMT

status changed from accepted to closed
resolution set to fixed