Users should be able to log in using either their username, or their email address, both of which are unique.

Will require a change to UsernamePasswordAuthenticator? in ckan.lib.authenticator.py and possible a useful User.by_email in the user model if it doesn't already exist.

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on ​Storate Metadata API.

Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.

Just like we display download counts we should display view counts.

This is a reworking of the existing apps extension.

Initial proposal at ​http://wiki.ckan.org/Proposals#Apps_in_CKAN and ​http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN

Naming

What do we call this extension?

• related
  • RP best IMO but perhaps too close to separate relationships concept?
• relatedstuff
• links
• references

Proposal

"Related Stuff": Apps as in an application (website/service/tool) that uses this dataset (as in Apps and Ideas extension)

• This case seems a general example of wanting to link some external thing to a dataset (e.g. this policy paper refs this dataset, this application uses this dataset, this visualization is built on this dataset cf ​http://lists.okfn.org/pipermail/ckan-discuss/2011-August/001607.html)

Implementation

New table named Related with following structure

|| id (int) || type || title || description (markdown) || image_url || owner_id || url || created (timestamp)

• type = Idea | App | API | Visualization | Post | Paper | News Article
  • Suggest we make this a ckan.ini config option (comma separated ...?)
    • Do we want the possibility of different templates for different types of Related objects?

• image: ?? Depends where we store images. Simplest option would be to change to image_url and leave it to users to have already uploaded an image somewhere. If not we need to support image uploading and storage. See #1692 (add image attribute to datasets and groups) for more discussion, once implemented the URL here can be an internal url.

• owner_id = user_id or creating user (see authorization below)

Related2Dataset (note that related_id, dataset_id tuple should be unique). This allows for m2m connections. If a given related item is only with one dataset this could be simplified. May contain status so dataset owner can turn this on/off.

|| id || dataset_id || related_id || status

status should be used to allow for a dataset owner (for dataset_id) to de-activate the relationship between the dataset and the related.

Url

• /dataset/{dataset-name}/related/{related-item-id}/{related-item-title-stringified}
  • If a reference item could exist in its own right (and perhaps refer to multiple datasets then it should get its own url at e.g. /related/{id}
• /dataset/{dataset-name}/related/add => Modal dialog on related tab so we can use API to create them.

/dataset/{id}/related <- list

• use image_url for small icon etc, title description (shortened?)
• Click through to full related item (optional)
• dataset owner is shown show / hide button ... (or on /dataset/{id}/related/{id} )
• related owner sees an edit button / icon (pops up modal)

/dataset/{id}/related/{id} (optional)

/dataset/{id} will have a Related tab (with bubble with count).

• Drop down with Add Related -> Pop-up modal and save via API

(Not used: /related/add with dataset prefilled ... )

/related/{id}/edit

Authorization

Addition of related item be considered orthogonal to datasets (and hence with separate authorization i.e. i can add the info that site X uses dataset Y without needing permission to edit dataset Y).

Thus any logged in user could add a Related item. We set the owner of the related item to creating user and going forward only that user or a sysadmin can update or delete.

NB: we could have a system where datasets owners have to approve related items before they show up next to their dataset. This would add substantial complexity so I propose we leave out of phase 1.

Tasks and estimates {7.5d}

[x] Model + Migration for Related table. {0.75d}

[x] Controller for Related (or relevant sections in Package controller). {0.75d}

[x] Routing setup. {0.25d}

[x] Schema for related. {0.5d}

[x] Logic layer actions. {1.0d}

[x] Auth (default + publisher). {0.5d}

[x] Templates + Dataset changes (new tab etc). {1.0d}

[x] JS Application for adding Related objects in a modal. {1.0d}

[x] Testing. {0.75d}

[x] Dataset owner disabling of Related (via M2M table). {0.5d}

• Updated to allow author of related to delete as well

[x] Documentation. {0.25d}

Super is [1649]

We need to provide customisable mappings between CKAN models and RDF. This needs to be easily configurable for different clients and should take into account objects related to the model being mapped.

Current contenders for impl are:

1. ordf.org's Object Description Mapper (which uses OWL)

2. RDFAlchemy, although this looks like it may result in fixed mappings

3. A custom scheme with a simple json file

4. A template based approach using Genshi templates?

Approach is to get a mapping complete of the attributes directly in the package class, and then provide mappings for related objects (tags/resources etc).

We'll then need to be able to publish the RDF to a triplestore.

See ​http://ckan.okfnpad.org/rdfmapping for further analysis.

Most of the get_group/add_member style calls should be handled, or at least provided via logic layer.

Don't need to break existing code, but do need to provide access via logic layer

• create importer for geo tags
• add translations to database
• translate tags in WUI
• test
• deploy
• document how to add a geo coverage field in ​http://docs.ckan.org/en/latest/forms.html

User names and descriptions are not translated.

Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here: ​https://gist.github.com/2007985

Why?

• We already use quite a bit of boostrap
• We already plan to use bootstrap JS
• Bootstrap is excellent and pretty light

not all tests run and they take too long :(

The logic test did not have init. This caused lots of tests to fail because there were mock extensions that ran automatically in them. Fix plugin system so this can work.

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

• Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds

• Feed items should probably be cached somewhere

Suggested

• The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.

• For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference

• Caching is pretty crucial and should probably obey the ttl of the feed.

• (Recline) Better graphing (multiple graph types etc)
• (Recline) Bootstrapify
• Integrate

Document the ideas raised on 14/03 for CityData? and distribute to team for discussion

From Ira: Preview for google spreadsheets are not displaying correctly for me in Firefox v.10.0.02, fine in Chrome. ​http://i.imgur.com/KJaqz.png

Need google spreadsheet app to work with the new datastore. This should simplify it a lot.

Page views and Resources clicks need to be tracked.

User Stories

US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.

US1a Next to a resource or a dataset see how often it has been downloaded / viewed

US1b I want to see datasets or resources ranked by most downloaded or viewed

US1c See a trend graph for a dataset (and resources)

Adminstrative Dashboard (?)

• I want to see the traffic breakdown by country to my site ...
• Ditto for browser type, language, etc etc
• I want to see it graphed over time ...

Implementation Details

1. How do we store this data in CKAN?
2. How do we track (and store)?
3. How do we display

• Config option ckan.status.enabled = False (by default)

Storing Data

How does ckanext-googleanalytics do this? Current table:

package_id | count_recent | count_total

Move to a new stats_summary table

id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value

Do we store this data into the search (solr) so we can search by it?

Displaying Data

• Helper functions / dictize:
  • Helper function: h.stats_get(object_type, id, stats_type)
    • h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
  • Change to dictize
• Location in the default theme (do we show for example in search results too!)
• Support for ranking by most popular in search?

Tracking Data

• Our own solution (just write to site_tracking)
• Google analytics (plus extension for retrieving data) <- would need a refactor
• Piwiki

Own Solution

(For later: not as part of this ticket probably)

site_tracking table

id | url | timestamp | action (page_view, resource_download) |

• Make javascript to make request to ckan to store clicks and page views.
• Add middleware so these requests do not go through pylons and just store data quickly.

TD look at issues and then email ckan-dev

meeting notes from 21-3-2012 general ideas non specific

Problem:

• We want to remove the Transformer usage due to performance and its close tie to genshi
• Want a standard way for core code and extensions to put code into the templates where *templates request it* (templates control insertion rather than extensions ...)
• Stacking versus overwriting on entry points
  • Relationship to h.requires('jquery', version) - ​http://codex.wordpress.org/Function_Reference/wp_enqueue_script

Options:

• placeholders or helper methods
• {{user _info}} {{sub_menu}} is this equivalent to <span class="insert-user-info"></span>
  • [xyz arguments] -> h.load_snippet('xyz', ...)
• ${h.snipit()}

make core independent of TDH and make core as minimal app

• Customize text at bottom of forms
• Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...)

Check to see that all the strings are getting translated, i.e. that all user-visible strings are being passed to gettext. Fix any strings that are not getting translated.

Add generator=ckan tag in head of templates to identify site as generated by CKAN

• code review with D.R. for ckanext-ecportal
• merge and test all multilingual code
• fix server issues: routing, Git access to CKAN repository

Bugs to fix:

• rename groups to publishers
• update home page layout (recent updates along side publishers)
• don't add 'en' to main menu links if english
• resource edit page, fix missing images
• change dataset 'settings' to 'edit'
• translate 'published by' drop-down list
• fix JS on 'add a dataset page'
• prepopulate 'published_by' when clicking 'add a dataset' from group
• chosen broken (dataset edit) if in different language

This was part of #2251 but is now it's own ticket.

we need clarity on this. The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some? do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me.

From #2204

not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ...

### Comments

In the simplest form, right now all I need to store is store the url to the resource plus hashbang config for recline. That would be super easy ...

I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from TheDataHub? core, for example.)

• Switch to the latest minifed CSS + JS
• QA-sweep the site, especially dataset/edit

See ​http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

What needs doing, and how long will it take?

Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page.

• What should the sortBy controls/widget look like?
• Which fields should be exposed?

The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master.

[1] ​https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c

Needs re-testing after the ODS re-theme. ====

Simple image upload, text changes and css updates for a single instance. All done on a single configuration/admin page

See ​http://ckan.okfnpad.org/simple-themes for user stories

[x] Initial version will only take an image_url rather than allowing an upload.

[x] New page with form for title, tagline, image loader etc.

[x] Allow the user to paste in values for the CSS, do we really want a colour swatch?

[x] Generate local CSS override which should *somehow* override colors in the original CSS.

[x] Allow user to provide text for an about page.

User story #037 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

Finalised metadata model requires several changes to our existing model:

• new extra: alternative title
• new extra: identifier
• new extra: interop level
• new extra: version description
• new extra: dataset status
• dataset type: change to fixed vocab
• resource types: prefix with documentation/distribution/etc for clarity, add new resource types
• rename update frequency to accrual periodicity
• add new fixed vocab field: language coverage
• resources: missing issued date - add as new core field called 'created'

Also add functional tests for full metadata model.

Seanh and icmurray both touched on the html snippets in facets.html. So there's a merge issue. icmurray to merge changes in the multi-lingual branch prior to merging into master.

The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication.