CKAN: Ticket Query

#1129: CREP0002: Moderated Edits

kindly — Sun, 08 May 2011 10:44:58 GMT

Proposer: David Raznick

Abstract.

We are trying to achieve these goals.

To get people involved with making edits to CKAN metadata.
To have an ownership model as to who can moderate and validate these changes
To not put too huge a burden on these owners.

In order to achieve this, a feature which lets anyone edit a package but only let the moderator/owner accept it. The moderator should be able to look at a list of changes and accept the ones that

This cep is not about 'if' we need such a feature, it is about 'how' we go about implementing it. Another cep may needed for the 'if' case.

The Problem

We need the following to be possible.

Storing revision of objects that are not the current active one.
A way of the user viewing past revisions.
Accessing not only the history of a particular object but also of related objects at that time. i.e If a resource related to a package changes we need a way to see this when looking at the package.
A robust way of doing this in the face of database schema changes.
Make sure database queries are quick.

Solutions.

Store the whole dictization of the package and all its related objects every time you change anything in its dictized representation and only save to the database proper if accepted.

Pros

Easy to implement, we already have a preview which makes the dictized form of a package without actually saving it. This will just need to be persisted in some way.
Fast retrieval.
Potential to store a branching revision tree of changes.

Cons

No easy way to remake the dictized packages historically or if there is an there a change in the way we represent packages, i.e schema changes.
Will only work for the particular objects we decide to store these changes for.
Stores a lot of repeated information

Write specialized queries for every read of the database looking only at the revision tables.

This method requires there to be a change in the way we use VDM, so that we manage statefulness ourselves. We will need to add other states such as 'waiting for approval'.

Pros

No specialized storage required
Only need to change queries when schema changes
Can be made to work easily for other objects

Cons

Slower query time on read, as even looking at the last active package will need to do a fairly complicated query.

Implementation details.

A new table with columns id, user, package_id, timestamp, revision_id, parent_id, dictized_package. revision_id should be null unless it is actually persisted to the database. parent_id is the id that this package_dict was changed from.

We could store only the diffs of the dictized_package as long as we assure that everything inside the json is stably sorted, this will make getting the historical data out slower.

Getting out the history of the dictized packages is an intensive task, as it will require replaying the whole history of all the changes and creating the dict for each change. This re-caching will need to be redone for every change we make to dictized representation of a package.

Every normal packages read needs to look at the revision table to see the last accepted change in the dictized representation of the package. We also need to way to get what the dictized representation of the package was like at any point of its revision history. This querying is non-trivial in sql.

Participants

David Raznick to do it.

Progress.

Decided to go with option 2. However we will change the revisioning system to be like the schema attached. This gets rid of difficult querying problems caused by querying the revision tables by adding an end date, meaning you can do range queries.

The better and more normalized version of a revisioning system is outlined https://docs.google.com/drawings/d/1Y7nMgVsrs081Pame2RdbZHlCAlV33ddTZ8VAsab1j-0/edit?hl=en_GB&authkey=CJfd8vsB. We will be a step closer to that, with this change, but we will keep the current vdm more or less, intact.

#559: Fix package lookup on group edit form

pudo — Fri, 03 Sep 2010 19:39:36 GMT

Lookup is needed to add a package but won't show up at some times.

Originally reported by Anja Jentzsch, re questions.

#912: Rethink result row representation in dcat-tools/rdfsolr

pudo — Mon, 17 Jan 2011 12:11:39 GMT

Should license go in the bottom line?
Formats should be styled as in CKAN

#1053: Deletion in Model API

dread — Tue, 22 Mar 2011 10:33:59 GMT

Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted.

There is no way to delete objects at the moment - changes to 'state' are ignored in the API.

Do we need an alternative way to purge objects in the API?

#1140: Adding the package to the group is not search indexed

dread — Wed, 18 May 2011 11:31:30 GMT

To reproduce:

paster db init
paster create-test-data
In Web UI: create new group 'tilo', which includes package 'annakarenina'
curl http://localhost:5000/api/search/package?groups=tilo results in 0 results (WRONG)
paster search-index rebuild
curl http://localhost:5000/api/search/package?groups=tilo results in 1 result

#1353: No UI to remove resources

nickstenning — Thu, 22 Sep 2011 22:16:05 GMT

I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org.

#1431: Captcha field - foreign chars cause exception

dread — Mon, 31 Oct 2011 16:34:09 GMT

During registering a user, the user inputs foreign chars into the captcha field.

URL: http://thedatahub.org/user/register
...
Module ckan.lib.captcha:22 in check_recaptcha
<<                                     remoteip=client_ip_address,
                                          challenge=recaptcha_challenge_field,
                                          response=recaptcha_response_field))
           f = urllib2.urlopen(recaptcha_server_name, params)
           data = f.read()
>>  response=recaptcha_response_field))
Module urllib:1267 in urlencode
<<          for k, v in query:
                   k = quote_plus(str(k))
                   v = quote_plus(str(v))
                   l.append(k + '=' + v)
           else:
>>  v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 0: ordinal not in range(128)

#103: View a package at a given revision

rgrp — Wed, 26 Aug 2009 12:31:37 GMT

As a user I want to view a package at a given revision:

When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
package history page should provide links to these pages

Cost: 2h

#358: Resources in REST API

dread — Fri, 25 Jun 2010 16:28:02 GMT

(spun out of ticket:336)

Resource added to model API at:

api/rest/resource

Example model request

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028

returns:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors",
   "format": "CSV",
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "",
   "position": 2
 }]

Authorization

Have it generic (ie. not per resource) and use an action/role on system
Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
Introduce Resource in authorization system (requires migration)

Mixed model

Create / Edit:
if resource associated to package:
    check_permissions(package, update)
else:
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)

#764: Read-only CKAN Web UI

dread — Tue, 26 Oct 2010 11:13:39 GMT

Whilse using CKAN web interface, you are not tempted to edit stuff:

You know at all times this CKAN is read-only
All editing facilities are still seen but greyed-out with an indication why it is.

#810: Move "add packages" field up in group form

pudo — Wed, 17 Nov 2010 11:11:27 GMT

Subticket of forms super ticket #961

#852: [super] Dataset upload and archiving

wwaites — Tue, 07 Dec 2010 11:56:33 GMT

3 (4) use cases: (With and without access to auth keys.)

upload form on ckan (by default local system has auth key for storage)
datapkg upload (client) (by default without key) (plumbing done)
background job archiving downloads (by queue client)

Most important are:

Upload form on ckan.net (with access to keys)
API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

#826 - Resource 'extra' fields
#358 - Resources in REST API
#235 - Resource format normalization and detection

User upload via Web UI

#877 - File upload in WUI
#878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

#853 - plumbing for credential-less rest upload
#879 - Storage auth API

Also related:

#405 - Retrieval options for package resources

More Info

Upload/storage discussion: http://ckan.okfnpad.org/storage
- For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/

#910: Refactor CKANrdf to support extras

pudo — Mon, 17 Jan 2011 12:07:06 GMT

Extra fields should be represented in some manner.

#913: Handle licenses in dcat-tools

pudo — Mon, 17 Jan 2011 12:13:27 GMT

Create a list of all valid predicates where licenses are stored.
Create a license normalization registry in GSpread
Fill spreadsheet from OKD/OSI lists.

#941: Submit apps or ideas for apps related to datasets (extension)

rgrp — Sun, 30 Jan 2011 15:58:19 GMT

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

#954: [super] API version 3

rgrp — Fri, 04 Feb 2011 09:31:39 GMT

Child tickets:

#1107 Move package autocomplete from package controller and move to API
#1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

#958: ui for new kind field

kindly — Fri, 04 Feb 2011 12:34:56 GMT

Ui for related new field in resource (#957)

#961: [super] Refactoring of forms, validation and model synchronization

rgrp — Mon, 07 Feb 2011 09:10:49 GMT

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

Generating (or just filling) a form template with 'form data' (and errors)
Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

#926 - Pick a simpler form framework
#1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
- #1079 Refactor API to use new logic layer and dictization
- #1078 Refactor WUI controllers and forms to use logic layer
- cf existing dumper and importer code
- This will fix #662
[not ticketed yet] - validation layer (should work on serialized objects?)
#662 - Can't put entity that is returned by posting to package register (Defect)
#972 - Merge 'extras' into main package dict rather than having separate key
#1035 - Form impressions are given IDs
#810 - Move "add packages" field up in group form (easier to do this once forms are done)

#972: Merge 'extras' into main package dict rather than having separate key

rgrp — Thu, 10 Feb 2011 10:04:42 GMT

Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.

Issues

Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961

#1070: Plan a new domain model and layer architecture for CKAN

rgrp — Mon, 04 Apr 2011 09:55:40 GMT

See http://wiki.ckan.net/Domain_Model especially section on v2.

New domain model is planned but not yet finally agreed.
Layer architecture is complete and implemented

#1074: Refactor authz web user interface to have common code and templating

rgrp — Fri, 08 Apr 2011 14:06:26 GMT

Currently repeat the same template and code across Package Authz, Group Authz, and Authz Group authz.

Having now implemented a new, cleaner setup in ckanext-admin we should port this back into core.

Common template code (checkbox template)
Logic code (or just common code) for wiring into authz system
Look for all places thoroughout the system where usernames, authzgroups or groups need to be typed into boxes, and make sure that they auto-complete appropriately.

Will also deliver a significant improvement in the form of ajax user lookup.

#1078: Refactors WUI controllers and forms to use logic layer

rgrp — Mon, 11 Apr 2011 08:09:07 GMT

Deserialize forms to new dict format.
Replace controllers/forms to use dictization.

#1094: [super] Refactor the Auth System

thejimmyg — Mon, 18 Apr 2011 09:12:53 GMT

Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787.

Two man reasons for the changes are:

We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer.

Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able.

The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package).

The first thing the is_authorized method is a hook to a plugin

which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!)

Reason: authorization can be completely delegated to another system (or partially)

is_authorized method currently takes (username, action, object)

but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model.

Rename these two classes to better reflect what they are
- AuthorizationGroup? -> UserGroup?
- Group -> PackageGroup?

Rename the Editor role to PriveledgeUser? since Editors sometimes can't edit.

Although this sounds a bit radical we already have auth extensions.

Read-only CKAN Web UI

(Additional requirement from #764)

Whilse using CKAN web interface, you are not tempted to edit stuff:

You know at all times this CKAN is read-only
All editing facilities are still seen but greyed-out with an indication why it is.

#1142: [super] Major Overhaul and Extension of CKAN Documentation

rgrp — Wed, 18 May 2011 14:04:12 GMT

Child tickets:

#1041 Start Using the CKAN Wiki for Tutorial-style documentation
#1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927

CKAN 1-page overview (for enterprise and for data hackers)
Administrator's Guide (including install)
Extensions Guide
Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: http://wiki.ckan.net/Documentation_Plans

Minor Items

#1190: [super] CREP 0004 Data API and Data Processing System

rgrp — Thu, 16 Jun 2011 12:33:31 GMT

For some time (e.g. 1y+!) we have known that we want to integrate some kind of datastore / data processing system with CKAN. We've had a CREP in progress on this for some months (may copy that here at some point):

http://wiki.ckan.org/CEP0004

We can distinguish 3 modules that are needed:

"Webstore": A datastore with dataapi - #1208

Suggestion is this would be sqlite based with a simple sql based API. http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query}

Automated conversion of suitable resources into datastore upon resource creation so that e.g. they are accessible via the API. #1398

A data processing system which utilizes this datastore. One could

get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) - #1432

More info

Overview diagram: https://docs.google.com/drawings/d/1XK7dcpFXNlMzVFgLPYPZUXOsFRPzgCEvS-w7FcvydEQ/edit?hl=en_GB

#1211: [super] Drupal integration

kindly — Wed, 06 Jul 2011 13:20:42 GMT

Sub-tasks / tickets:

#1335 Action API - access to any logic layer function
Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
Add package_create_validate & package_update_validate logic functions
Allow extensions to provide logic layer functions (IActions)

#1249: Exclude script tag from extraction for i18n

rgrp — Fri, 29 Jul 2011 16:55:36 GMT

Currently have this script section put in for i18n. It shouldn't be.

<script type="text/javascript">
//<![CDATA[
(function($){
    $.fn.ajaxCreateSlug = function(name, url) {
        var title = this;
        var updater = {
            init: function(title, name) {
                // Add a new element where the validity of the package name can be displayed
                this.name_field = name;
                this.title_field = title;
                this.name_field.parent().append('<div id="package_name_valid_msg"></div>');
                this.title_field.blur(this.title_change_handler())
                this.title_field.keyup(this.title_change_handler())
                this.name_field.keyup(this.name_change_handler());
                this.name_field.blur(this.name_blur_handler());
                this.url = url;
            },
....

#1250: Search results should be sorted by score rather than alphabetical

rgrp — Fri, 29 Jul 2011 18:39:12 GMT

At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases.

Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard).

Options:

Default this in solr (no need to touch code) but fragile and affects everything ...
Do it in code and default to score
Do it in code and have alphabetical (on name or title?) when no criteria otherwise score

Aside: may also wish to support search in query api but that is for later!

#1252: Deprecate ckan.net in favour of thedatahub.org

rgrp — Fri, 29 Jul 2011 18:48:59 GMT

Deprecate ckan.net in favour of thedatahub.org

Announce
Create thedatahub.org and have it run same data as ckan.net
Announce closing down of ckan.net with 302
Implement 302 permanent

Also redirect wiki.ckan.net to wiki.ckan.org

#1282: JS and WUI tidy up

rgrp — Tue, 23 Aug 2011 09:06:29 GMT

Consolidate js. Remove cruft.

Move stuff to vendor
- Have all vendor libs local (?)
- jquery, jquery-ui etc
Consolidate autocomplete to use jquery-ui
- package autocomplete (groups)
- tag autocomplete (package)
- user autocomplete (authz group create)
  - turns out that authz group edit had autocomplete disabled so refactoring meant enabling ...
Move forms.css into main
Remove tag cloud from front page (can be example going forward)
Remove bookmarklet (since in wiki at: http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F)
Move name slug generation into standard js and clean up

Possible:

JS unit tests for the given pages ...
Option to use external site to serve js and css (e.g. CDN)

#1294: [super] Package creation and editing UX improvements

rgrp — Sat, 27 Aug 2011 00:43:31 GMT

Largely an integration of work from ckanjs

#1295 Simplify package create form
Improvements to edit form
- Split edit form into sections
- Remove preview from edit form (introduce inline preview on description/notes field)
#1296 Improved resource adding/editing on dataset/page page
#878 File upload integrated into resource creation
#1297 In-place editing of notes field for existing packages

#1354: Pass the context object to plugins implementing the IPackageController and IGroupController

amercader — Sun, 25 Sep 2011 13:01:20 GMT

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

#1361: Simple search support

rgrp — Mon, 26 Sep 2011 23:04:36 GMT

Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install).

Propose:

ckan.simple_search config option
If set:
- query via simple query to database backend
- do no specialized indexing

Extras

Remove

TagSearchQuery? from lib/search (just do a search directly)?
#1360: filter by downloadable and filter by open search options

Possible Extras

(Probably future improvements)

Re-introduce full-text search indexing where supported in e.g. postgres and use for querying

#1410: Add Gravatars to user profiles

zephod — Fri, 21 Oct 2011 12:39:34 GMT

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

#1245: Reorganise ckan.org pages

rgrp — Thu, 28 Jul 2011 14:01:48 GMT

Here is Rufus' plan for ckan.org reorganisation from http://ckan.okfnpad.org/documentation (22/7/11) (ticket made by dread):

/solutions
  /overview <--- current 'about' page (the software)
  /customers <-- say "in addition to our clients..."
  /features <-- needs most work (pictures etc)
/pricing/  <-- simplest thing possible ... talk with Ira (cover pricing and support)
/developers <--- links to http://docs.ckan.org/, getting the software, the wiki, the mailing list /contact
/blog

Tasks:

add tagline in header - "the data portal software"
make and add glossy PDF
change "data hub" to "data portal" passim (if only using one term)
add info@…
image banner on home page - redo screenshots without menus