CKAN: Ticket Query

#2362: Improve plugin documentaion, including examples.

kindly — Tue, 01 May 2012 16:13:11 GMT

Improve documentation for plugins and add examples. Could use ckanext-example as a base.

#2550: User types

ross — Mon, 18 Jun 2012 11:06:30 GMT

Requirements

In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub).

Required interface

Once changes have been made to the user schema, for a given user we want to be able to:

determine if they have a paid or a free account, and
get a string name of the type of paid account.

Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin.

User Stories

User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases. For now it is adequate that we can manually control these things through paster commands.

Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all.

As a sysadmin I would like to be able to use a paster command to manually set a user's payment level, or remove it entirely.

As a sysadmin I would like to be able to run a paster command to view a list of users who have a payment plan, grouped by the plan that they have.

As a sysadmin I would like to be able to use the API to change the payment status of a specific user through user_create and user_update. This shouldn't be available to anybody else.

As a user, and only if I have one, I'd like to see my current payment level on my user profile page.

Tasks

[x] Tests

[x] Plugin based migration

[x] Code

[x] Model

[x] API

[x] Documentation

Estimates

#2867: 1.8 url_for error

ross — Thu, 16 Aug 2012 10:57:49 GMT

From: <thedatahub.org@…> Date: Thu, Aug 16, 2012 at 10:30 AM Subject: WebApp? Error: <class 'routes.util.GenerationException?'>: url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6 To: ckan-sysadmin@…, kindly@…

URL: http://ckan.net/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:155 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:131 in call << r'\1', oldpath)

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:239 in call << try:

res = WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

res = WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.storage:190 in file << return fapp(request.environ, self.start_response)

else:

h.redirect_to(file_url)

h.redirect_to(file_url)

Module ckan.lib.helpers:57 in redirect_to << if are_there_flash_messages():

kw__no_cache__? = True

return _redirect_to(url_for(*args, kw))

def url(*args, kw):

return _redirect_to(url_for(*args, kw))

Module ckan.lib.helpers:79 in url_for << # fix ver to include the slash

kwver? = '/%s' % ver

my_url = _routes_default_url_for(*args, kw) kw__ckan_no_root? = no_root return _add_i18n_to_url(my_url, locale=locale, kw)

my_url = _routes_default_url_for(*args, kw)

Module routes.util:265 in url_for << if not isinstance(url, str) and url is not None:

raise GenerationException?("url_for can only return a string, got "

"unicode instead: %s" % url)

if url is None:

raise GenerationException?(

"unicode instead: %s" % url)

GenerationException?: url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6 CGI Variables CKAN_CURRENT_URL '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'utf-8;q=0.7,iso-8859-1;q=0.2,*;q=0.1' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@…)' PATH_INFO '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '37236' REQUEST_METHOD 'GET' REQUEST_URI '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '<address>Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7f5312a51650> beaker.cache <beaker.cache.CacheManager? object at 0x7f5312a51790> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7f5312a516d0>> beaker.session {'_accessed_time': 1345109433.273328, '_creation_time': 1345109433.273328} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: >, ) paste.registry <paste.registry.Registry object at 0x7f5303b05610> paste.throw_errors True pylons.action_method <bound method StorageController?.file of <ckan.controllers.storage.StorageController? object at 0x7f530891fa10>> pylons.controller <ckan.controllers.storage.StorageController? object at 0x7f530891fa10> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7f52f864f850> pylons.routes_dict {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'} repoze.who.logger <logging.Logger object at 0x7f53127b2b50> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 139994769488336>, 'friendlyform': <FriendlyFormPlugin? 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin? 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>} routes.route <routes.route.Route object at 0x7f531296e690> routes.url <routes.util.URLGenerator object at 0x7f52f82cd450> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52f5870cd8> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7f52f82cd450>, {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'})

#235: Resource format normalization and detection

dread — Mon, 18 Jan 2010 15:13:24 GMT

Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service.

Sub-tasks:

Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -> "text/csv" and "Name mappings", i.e. "text/csv" -> "Comma-Separated Spreadsheet".
Collect and map surface forms from all CKANs
Access this via Swiss and apply, store as a PackageResource? extra field pending #826 (Resource extras).
Add heuristics for format auto-detections:
- Map well-known file extensions
- Recognize obvious magic (Zip, Tar)
- Peek into Zipfile/Tarfiles?
Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar)
See also: #816 (Autocomplete for the resource format field)

#250: RDF link in Atom feed

dread — Thu, 18 Feb 2010 15:41:35 GMT

Add link to RDF representation of a package in our Atom feed.

#253: Package relationships

dread — Mon, 22 Feb 2010 16:05:21 GMT

Overview

Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management.

Examples

There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial.

In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.)

OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it.

IPCC collection of data linked / mirrored. Not sure if there are useful relationships here?

Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM.

See more examples discussed here: http://trac.ckan.org/ticket/253

Implementation

This is split into four tickets:

Model: ticket:254
Read in WUI: ticket:255
Edit in WUI: ticket:256
API: ticket:257

No need for write access to be provided API for the moment.

This ticket also encompasses ticket:169 (Package derivations) and ticket:176 (Package dependencies).

#277: Set some config options / settings in WUI (extension)

dread — Mon, 22 Mar 2010 16:21:01 GMT

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN
## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png
## Site tagline / description (used on front page)
ckan.site_description =
## Used in creating some absolute urls (such as rss feeds, css files) and
## dump filenames
ckan.site_url =
## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico
## An 'id' for the site (using, for example, when creating entries in a common search index)
## If not specified derived from the site_url
# ckan.site_id = ckan.net
## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net
## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end =

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

[namespace]: ? only if KeyValues? (for settings this would then always be settings)
key
label
value (json)
type (e.g. date and to specify in advance what type should be)
description
tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

/ckan-admin/settings
Show label, plus description plus text field

Depends

Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)

#285: Paginate list of packages on tag read page

rgrp — Wed, 07 Apr 2010 18:23:26 GMT

Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ...

#301: Package discussion pages

rgrp — Mon, 26 Apr 2010 16:57:13 GMT

A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package.

It provides a way for people to make suggestions about a package without needing access to main package.

#331: Timezone of CKAN timestamps should be configurable

rgrp — Mon, 31 May 2010 10:40:40 GMT

Revisions are timestamped using the server's clock, which may not relate to the expected timezone for the site. e.g. the Norway site has a server on GMT. No timezone info is displayed either.

Would like to set timezone for a CKAN instance to use in rendering revision timestamps. For example, use CET or EST timezone.

#350: Search engine optimisation

dread — Tue, 15 Jun 2010 09:35:41 GMT

Need to research what can easily be done to improve CKAN packages in the search rankings.

Comments from Glen Barnes:

We've been pretty successful at SEO without even really trying (see http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (http://www.ckan.net/package/ascoe -> http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly ("Atmospheric Chemistry Studies in the Oceanic Environment" on the example above). Some basic SEO 101 on page optimisations.

#351: Homepage: list new, updated and 'hot' packages

dread — Tue, 15 Jun 2010 09:56:56 GMT

Have a simpler list of exciting data, as opposed to the big revision list.

For example:

Hot data
===========
New packages: package1, package2, package3
Updated resources: package1, package2, package3
Popular packages:

#369: "Package Listing Key" should appear on Tag results

shudson@… — Thu, 22 Jul 2010 18:00:34 GMT

Currently there's a nice legend titled "Package Listing Key" that appears in right side of "Browse Packages" results. The same key should show on other search results like when searching for a tag.

#370: Use better email encryption for author_email and maintainer_email

shudson@… — Thu, 22 Jul 2010 18:03:39 GMT

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.

#653: Trackback links for packages

dread — Tue, 21 Sep 2010 09:40:25 GMT

When people link to a package, a track-back link is auto-created. (Similar system as for blogs).

As suggested by Tim Davies:

Allowing some form of ‘track back’ against datasets When a non-technical user comes to look at a dataset it would be really useful for them to be able to see if anyone has created an interface interpretation of it already.

I found quite a few cases in research of end-users struggling to make sense of a dataset when good interfaces to that data had already been built and blogged about, but without there being any link from the dataset listing to those data uses. Accepting track backs could also make it easier for technical users to find blog posts / shared code etc. relating to a given dataset.

#737: Markdown syntax summary page

dread — Fri, 22 Oct 2010 16:59:09 GMT

I suggest we produce a quick Markdown cheat-sheet page, showing the key runes: e.g. create a title and quote some text. This page can link to the full Markdown docs for advanced users.

A user going to the Markdown docs that we link will have to read a couple of pages of the raison-d'etre of Markdown before he gets to the syntax. And it's not very easy to read, and being white on black it looks like proper geek stuff.

#765: Read-only mode - API usage

dread — Tue, 26 Oct 2010 11:18:17 GMT

All writes to the API are captured and you are returned an error explaining the reason.

Possible errors:

503 temporary maintenance
403 forbidden (if server if permanently read-only)

#811: Extra field editing form layout breaks when there are long field names

cygri — Wed, 17 Nov 2010 11:53:32 GMT

The layout of the editing section for extra fields breaks when a field name is slightly too long. Field names jump over to the right. See http://ckan.net/package/edit/dbpedia for examples.

#812: Package edit form only allows three extra fields

cygri — Wed, 17 Nov 2010 11:56:50 GMT

Rationale

The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview).

Implementation

A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

#818: Rethinking the author and maintainer fields

cygri — Wed, 17 Nov 2010 14:18:44 GMT

The semantics of the Author and Maintainer fields are really unclear at the moment. This leads to very inconsistent usage. Also, perhaps Name and Email are not the only fields that are needed for a contact.

Here is a table that shows the current usage of these fields in CKAN: http://richard.cyganiak.de/2010/ckan/ckan-ppl.html

We note several problems:

Author and Maintainer are often the same
Author and Maintainer are often used interchangeably
People really want to specify URLs for the contacts and stick them into random places because there is no field for it
Multiple comma-separated names in a single field

I'm not sure what to do about this, but a redesign is necessary in my opinion.

Some ideas:

Remove the maintainer field?
Make really clear that Author doesn't refer to the metadata on CKAN, but to the original data
Add an “author URL” field?

#837: CKAN integration with freebase gridworks / google refine

rgrp — Tue, 30 Nov 2010 18:10:09 GMT

Thread: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

Scenario 1

User installs Refine and CKAN extension for refine
On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
They edit the dataset on Refine
On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)

NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required)

Scenario 2

User visits a package on CKAN.net (or another CKAN instance)
There is a button on the page "View and edit this dataset in Google Refine"
Click button -- ask them if they have Google refine installed
- Yes: instructions for loading dataset into refine
- No: load dataset in hosted version of google refine (we could run this)
User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset.

#895: Add version number (or simular) to css/js includes query string

memespring — Thu, 06 Jan 2011 19:49:42 GMT

Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g.

mycssfile.css?v=12345678

#909: DCat importer for CKAN

pudo — Mon, 17 Jan 2011 12:05:58 GMT

Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported:

CKANrdf generated exports
opengov.se RDF (not really DCat)
Sunlight Nationdatacatalog as harvested by dcat-tools.

#979: Edit Resource extras in the API

dread — Fri, 11 Feb 2011 13:09:37 GMT

Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API.

#1009: Improvements to user accounts sytem

rgrp — Fri, 25 Feb 2011 12:13:11 GMT

Forgot password (email a new password)
Confirm email
Do not show register page if you are logged in (redirect to home page)
ticket:1010 - listing of users.
- Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)

#1069: Stub datasets (request for datasets)

rgrp — Fri, 01 Apr 2011 14:08:39 GMT

Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki.

We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ...

(Just as we have datasets listed that exist but aren't available ...)

Alternative would be to have a request for datasets subsystem.

I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists.

Implementation

Agree a new dedicated tag. e.g. todo.does-not-exist

This ticket has clear links to http://getthedata.org/

#1077: Move to simpler vdm system

rgrp — Fri, 08 Apr 2011 23:14:24 GMT

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset
for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))

Question:

does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

Easier support for pending state and similar behaviour
No need to introduce new tables (and hence migrations) when making something revisioned (or not).

Disadvantages

Migration is required
More difficult to query revision history.
- Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
Performance (?)
- Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
  - Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table

Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

#1096: [super] CKAN Hosted

pudo — Tue, 19 Apr 2011 17:44:22 GMT

Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

introduce model.Site and c.site
- site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI)
Subdomain detector to activate sites.
use site in Authorizer instead of System, have a NullSite? for global things
allow cross-site search
packages are in a list of sites, m:n rather than 1:n
- list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally.

#1120: Atom feeds of each tag

tsm — Mon, 02 May 2011 04:08:05 GMT

Tags could/should have an Atom feed. This would mean that every edit to relevant packages could be easily monitored. See [1].

[1] http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001162.html

#1130: First time users

lucychambers — Mon, 09 May 2011 10:59:21 GMT

Send users to FAQ first time on CKAN

#1144: Support DSPL

timmcnamara — Thu, 19 May 2011 00:06:49 GMT

DSPL, the Dataset Publishing Language, is being promoted by Google for its "Google Public Data Explorer" system. It is an XML format with metadata.

The format is described on the developer docs ofthe Google Code site.

Google provides a Python script which reads CSV data and generates DSPL

Sample from http://code.google.com/apis/publicdata/docs/dspl_sample.html:

<?xml version="1.0" encoding="UTF-8"?>
<dspl xmlns="http://schemas.google.com/dspl/2010"
    xmlns:geo="http://www.google.com/publicdata/dataset/google/geo"
    xmlns:geo_usa="http://www.google.com/publicdata/dataset/google/geo/us"
    xmlns:time="http://www.google.com/publicdata/dataset/google/time"
    xmlns:quantity="http://www.google.com/publicdata/dataset/google/quantity"
    xmlns:entity="http://www.google.com/publicdata/dataset/google/entity">
  <import namespace="http://www.google.com/publicdata/dataset/google/time"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/quantity"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/entity"/>
  <import namespace="http://www.google.com/publicdata/dataset/google/geo"/>
  <info>
    <name>
      <value>My statistics</value>
    </name>
    <description>
      <value>Some very interesting statistics about countries</value>
    </description>
    <url>
      <value>http://www.stats-bureau.com/mystats/info.html</value>
    </url>
  </info>
  <provider>
    <name>
      <value>Bureau of Statistics</value>
    </name>
    <url>
      <value>http://www.stats-bureau.com</value>
    </url>
  </provider>
  <topics>
    <topic id="geography">
      <info>
        <name><value>Geography</value></name>
      </info>
    </topic>
    <topic id="social_indicators">
      <info>
        <name><value>Social indicators</value></name>
      </info>
      <topic id="population_indicators">
        <info>
          <name><value>Population indicators</value></name>
        </info>
      </topic>
      <topic id="poverty_and_income">
        <info>
          <name><value>Poverty & income</value></name>
        </info>
      </topic>
      <topic id="health">
        <info>
          <name><value>Health</value></name>
        </info>
      </topic>
    </topic>
  </topics>
  <concepts>
    <!-- As noted in the tutorial, this concept should extend quantity:amount.-->
    <concept id="population">
      <info>
        <name>
          <value>Population</value>
        </name>
        <description>
          <value>Size of the resident population.</value>
        </description>
      </info>
      <topic ref="population_indicators"/>
      <type ref="integer"/>
    </concept>
    <!-- This country concept is defined for educational purposes only. A country
    concept exists in the Google geo dataset. See:
    http://code.google.com/apis/publicdata/docs/canonical/geo.html -->
    <concept id="country" extends="geo:location">
      <info>
        <name>
          <value>Country</value>
        </name>
        <description>
          <value>My list of countries</value>
        </description>
      </info>
      <type ref="string"/>
      <property id="name">
        <info>
          <name><value xml:lang="en">Country name</value></name>
          <description>
            <value xml:lang="en">The official name of the country</value>
          </description>
        </info>
        <type ref="string"/>
      </property>
      <table ref="countries_table"/>
    </concept>
    <!-- This US state concept is defined for educational purposes only. A US state
      concept exists in the Google geo US dataset. See:
      http://code.google.com/apis/publicdata/docs/canonical/geo.us.html -->
    <concept id="state" extends="geo:location">
      <info>
        <name>
          <value>State</value>
        </name>
        <description>
          <value>US states</value>
        </description>
      </info>
      <type ref="string"/>
      <property concept="country" isParent="true"/>
      <table ref="states_table"/>
    </concept>
    <concept id="gender" extends="entity:entity">
      <info>
          <name>
          <value>Gender</value>
        </name>
        <description>
          <value>Gender, Male or Female</value>
        </description>
        <pluralName><value>Genders</value></pluralName>
        <totalName><value>Both genders</value></totalName>
      </info>
      <type ref="string"/>
      <table ref="genders_table"/>
    </concept>
    <concept id="unemployment_rate" extends="quantity:rate">
      <info>
        <name>
          <value>unemployment rate</value>
        </name>
        <description>
          <value>The percent of the labor force that is unemployed, not seasonally
            adjusted.</value>
        </description>
        <url><value>http://www.bls.gov/cps/cps_htgm.htm</value></url>
      </info>
      <topic ref="social_indicators"/>
      <type ref="float"/>
      <attribute id="is_percentage">
        <type ref="boolean"/>
        <value>true</value>
      </attribute>
    </concept>
  </concepts>
  <slices>
    <slice id="countries_slice">
      <dimension concept="country"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <table ref="countries_slice_table"/>
    </slice>
    <slice id="states_slice">
      <dimension concept="state"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <metric concept="unemployment_rate"/>
      <table ref="states_slice_table"/>
    </slice>
    <slice id="countries_gender_slice">
      <dimension concept="country"/>
      <dimension concept="gender"/>
      <dimension concept="time:year"/>
      <metric concept="population"/>
      <table ref="countries_gender_slice_table"/>
    </slice>
  </slices>
  <tables>
    <table id="countries_table">
      <column id="country" type="string"/>
      <column id="name" type="string"/>
      <column id="latitude" type="float"/>
      <column id="longitude" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">countries.csv</file>
      </data>
    </table>
    <table id="countries_slice_table">
      <column id="country" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <data>
        <file format="csv" encoding="utf-8">country_slice.csv</file>
      </data>
    </table>
    <table id="states_table">
      <column id="state" type="string"/>
      <column id="name" type="string"/>
      <column id="country" type="string">
        <value>US</value>
      </column>
      <column id="latitude" type="float"/>
      <column id="longitude" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">states.csv</file>
      </data>
    </table>
    <table id="states_slice_table">
      <column id="state" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <column id="unemployment_rate" type="float"/>
      <data>
        <file format="csv" encoding="utf-8">state_slice.csv</file>
      </data>
    </table>
    <table id="genders_table">
      <column id="gender" type="string"/>
      <column id="name" type="string"/>
      <data>
        <file format="csv" encoding="utf-8">genders.csv</file>
      </data>
    </table>
    <table id="countries_gender_slice_table">
      <column id="country" type="string"/>
      <column id="gender" type="string"/>
      <column id="year" type="date" format="yyyy"/>
      <column id="population" type="integer"/>
      <data>
        <file format="csv" encoding="utf-8">gender_country_slice.csv</file>
      </data>
    </table>
  </tables>
</dspl>

#1145: Support the Handle System

timmcnamara — Thu, 19 May 2011 00:26:15 GMT

The Handle System is an initiative to provide persistent references for resources. That is, it's basically a proxy system for preventing link rot.

Its documentation is here: http://www.handle.net/. Servers running CKAN could host a "Local Handle Service", which redirects a hash of a resource to an actual URL.

Some suggested use cases:

Researcher would like to cite where data came from
Agencies would like to have a way to prevent vendor lock-in from CKAN if they decide to move to another platform

#1165: Add multi-site support to ckan

nils.toedtmann — Thu, 26 May 2011 12:41:07 GMT

Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3).

That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's "<VirtualHost?>", or tracd). Depending on the "Host:" header in the HTTP1.1 request, it would choose which local ckan ini file to obey.

I see two ways to constitute the map hostname-to-ini-file map:

ckan reads a set of ini files, and each ini file declares which servers names it is responsible for
In a global ini file, there are directives mapping servernames to ini files.

In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc.

#1179: Support tag aliases

timmcnamara — Tue, 07 Jun 2011 06:47:01 GMT

A small number of tags are near-duplicates of each other.

Perhaps we could support word stemming from NLTK and/or manual tag aliases:

statistics statistik ... survey surveying surveys

#1182: Comments from deleted packages appear in "Recent Comments" feed

timmcnamara — Thu, 09 Jun 2011 22:24:11 GMT

When a package has been deleted, say for spam moderation, comments still appear in the recent comments section.

This is a problem because non-admin users will be shown a warning that they're not authorised to view the package if they click on the link.

At CKAN.net currently, this affects the most recent comment.

#1184: Support Wuala as CKAN storage option

timmcnamara — Tue, 14 Jun 2011 06:59:11 GMT

Most of CKANs storage options are tied to the USA. This brings concerns of data security for some organisations who may wish to adopt the system. Wuala is a distributed file system that stores data in a peer-to-peer manner. The company behind it, LaCie? sells storage for a fee. However, they also enable clients to have 'free' storage space when machines act as a storage node.

In order to be a storage node, a machine needs to be online for more than 14% of the time - roughly 4h per day. Most CKAN servers are likely to have a far greater uptime than this.

Supporting Wuala would go some way to enabling CKAN to be used in a secure manner. That is, CKAN could be promoted for organisational use where there is lots of data to be stored and large geographic distances to be managed. There is a Python client available and a fairly long Google Tech Talk that overviews the system.

#1185: Administrators can't delete packages from web UI

timmcnamara — Wed, 15 Jun 2011 04:23:38 GMT

Administrators have "View", "Edit" and "History" tabs. However, I can't see a way to delete a package from the web UI.

Version: CKAN.net as of today

#1188: Allow diffing against initial (blank) package version

nickstenning — Wed, 15 Jun 2011 15:52:40 GMT

Currently the history page only allows diffing between different versions of a package, but there doesn't appear to be any easy way to see the changes introduced by the first version of a package.

I'm requesting the ability to diff against a "blank slate" initial state of a project, so I can see the content of the first project commit.

Not sure if this is a vdm feature, so I'm putting this ticket in against ckan.

#1201: seperate out logic in atom feeds to logic layer.

kindly — Fri, 24 Jun 2011 15:21:32 GMT

Simplify the logic in the atom feed an make all feeds use logic layer to return lists.

#1227: Display packages' tags in search results

timmcnamara — Tue, 19 Jul 2011 00:11:02 GMT

In when displaying search results, it would be useful to also display the tags of a package. Sometimes it's difficult to infer the scope of what the package does from the title and the first sentence of the description. Tags are quite concise way to display rich information.

ENV=datacatalos.org, with CKAN 1.4.2a

#1232: [super] Interface improvements

thejimmyg — Wed, 20 Jul 2011 16:11:36 GMT

Child tickets:

#1194 "Welcome back" message for newly registered user
#1202 Links to datapkg utility don't lead to info about it
#925 Change the search box icon to remove the down arrow
#923 Search box doesn't work in leaderboard page in stats extension
#1034 Flash message cached
#737 Markdown syntax summary page
#811 Extra field editing form layout breaks when there are long field names

#1233: [super] Improve wiki-style functionality for history

thejimmyg — Wed, 20 Jul 2011 16:29:52 GMT

At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas:

#191 Searching by modification date
#193 Searching by time-related field
#301 Package discussion pages
#1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages
#1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page
Other improvements as per my word doc.

#1240: [super] API v4

rgrp — Sun, 24 Jul 2011 16:41:00 GMT

(Just creating this ticket as somewhere to keep notes)

Decide on REST api versus action API
- Do we want to support both?
Tidying
- Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
- extras merged into normal field list in package
- Get rid of /rest/ so just have api/v1/package
- Get rid of separation of search api from 'rest' api
  - Propose that GET on REST index is search e.g. /package/?q=...
    - This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
Resource read/write in API (separate from package)
- Does this need authorization work?
user/account API - read/write
Remove autocomplete -- can just use search
- Do not worry about backwards compat as should only be used in our js (if others using it too bad!)

#1255: Drupal consistancy checks.

kindly — Mon, 01 Aug 2011 17:32:48 GMT

Make a robust way to make sure the drupal database is consistent with the ckan data.

#1273: Create docs for API v3

amercader — Mon, 15 Aug 2011 12:41:23 GMT

#1278: Refactor authorized_query calls

amercader — Mon, 15 Aug 2011 13:32:57 GMT

There are some functions that still use the Auhtorizer().authorized_query method:

./ckan/controllers/authorization_group.py:24:        query = ckan.authz.Authorizer().authorized_query(c.user, model.AuthorizationGroup)
./ckan/lib/base.py:237:        groups = ckan.authz.Authorizer.authorized_query(c.user, model.Group,
./ckan/lib/search/sql.py:55:        q = authz.Authorizer().authorized_query(username, model.Group)
./ckan/lib/search/sql.py:118:        q = authz.Authorizer().authorized_query(self.options.get('username'), model.Package)
./ckan/logic/action/get.py:154:    query = Authorizer().authorized_query(user, model.Group, model.Action.EDIT)
./ckan/tests/test_authz.py:158:        q = self.authorizer.authorized_query(self.notadmin.name, model.Package)
./ckan/tests/test_authz.py:353:        q = self.authorizer.authorized_query(self.notmember.name, model.Package)
./ckan/tests/test_authz.py:357:        q = self.authorizer.authorized_query(self.member.name, model.Package)
./ckan/tests/functional/test_authorization_group.py:44:        group_count = Authorizer.authorized_query(u'russianfan', model.AuthorizationGroup).count()

#1287: NAVL validation errors - Junk fields should be listed explicitly

dread — Wed, 24 Aug 2011 16:25:02 GMT

When you create a package, but specify a key that is not allowed (e.g. 'relationships') then you get error message:

{"__junk": ["The input field __junk was not expected."]}

It should mention the actual key which is not expected. e.g.

{"relationships": ["The input field 'relationships' was not expected."]}

Kindly said that James' version of NAVL was better in this respect, so this might be best solved by moving to that.

#1341: Delete spam users from ckan

kindly — Wed, 14 Sep 2011 10:10:34 GMT

Spam users where added to thedatahub and we need to clean them.

#1355: Package extras property does not include the newly created ones

amercader — Mon, 26 Sep 2011 10:48:46 GMT

The extras in the package object sent to the extensions after editing (https://bitbucket.org/okfn/ckan/src/01efd5649c10/ckan/logic/action/update.py#cl-226) do not include the newly added.

#1358: Generate configuration documentation automatically from the deployment_ini_tmpl file

rgrp — Mon, 26 Sep 2011 22:32:30 GMT

At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs.

Suggest we write a script that automatedly generates reference documentation for the config from the source.

May be obsoleted by #277 (some config in db)

#1384: CKAN wiki needs updating to refer to thedatahub.org instead of ckan.net and datasets instead of packages

shevski — Wed, 12 Oct 2011 10:07:57 GMT

Most articles still refer and link to ckan.net, wiki.ckan.net and to packages

#1403: Refactor groups index page

zephod — Mon, 17 Oct 2011 10:31:52 GMT

Groups are listed alphabetically with paging - not an ideal user experience. We would like to list groups in order of 'popularity': The number of datasets they contain.

Following this chain of thought, then, it would be nice to rearrange the groups table by clicking on column headers and having it sort by that column.

Furthermore, then, we'd like to implement a full-fledged groups search feature (if this is at all feasible).

The forthcoming groups refactor will probably have some bearing on this task.

#1406: Re-enable RSS subscriptions

zephod — Mon, 17 Oct 2011 14:22:07 GMT

RSS 'subscribe' buttons appeared in many places on the site but were not very helpful. They took (confused) users pointed to the raw feed code, and Google Reader could not understand the feed. Safari, however, could interpret it correctly.

Their presentation needs to be clear and consistent; the RSS feed really needs testing in a variety of readers; and we need to decide exactly which items should get a feed. (Package updates? Groups?)

#1411: Force resource format to be lower case (also mimetype)

rgrp — Sat, 22 Oct 2011 21:40:04 GMT

Format should be lowercase. Automatically lower case (for extra points have a bit of javascript to force lower case when entering).

Even more points: do a update on thedatahub repo to make all format lower case (or script this as an update?)

#1414: track user log-ins on thedatahub.org

shevski — Mon, 24 Oct 2011 11:13:02 GMT

Set up tracking for user logins so that we have stats about how many active users of thedatahub exist want to be able to see who logged in the the last x months

#1423: Edit resources suggestions

markbrough — Wed, 26 Oct 2011 15:11:46 GMT

Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: http://thedatahub.org/dataset/edit/iati-registry
Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in.

#1429: Provide DOIs for datasets in a CKAN instance

rgrp — Sun, 30 Oct 2011 12:21:45 GMT

DOI = digital object identifier = http://www.doi.org/

As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way.

Details

Probably implement as extension rather than core

#1432: [super] Data processing system for CKAN and Webstore

rgrp — Tue, 01 Nov 2011 10:19:07 GMT

Super ticket: #1190

A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)

#1457: Bug with DataNL instance

jilly.mathews — Thu, 10 Nov 2011 11:37:21 GMT

"when logging into http://register.data.overheid.nl/ with OpenID, the /user/me page gives a 404. CKAN version 1.3.4."

n the manual it says an API key kan be created via http://test.ckan.net/user/me /. However, when I try the corresponding http://register.data.overheid.nl/user/me, I get a 404 error (not found).

#1458: Support previewing kml files in data viewer

rgrp — Thu, 10 Nov 2011 14:48:08 GMT

Super ticket: #1151 (viewing geo data)

E.g. preview of http://thedatahub.org/dataset/louisvillecrime should bring up a map

#1459: Featured Dataset feature

rgrp — Sat, 12 Nov 2011 15:50:12 GMT

Provide way to mark a dataset as featured. Featured database show up on the front page.

TODO: detail this more.

#1507: Minor fixes to dataset add on Group edit form - 0.5d

rgrp — Mon, 05 Dec 2011 12:33:49 GMT

Group edit dataset add form needs some work

Dataset name is not cleared when you add
No way to remove item from list of datasets to be added if I make a mistake
(2nd Apr 2012) It now seems that option to add multiple datasets at once has disappeared (perhaps during the CSS/HTML refactor ...)

#1535: Plump for auth header of: X-CKAN-API-KEY

dread — Wed, 07 Dec 2011 17:31:22 GMT

When using the API, the apikey needs to be supplied in a header called 'Authorization'. Because some proxys / deployments use this header for other things, a configurable header was provided as an alternative, with default "X-CKAN-API-KEY".

Rufus suggests having *one* way for this. a) making this not configurable any more b) making X-CKAN-API-KEY the default

(keep Authorization allowed, but not documented, for backwards compatibility)

#1550: Allow simple auth via the API

ross — Fri, 16 Dec 2011 15:29:29 GMT

It should be possible to pass userid/username and api key and obtain a response from CKAN for external services that use CKAN auth. Those services shouldn't be talking to the DB directly.

#1557: Complete Webstore Preview Extension

jilly mathews — Mon, 19 Dec 2011 10:40:53 GMT

Finish any work out standing on web store preview extension to be able to package and release.

Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted.

#1558: Publisher Tools

jilly mathews — Mon, 19 Dec 2011 10:46:13 GMT

Summarise final set of requirements for this and finish development and test. Estimated 10 working days.

#1560: Follow extension

jilly mathews — Mon, 19 Dec 2011 10:51:19 GMT

Estimate 2 days to finish dev and test.

David can you add any info needed here?

#1561: To do extension

jilly mathews — Mon, 19 Dec 2011 10:52:52 GMT

Can we finish this ready for release on data hub and CKAN Hosted.

#1562: Finish Geo Spatial

jilly mathews — Mon, 19 Dec 2011 10:56:33 GMT

Estimated 4 weeks of Adria's time. I guess this will need to be broken down into more tickets. This feature is being requested by a number of potential customers and we have some ideas of requirements between Rufus and Jilly for this. This is the most popular new feature we talk about to new clients.

#1564: Structured Data (Data API)

jilly mathews — Mon, 19 Dec 2011 11:07:14 GMT

Basic websotre exists but this may be not what is described yet.

CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first.

#1565: Admin dashboard finished?

jilly mathews — Mon, 19 Dec 2011 11:11:32 GMT

Is testing complete and ready for release?

#1567: Finish QA extension

jilly mathews — Mon, 19 Dec 2011 11:19:59 GMT

Requires change to celeryd. Estimated 4 weeks.

#1569: Wordpressser

jilly mathews — Mon, 19 Dec 2011 11:27:36 GMT

How much effort will this be to be ready to use?

#1571: [super] Issues Extension

jilly mathews — Mon, 19 Dec 2011 11:30:25 GMT

"Proper" issues extension.

Etherpad: http://ckan.okfnpad.org/issues
Code: https://github.com/okfn/ckanext-issues

Overview

This would be an extension and improvement of existing todo extension - #842 (https://github.com/okfn/ckanext-issues)
- Current extension does work but lacks polish
- Should we polish or do significant improvements?
Functionality - see etherpad

Steps

~~(Rename todo(s) -> issue(s) in existing extension)~~

#1572: Meta data Harvester

jilly mathews — Mon, 19 Dec 2011 11:35:09 GMT

Need to write custom harvesters for each client. Is it worth having one for data hub?

#1573: Apps and Ideas

jilly mathews — Mon, 19 Dec 2011 11:36:33 GMT

Estimate 2 weeks for someone to finish and test.

#1578: [super] Re-enable and refactor ratings

rgrp — Mon, 19 Dec 2011 19:20:43 GMT

#1596: Refactor authz roles

dread — Thu, 22 Dec 2011 10:31:28 GMT

Suggestions from rgrp:

Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

(This ticket is split off from #1065)

#1598: Reinstate Ratings

rgrp — Thu, 29 Dec 2011 16:52:04 GMT

Ratings were disabled approximately a year ago because:

Unclear purpose and UX. What did ratings tell you? How useful were they?
Spamming (esp by bots: you could submit an anonymous rating via a GET request which caused problems)

Both problems are solvable and it would be nice to have this feature reinstated.

Purpose: can make this more purposable by limiting to logged in users (or at least distinguishing logged in from non-logged in users)
- Even better we could allow ratings to be made public (I'm interested in what someone else I respect finds important)
Spamming: limit to logged in users and / or use AJAX over an API to submit ...

#1604: Get ckanext-moderatededits working with CKAN 1.5+ templates

dread — Sat, 31 Dec 2011 17:27:09 GMT

ckanext-moderatededits requires an old and possibly development version of CKAN. It would be good to update it for later CKAN versions.

According to the README, you need CKAN from branch feature-1141-moderated-edits-ajax but the changelog suggests this branch went into version 1.4.2. So it possibly works with 1.4.2 and 1.4.3(.1). But CKAN 1.5 has revamped templates, so the genshi stream filters definitely don't work.

BTW history_ajax/read_ajax calls have been deprecated in CKAN since 1.5.2a and will need fixing up to use the Action API too as part of this.

#1606: metadata license config option

dread — Mon, 02 Jan 2012 10:45:30 GMT

Add a config option to choose the metadata licence. Set default to Open Database License.

Currently the dataset edit form says "Important: By submitting content, you agree to release your contributions under the Open Database License." This is hard-coded, but not suitable for when DGU uses the CKAN form - they use the OGL.

#1635: Email notifications (e.g. for activity streams)

seanh — Wed, 11 Jan 2012 17:56:27 GMT

CKAN should be able to send email notifications to users.

Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users.

One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well.

Need to implement (at least some of) #1634 before this can be implemented, in order to have something to send notifications about.

Analysis here: http://ckan.okfnpad.org/27

#1642: Extra link generators generate garbled HTML

pudo — Thu, 12 Jan 2012 15:29:31 GMT

I had a package descriptions with URLs that contain "group:foo". This produces garbled output as the system tries to generate two sets of links: the outer link and an inner link.

Need to fix the parser.

Text:

Webdienst basierende Bereitstellung von Geobasisdaten der Freien und Hansestadt Hamburg. Folgende Geobasisdaten werden als WebMapTileService? (WMT-S) für die Dauer des Wettbewerbs netzbasiert unter der Creative Commons Lizenz zur Verfügung gestellt: Digitale Orthophotos 40 cm Auflösung (Layer: apps4d_DOP40), Digitale Stadtkarte (Layer: apps4d_DISK), Digitale Regionalkarte (Layer: apps4d_DIRK), Digitale Karte 1:5000 (Layer: apps4d_DK5).

Metadateneinträge zu den Daten im PortalU:

One fix is quoting the URLs

#1643: Add fixed tags to thedatahub for better browsing

shevski — Thu, 12 Jan 2012 18:34:53 GMT

Similar to publicdata.eu, want to have themed areas such as finance, environment, census, etc and country tags

#1644: Order default dataset page by most downloaded resources on thedatahub

shevski — Thu, 12 Jan 2012 18:39:02 GMT

Instead of alphabetically as we do currently, alternatively by most viewed datasets

for http://thedatahub.org/dataset

#1652: How we intergrate with Drupal Multiligual?

ross — Mon, 16 Jan 2012 10:31:34 GMT

Analysis of how/where we can integrate with Drupal Multilingual

Is it likely to access CKAN via /en/data/... or /data/en/... (not sure, probably former) and we need to make sure that it ends up at /en/...

Check with Ian for other issues.

#1667: Add an extension point for rendering activity streams

seanh — Tue, 17 Jan 2012 10:29:11 GMT

Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML.

There needs to be an extension point where extensions can register their own rendering functions for particular activity types.

#1668: repoze version discrepency

dread — Tue, 17 Jan 2012 12:02:26 GMT

There's a discrepency in repoze.who versions between the source and package installs:

repoze.who - package 1.0.18 vs source 1.0.19
repoze.who-friendlyform - package 1.0b3 vs source 1.0.8

We get a test failure [1] with the 1.0b3 version (from the ubuntu 10.04 python-repoze.who-plugins package). But we've not noticed any problems on s057 instances (br, no, ie etc) which have the package versions of repoze.who.

The reason the package install uses the earlier packaged versions rather than the ones we'd like is that repoze uses all sorts of horrendous import hacks, making it too difficult to put into our 'ckan-conflict' source package.

James suggests we 'do something horrible like dynamically patch repoze on CKAN import'.

[1] http://buildbot.okfn.org/builders/builder-ckan/builds/1371/steps/shell/logs/stdio ERROR: ckan.tests.functional.test_user.TestUserController?.test_user_create_unicode

#1679: Default roles problem

dread — Tue, 17 Jan 2012 17:57:22 GMT

The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py).

The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different.

We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?

#1684: Remove all config from ckanext-archiver

ross — Thu, 19 Jan 2012 14:37:01 GMT

ckanext-archiver currently has a settings file (and a default) and it should be passed in all relevant information from the context.

Remove all settings (ARCHIVE_DIR and MAX_CONTENT_LENGTH and others) and pass them in from CKAN.

#1697: A Configurable list of states for a Dataset

rgrp — Tue, 24 Jan 2012 10:23:50 GMT

Currently have 'active' and 'deleted' suggest also:

'draft'
'hidden'

(Do we need both). Also write out workflows related to these.

#1717: [super] Search UX improvements

shevski — Thu, 26 Jan 2012 18:53:01 GMT

Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search)

Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page).

Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it.

More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu

Blue search button should be displayed in line with the search bar, not underneath

Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name

#1740: Get rid of `from module import ...`

seanh — Wed, 01 Feb 2012 11:06:09 GMT

It's really bad to do from module import * and CKAN has a lot of them. I suggest a three-pronged approach:

Don't add any more of them.

When you're programming if you see an easy opportunity to remove one then do so.

At some point we should task someone to go through the code and remove them all (which is what this ticket is for), but this will be a big job and may break things.

We should also get rid of most or all of the from module import foo and from module import foo, bar statements.

I think the right thing to do is just import module and then use module.foo in your code. But if you find yourself doing module.foo.bar then you may have a code smell.

See: http://docs.python.org/dev/howto/doanddont.html

#1745: Dataset search UX improvements as of Jan 2012

rgrp — Thu, 02 Feb 2012 23:22:21 GMT

Changes to make search both more exploratory and more satisfying to use

Search query build - #1603
- Ability to add new facet fields "live"
- That is add fields which then contain faceted options (a bit like data.hri.fi)
(??) Autocomplete / drop down on search (i.e. search while you type)
- Dubious about value / cost ratio here

Probably would involve to pure JS and HTML implementation.

Implementation

Probably require

API changes to expose solr style API directly #1737

#1747: Expire old activities

seanh — Sun, 05 Feb 2012 12:56:29 GMT

Currently the activity streams database tables just get longer and longer over time. Do we want to eventually delete the oldest activities, to keep the length of the table within limits?

#1748: Make activity streams conform to http://activitystrea.ms/ standard

seanh — Sun, 05 Feb 2012 14:06:24 GMT

At the very least we should make our JSON output conform to their spec:

http://activitystrea.ms/

#1749: Allow creating activity details through API

seanh — Sun, 05 Feb 2012 18:10:06 GMT

Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts.

#1750: Move ckan/lib/activity.py into the model

seanh — Sun, 05 Feb 2012 18:18:08 GMT

Move ckan/lib/activity.py moved to into the model - say ckan/model/activity_extension.py, because it's so tightly knit with the model code, whereas most of the lib code is used in the controllers.

#1778: Replace classmethod's with module-level functions

seanh — Mon, 06 Feb 2012 15:53:34 GMT

In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the get() classmethods in the model/ classes.

Doing it with module functions lets us type module.function() instead of module.Class.method().

In many cases we're importing classes directly with from module import Class and then doing Class.method(), but from module import foo is bad style: http://docs.python.org/dev/howto/doanddont.html

#1782: Improvements to datasets page UX (Feb 2012)

rgrp — Tue, 07 Feb 2012 10:10:41 GMT

Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?]
~~Icons for *types* of resources in resource list~~
List of 'Owners' (administrators) at top right of sidebar (a la group page)
- Much prefer owner to administrator ...

CKAN: Ticket Query

#2362: Improve plugin documentaion, including examples.

#2550: User types

Requirements

Required interface

User Stories

Tasks

Estimates

#2867: 1.8 url_for error

#235: Resource format normalization and detection

#250: RDF link in Atom feed

#253: Package relationships

Overview

Examples

Implementation

#277: Set some config options / settings in WUI (extension)

Use case

Implementation

Settings to be in DB

Settings / Options / KeyValues? Table

Loading settings from DB

WUI

Depends

#285: Paginate list of packages on tag read page

#301: Package discussion pages

#331: Timezone of CKAN timestamps should be configurable

#350: Search engine optimisation

#351: Homepage: list new, updated and 'hot' packages

#369: "Package Listing Key" should appear on Tag results

#370: Use better email encryption for author_email and maintainer_email

#653: Trackback links for packages

#737: Markdown syntax summary page

#765: Read-only mode - API usage

#811: Extra field editing form layout breaks when there are long field names

#812: Package edit form only allows three extra fields

Rationale

Implementation

#818: Rethinking the author and maintainer fields

#837: CKAN integration with freebase gridworks / google refine

Scenario 1

Scenario 2

#895: Add version number (or simular) to css/js includes query string

#909: DCat importer for CKAN

#979: Edit Resource extras in the API

#1009: Improvements to user accounts sytem

#1069: Stub datasets (request for datasets)

Implementation

Related

#1077: Move to simpler vdm system

Option 1: 'Changeset' Model

Implementation

Option 2: Simplify Revision Object Model

Discussion

Disadvantages

Conclusion

#1096: [super] CKAN Hosted

#1120: Atom feeds of each tag

#1130: First time users

#1144: Support DSPL

#1145: Support the Handle System

#1165: Add multi-site support to ckan

#1179: Support tag aliases

#1182: Comments from deleted packages appear in "Recent Comments" feed

#1184: Support Wuala as CKAN storage option

#1185: Administrators can't delete packages from web UI

#1188: Allow diffing against initial (blank) package version

#1201: seperate out logic in atom feeds to logic layer.

#1227: Display packages' tags in search results

#1232: [super] Interface improvements

#1233: [super] Improve wiki-style functionality for history

#1240: [super] API v4

#1255: Drupal consistancy checks.

#1273: Create docs for API v3

#1278: Refactor authorized_query calls

#1287: NAVL validation errors - Junk fields should be listed explicitly

#1341: Delete spam users from ckan

#1355: Package extras property does not include the newly created ones

#1358: Generate configuration documentation automatically from the deployment_ini_tmpl file

#1384: CKAN wiki needs updating to refer to thedatahub.org instead of ckan.net and datasets instead of packages

#1403: Refactor groups index page