CKAN: Ticket Query

#1037: More Robust Harvesting for DGU

thejimmyg — Tue, 15 Mar 2011 14:00:02 GMT

CKAN's harvesting facility is now live on DGU but there are some major improvements that could be made to make it more robust and better fit the generic CKAN harvesting framework proposed in #987.

Some of the key issues:

Error reports do not currently contain the ID or title of the document with the error.
We only have "added" and "error" logging on jobs when we really need a report of "added", "updated", "not changed" and "errors" with the items in each referencing a real metadata document for which harvesting was attempted
We need deletion and editing of sources, without deleting the harvested documents or packages
We need a more robust harvesting mechanism than a cron job or we need to deal with the case of multiple cron jobs running at once.
We need to know the last time a list of documents was scheduled for harvest and the last time each one was fetched.

#515: Inconsistent use of 'location' header in API

dread — Wed, 25 Aug 2010 17:29:20 GMT

When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all.

I've removed the info about the header in the docs in the meantime.

#829: Admin CRUD broken

dread — Thu, 25 Nov 2010 14:24:03 GMT

Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception.

#833: [super] Administrative dashboard extension

rgrp — Fri, 26 Nov 2010 10:05:21 GMT

Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:

Purge revisions (or sets of revisions) and purge objects #1076
Set roles for users #1075
Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
- WONTFIX
Overview of activity
- WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.

https://bitbucket.org/okfn/ckanext-admin

Tickets

ticket:1031 - user autocomplete api in ckan

Notes

Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system

#840: On/off switch for ETags cache

dread — Thu, 02 Dec 2010 16:52:07 GMT

Use config variable to switch ETags caching on or off. Consider joining in with 'cache_enabled'.
Default setting for (all) caching should be off.
Needs documenting in configuration.rst

#854: Tests for authorization_group

dread — Tue, 07 Dec 2010 12:10:15 GMT

The coverage tool (run by buildbot in the ckan build) reports that only 24% of lines of ckan.controllers.authorization_group are run in tests and 38% of ckan.forms.authorization_group. This suggests a need for more tests.

#919: Package preview contains API & datapkg instructions

dread — Tue, 18 Jan 2011 10:32:10 GMT

Edit a package
Hit 'preview'.

The preview contains the section "CKAN API / datapkg" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the "CKAN API / datapkg" section too?

#936: Follow / watch package extension

rgrp — Sat, 29 Jan 2011 22:11:19 GMT

As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis).

NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'.

Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). Decision: use follow

Implementation

Interface

Become a follower:

Follow button on packages (if already watching say 'unfollow')

Package-related changes:

Show number of followers on a package
List followers of a package at /package/{name}/followers
- On a separate page

a package User-related changes:

List followed packages
- Either on user's page on a separate 'following' page. (NB: called 'following')
Does watching involve notifications (by email)
- Probably not: you can already subscribe to RSS feed after all and email not that necessary (?)
[Future - don't have activity stream yet] Show what packages a user has started/stopped followed on a user's public activity stream on their user page

Nitty-Gritty

Want to do this in ajax-y manner
API endpoint: /api/2/follower
Store data in a new follower table

API

/api/2/follower

follow => PUT / POST
{
   user_id
   object_type
   object_id
}

If this is submitted by a user with user.id != user_id => error (401)

unfollow => DELETE
/api/2/follower/package/{id}
=> list of followers
[
    { safe dictized user }
]

NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method.

Table

Called 'follower'

user_id, table, object_id, created
xxx, package, yyy, ...
xxx, user, yyy, ... [future]

Random Extras

What about following users as well

#1056: User links for OpenID users are broken

pudo — Fri, 25 Mar 2011 13:37:29 GMT

Use case:

Login using OpenID
Click on 'My account' - results in 404

Solutions:

User user.id instead of their name
Escape the URL properly.

#1075: Administrative dashboard - Edit Authorization related to System object

rgrp — Fri, 08 Apr 2011 16:23:19 GMT

Roles on System object are important because admin role on system equates to being a 'sysadmin' (i.e. able to do anything).

Make users sysadmin (either as separate action or as part of editing roles on system object)
/authz subpage for editing roles on system object
- Add and update user roles
- Add and update authz group roles
- List actions associated to roles at top of page (extra points for checkbox table with editability)
Document on http://wiki.ckan.net/Authorization what roles on System object 'mean' esp sysadmin role on System

Related Tickets

super ticket: #833
authz lib improvement and authztool refactor #1050

#1084: ckan.net RDF links changed

wwaites — Tue, 12 Apr 2011 13:58:37 GMT

need to make some changes for the links to semantic.ckan.net. it should use http://semantic.ckan.net/record/<package_id> now

append .rdf, .ttl, .nt, .dot, .json (even .html for an ugly table) to taste (or just leave off the suffix and let content negotiation take care of it)

the base url is changed, but it now uses id not name.

see for example:

#1089: Check for "--ckan" when running nosetests

dread — Tue, 12 Apr 2011 17:59:49 GMT

(because if you forget, you get difficult to understand errors, and more than one person has tripped up on this)

#1090: Visitor can't create packages on new CKAN install

dread — Tue, 12 Apr 2011 19:06:59 GMT

Default visitor roles in default config is reader, not anon_editor.

Problem caused by changes in #1066 (released in 1.3.3)

New installs will be affected, although simple to just increase permissions when the installer realises a visitor can't create packages.

The solution to the config getting out of sync with the code like this is to not have the default_roles in the config - refer to the code in the configuration instructions.

#1092: refactor logic layer to seperate out api, form logic

kindly — Thu, 14 Apr 2011 10:45:29 GMT

The logic layer is a bit too api centric. Make the reusable parts separate in preparation for the wui refactor.

#1093: 500 errors on GET to api/rest/licenses

dread — Fri, 15 Apr 2011 10:11:01 GMT

CKAN gets its license list from a license service, which can be a local file, but is often the http://licenses.opendefinition.org/2.0/ckan_original server. This server is currently flakey, but I think we only request the list on start up. The problem is we query it much more often than required. It is queried for every request to api/rest/licenses, and we are returning lots of 500 errors when the license server is timing out.

#1097: Sidebar hideable

dread — Wed, 20 Apr 2011 09:56:56 GMT

The web interface has a sidebar (#primary) which should be hidden in some pages. This is for QA extension and useful for package new and edit pages. Must be compatible with DGU theme.

#1098: --ckan-migration tests not initialised correctly

dread — Thu, 21 Apr 2011 09:15:36 GMT

Only tests with failing --ckan-migration fail, due to authz not being initialised.

#1100: Get buildbot running on ckan branches

dread — Thu, 21 Apr 2011 11:27:47 GMT

Need some changes to pip-requirements files in release branches.

#1106: Bugs related to routes arising from API refactor + removal of default routes

rgrp — Mon, 25 Apr 2011 16:02:40 GMT

Various bugs I've been encountering:

Autocomplete of tag names no longer works (no longer works on http://ckan.net/). Appears to be due to no longer have a routes for apiv2 (i'm seeing failing calls to: http://ckan.net/apiv2/package/autocomplete?callback=callback&incomplete=a)
Incorrect url generated for API in page footer (e.g. http://ckan.net/rest/get_api) due to use of old 'rest' rather than new 'api'

Latter issue was masked by existence of 'default' routes:

   map.connect('/{controller}', action='index')
   map.connect('/:controller/{action}')
   map.connect('/{controller}/{action}/{id}')

Having these is, I think, bad practice as it is better to be explicit and we should therefore remove asap.

In addition I think we should be cautious about 'default' routes in core such as:

    map.connect('/api/rest/:register', controller='api', action='list',
        conditions=dict(method=['GET'])
        )

As it makes it harder for extensions to introduce their own APIs (here one could perhaps add something at /api/rest/{my-object} but only by using before_map rather than after_map).