{22} Trac tickets (2647 matches)

Authorization Groups are going to be deprecated in 1.8

It is still unclear how Authorization Groups fit in the new authz model, so they have been left out of the logic and auth refactoring (Tickets #1229 and #1253)

A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else.

(But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.)

Same in the Web interface and API.

This ticket ties in with #948 highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can #1076.

Stuff I've spotted:

• forms/*
• template/group/edit_form.html
• template/package/edit_form.html

This can go once new DGU form is in.

When installing CKAN, when doing "paster serve development.ini", lots of users encounter the error for every request:

ProgrammingError: (ProgrammingError) relation "user" does not exist

This is because the database tables have not been created - they have forgotten or missed the "paster db init" step.

Can we provide a better error to say that the database is not initialised yet?

Implementation options

1. At the start of every request we reflect the database tables and check they are there. This is rather expensive!

2. Uncached requests to the home page start with a cheap database query. If there is an exception then return this error about database setup. I really like Drupal's page for this that has in large letters that the site is currently off-line. Below a line, in small letters, there are developer suggestions on what is wrong and where to look to fix it.

• Review process of releases and updating strings (pot) and translations (po) files
  • c.f. ​http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html
• Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?)
• Review long list of strings for improvement
  • Consolidate any?
  • Get rid of i18n for strings in api
  • Try and remove javascript
  • Add in extensions
• #1374 Fix switching to en from other default language
• #1417 Fix browser language detection

Largely an integration of work from ckanjs

• #1295 Simplify package create form
• Improvements to edit form
  • Split edit form into sections
  • Remove preview from edit form (introduce inline preview on description/notes field)
• #1296 Improved resource adding/editing on dataset/page page
• #878 File upload integrated into resource creation
• #1297 In-place editing of notes field for existing packages

Subticket of #1294.

• Nice way to edit all resource attributes
• Nice way to add new resources
  • Including file upload (cf #878).
    • Cf work in ckanjs (this can be a straight integration).

This should be done from the logic layer or automatically from a session extension.

Core changes to theme in order to make it easier to re-theme:

• page_heading
• side bar menu must be switchable (left to right) (through config or css)
• Add optional_footer to complement optional_head
  • And adopt rule for template writers that all extra js must go in optional footer
  • That way we could move optional js to the bottom of the page to improve page load times.
• Re-add support for body-class ... (seems to be removed by #1108)
• more divs.
• better labelling for css (and javascript)

AFAICT there are no tests for the logic layer at the moment. I imagine this is an issue (if it is not please explain and close as wontfix).

Registering a new user directs you to that user's profile page but does not log you in. There is an empty set of "Recent changes" and no ability to modify the page. Users must follow the login link in the top-right corner to continue.

Registering a new user should immediately log you in.

From Florian:

I am having trouble with the characters "=&;".
if they occur e.g. in the title of a package, I get a 400-Bad-Request
when I try to register a package by POSTing to ckan/rest/api/package

However, when I PUT the same package as an edit to
ckan/rest/api/package/existing_package, it works and the title shows
correctly on the front end and API.

Reproduced

(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{"name": "test3", "title": "Test &"}' -H "Authorization:tester" 
HTTP/1.0 400 Bad Request
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 14:09:31 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Length: 87

"Bad request - JSON Error: Unterminated string starting at: line 1 column 27 (char 27)"

Apparently the search generator always makes two requests, even if you don't want to see the search results, which might be slow. Can this be optimised?

Maybe we should also provide a second search function that doesn't use the generator - the original simple search function (that leaves the user to deal with limit & offset).

Superticket: #1343

This avoids difficulties when they forget their password.

• (?) Email validation (navl does not have one!)
• Require email confirmation to be activated (?) (Answer: no, not at present -- see discussion below)

Focusing on improving Action API as the v3 API:

• have an optional parameter of the data_dict called "options". Options would contain items that would get passed into the context. e.g. {"options" : {"ref_package_by": "id"}}.
• instead of using API version to change the way packages are referenced, use the ref_package_by.
• All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'.
• Action API is v3 of api, replacement for v1 & v2. Default for most urls is still v1, but if url is /api/action then default to v3.

Next steps:

• Add search API (package, resource,
• Add Util API
• Clarify JSONP still applies
• Add doc strings, clarifying parameters

add extra fields to resource as outlined in

​http://wiki.ckan.net/Domain_Model/Resource

Specifically (all unicode strings except for those noted):

• name: a name for this resource (could be used in a ckan url)
• type: the type of the resource. One of: file | api | service | listing
• mimetype: standard mimetype (e.g. for zipped csv would be application/zip)
• mimetype_inner: mimetype of innermost object (so for example would be text/csv)
• size: [integer] size of the resource (content length) in bytes. Usually only relevant for resources of type file.
• last_modified: [datetime] the date when this resource's data was last modified (NB: not the date when the metadata was modified).
• cache_url: url for cache of object in ckan storage
• cache_last_updated: [datetime] timestamp when cached version was created
• webstore_url:
• webstore_last_updated: [datetime] timestamp when webstore was last updated

Split out of #852. Automated archiving of datasets (related to QA).

Automated archiving using worker process

• #890 - Timed actions in ckanext-queue
• #891 - Resource download worker daemon
• #892 - Make stored data available in WUI

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

I ran into a bug with the size field on resources.

• It would not accept an empty value from form (IMO this clearly equates to null/None)
• This could be fixed via using ignore_empty instead of ignore_missing
• However using this means there was no way to empty the field (e.g. i may just want to set the size field back to null not just change to another value)
• similar issues could arise around other fields (such as last_modified ...)
  • cf cset:645031d07b60

To solve this (cset:58acdcfe6d4e) i created an int_converter temporarily in logic/schema.py (this is almost certainly the wrong place). But I think it raises a bigger issue about the conversion layer and how it works.

• Disallow account creation via openid - #1386
• Require email field - #1319
  • Require email confirmation to be activated (?)
• Improvements to user page (e.g. show activity and more info about user) - #1396
• Modal user login - #1311

There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (​ticket) we experienced extraordinary memory usage peaks (​ticket). Here are the observed value with Apache default settings:

• eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
• s055, mpm-prefork: base level ~1GB, peaks up to 4GB
• s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (​changeset). This (together with two other tweaks) changed the situation drastically:

• s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to processes=3), and then observing the relation between maximum-requests=25...500 and memory consumption.

On 14/09/11 17:49, David Read wrote:

Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1

Various things in templates can be tidied up after work has finished on #1294:

• remove styling in page in group/index.html (put in css main)
  • ditto for package/search.html
  • ditto for home/index.html
• Forms.css - to merge into style.css; remove explicit imports elsewhere
  • @import at the top of style.css @import url(forms.css)
• Make a note: Success/failure controllers clash CSS with success/failure flash banners

The template for editing a user's profile seems to output the same form twice in one page. Furthermore, the page is made needlessly complex by the preview functionality. The neat inline preview used in package will make this page a lot neater.

• Fix the duplicate form
• Strip the preview functionality from the front-end, replacing it with js editor
• Strip the preview functionality from the back-end and test

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

We have an all new and much improved data preview system developed as part of QA in July. Need to integrate this.

• Is pure javascript
• Will have this in CKAN core but considered to be a plugin (so need datapreview in ckan.plugins list to work -- will enable this by default)
• Will depend on the ​http://wiki.ckan.org/DataExplorer

Major UX grip and incredibly simple to do. May also want to have basic autocomplete support within the list (trivial to do as well I believe).

(Talk to RP about this re how we define lists of licenses).

Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install).

Propose:

• ckan.simple_search config option
• If set:
  • query via simple query to database backend
  • do no specialized indexing

Extras

Remove

• TagSearchQuery? from lib/search (just do a search directly)?
• #1360: filter by downloadable and filter by open search options

Possible Extras

(Probably future improvements)

• Re-introduce full-text search indexing where supported in e.g. postgres and use for querying

We should ignore these but log the packages that are causing this to investigate.

ckan.js can be rearchitected to avoid instantiating showdown.js in the root namespace. That way, layout_base.html doesn't need to import the script (because we don't use the showdown-dependant parts of ckan.js).

The background tasks will use the logic API to access the task_status table so new logic layer functions are needed.

Regression found by updating datacatalogs.

While integrating the new theme and making a large number of UX tweaks, I've allowed the CSS file to bloat out of control with some very specific rules and subtle hacks, as well as a couple of clearfix <div> objects in the markup which isn't really good practice.

In general we want a logical CSS structure which is as easy to modify as possible. You should not find yourself being overwritten by highly specific rules all the time!

Actions

1. Clean up css
2. Add a config option ckan.template_head_end - arbitrary string inserted at end of <head> tag so you can add custom css etc (may also want to update theming docs to reflect possibility of using this)
3. (While we're at it): Add ckan.template_footer_end config option to deployment.ini_template. (This already exists, we just haven't told anybody)
4. The /users index page isn't linked anywhere. Clean it up and link it in the footer next to Groups etc.
5. package/new, package/edit, group/new and group/edit all include their forms differently. Use h.literal in all cases.
6. Fix the padding on minornavigation.

Background

David Read sent me the following as background:

22. basic theming is described here: ​http://docs.ckan.org/en/latest/theming.html

Deep customisation of the templates is done by creating an extension, with this as a template: ​https://bitbucket.org/okfn/ckanext-exampletheme/overview

That's what Augusto and the Brazilians have done ​http://alpha.dados.gov.br/dados/ and their code is here: hg clone ​http://dev.dados.gov.br/codigo/dev/tema-ckan <snip> Another thing of interest was us becoming compatible with Wordpress themes last December. Richard Pope worked with Rufus on this. I believe the base theme is 'twentyten' and I'm guessing that one would swap that css file with another to change theme, but I don't know. I don't know if this feature has been dropped since then. ​http://lists.okfn.org/pipermail/ckan-dev/2010-December/000073.html

Dave

Need to add way for extensions to trigger based on resource change and just resource url change.

Superticket: #1343

Creation of accounts from OpenID causes large number of problems:

• No guarantee of other profile info
• Poor username (just from openid)

We therefore will:

1. Require creation of user account via register (no auto-creation of accounts via openid)
   • sidebar of register page should point to login page rather than openid signin

2. Permit association of an openid with an account ...
   • How: paste in your openid url (requires change to edit form to have this)
     • won't work for google with their weird openid urls
   • [future] Nicer way is to have login via openid while logged in (which association then happening in background)

3. Migration (for 1+2):
   • (?) Generate decent usernames for all existing accounts
   • As this is nontrivial suggest instead we allow editing of usernames (by account owner and sysadmins). This is useful in its own right and is a reasonable 80/20 solution.
     • This does present the problem of re-associating commits with the new username. Best solution to this would be to switch revisions to point to userid rather than username. See #1534

4. [optional] Change display_name to always be username
   • Change My Account to Username at top right of all pages (once we know usernames are short ...)
     • could do this straight away by truncating long usernames (e.g. truncate at 20 chars ...)
     • Also should we lose the icon?

Now we don't use postgres search, all the tests involving search now don't need to be skipped when running on sqlite. Should help coders spot earlier if these tests break.

We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: ​http://ckan.okfnpad.org/queue (most relevant parts inlined here)

Preliminaries

• Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)

Configuration setup for daemons

Pass config through to workers i.e site_url, user, api_key. Need to make site user account. #1408

celeryd config:

All providers of tasks will add an item to the following entry point:

[ckan.tasks]
name = ckanext.{name}.tasks:....

celeryconfig.py

from pkg_resources import iter_entry_points
for entry in iter_entry_points:
     celeryimports.appen(....)

CELERY_IMPORTS = celeryimports

Work Items

• Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383
• Generate archiving request on resource url change [0.25d][0.25d] - #1399
• Make site user account.
• Make entry point system for celery config
• Archiver daemon #891
  1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
  2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
• Write to resource and task status table.[1d][0.75d]
• [Required?] Make archived data available in WUI - #892
• Documentation - #1400

Superticket: #1397

Generate archiving request on resource url change by implementing IResourceUrlChange and sending tasks to celery. [0.25d][0.25d]

Will be part of of ckanext-archiver extension.

Process:

1. A resource is added to CKAN
2. IResourceCreate event generated
3. IF: resource url points to ckan storage or falls within some other set of exclusion conditions then END else continue
4. Generate a Archiver.Update task with resource.id

Plan to migrate from mercurial git

• Convert from mercurial to git
  • Map named branches to normal branches
  • Also, if possible map usernames for RP: rgrp <​http://rufuspollock.org/> (mercurial) -> rgrp <rufus.pollock@…>
• Place at ​http://github.com/okfn/ckan (can delete current repo there as an out of date mirror)

Process:

• Do trial run
• Announce conversion date / time
• Require everyone to have pushed all outstanding changes at that time
• Do conversion
• Test
• Announce on list and ckan.org/

[Refactored] :: Follows on from #1506 UX changes.

• Declutter the sidebar. No yellow box.
• Facets to go on the left, rather than the right. More logical flow.
• Did you know you can search by author? Probably not. Find a nice way of presenting extended search options.
• Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results).
  • Update that look-and-feel to include the new resource icons created in #1506

Give site user system administrative rights.

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

I'd like us to display a one-off banner that prompts users who log into thedatahub.org to go and update & their profile & fill in their email address (if we could do it just for those who don't have an email address, then even better) That is, once they log in, they see a banner which says: Please update your profile [here] and add your email address, so you can receive notifications, be able to reset your password and get updates

Then when we build notifications or if we decide to survey people, we can actually contact them. Email address should be required as standard

DR: Also, mention their Full Name too - v. useful for selecting correct user in group curation

This is a scratch pad ticket with some comments on the current status of our ckan deb packages. I know that some of it is the deb packaging roadmap anyway, please forgive me if i mention them here again.

Rufus and me re-deployed some community ckan instances onto s022 (see ​http://trac.okfn.org/ticket/926). We followed the documentation ​http://docs.ckan.org/en/latest/install-from-package.html

• Deb package version number: the version of the deb package is "python-ckan 1309471251~149be76faabc+lucid-1", and it's hard to guess from there that it contains a ckan 1.4.2a
• When is 1.4.3/1.5.x expected as deb package?
• There was a bug in the DB upgrade script /usr/share/pyshared/ckan/migration/versions/029_version_groups.py (line 150) which looks like it was fixed 1.4.1==>1.4.2 but was nevertheless present in this deb package.
• The current script /usr/bin/ckan-std-install
  • does not set the Apache ServerName? according to the $INSTANCE variable
  • automatically configures a ckan extension named after $INSTANCE
  • depends on local postgres
  • could be replaced with "/usr/bin/ckan-deploy --name=ckan-std --domain=ckan-std.localhost (see next point)
• (i think this is exactly James' plan): have more generic deployment script /usr/bin/ckan-deploy as part of python-ckan which takes arguments like
  • --domain=cc.ckan.net
  • --aliases=$list-of-domains
  • --name=cc (defaults to "domain")
  • --no-db (does not configure a DB)
  • --sql-alchemy=$DB_CONFIG_STRING (also runs "paster --plugin ckan db upgrade --config")
  • --extension $list-of-extesions
  • ...

Currently can run as a celery task launched by paster command.

Still have to add a CKAN plugin so that QA will run when a resource is added or URL changed, and then update the extension controller/templates to look for QA data from the task_status table instead of directly on the resource.

• Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: ​http://thedatahub.org/dataset/edit/iati-registry
• Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in.

From Jonathan gray:

At the moment to delete a package you have to go to 'state' and change from 'active' to 'deleted'.

For me this was not very obvious, and someone recently wrote to me about this too.

Its a small issue - but unless you know what 'state: active' refers to, its not obvious what this means, nor that this is where to go if you want to delete a package.

From UI perspective this could be replaced by either a trash can or some text that says 'delete', perhaps with some prompt that explains what happens when you delete a package - explaining that non-admins won't be able to see it, and how you purge, etc.

DOI = digital object identifier = ​http://www.doi.org/

As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way.

Details

• Probably implement as extension rather than core

Why: current stable version of sqlalchemy. geoalchemy stuff required 0.7 and likely that some other things will require it soon.

Probably requires work on vdm ​https://bitbucket.org/okfn/vdm

NB: should have discussion before making 0.7 the default required version in CKAN core.