{22} Trac tickets (2647 matches)

I need to completely munge the rendered data coming from the app. There's not currently a plugin hook for this, where I get access to the entire environment. The most generic form of plugin would be WSGI Middleware. I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system.

Document how license system works and specifically how licenses can be configured.

Cost: 2h

Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE).

This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search.

We should fix this by finding a way to display placeholder attribute in all browsers.

• db_upgrade (in ckan/model/init) makes call to validate_authorization_setup but forgets about rest of initialisation.

• it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?)

We've fixed a few things in CKAN that were discovered and tracked by HRI in their own issue tacker. This includes:

• add authz checks to package edit links (cset:0752316cd2fe)
• replace gettext with _ in controllers, to support unicode error msgs (cset:822340e6077e)
• handle broken html in notes field without crashing on package read (cset:4b6be037dda0)
• update i18n (cset:37d57dc3c492, cset:ea03173f5e77, cset:f16f4ee40fe7)

We can just use the branch name to pull down specific versions of pip requirements so we don't need all the different versions in head.

Remove them and update any client systems.

Cost: 20m

The migration scripts have not been inline with the upgrade scripts. There are lots of discrepancies.

Should have the following features:

• A list of all packages with broken links on the resource
• Under each packages a description of any resources with broken links to include:
  • Link
  • Description
  • Format
  • First found to be broken date

[If you can't do the last one yet, don't worry]

Form for new package has CheckboxExtraField? checked, when the value is False. (as used in ckanext-dgu package v3 form)

The link for About > 'Project Home Page' on the CKAN.net footer should be changed to point to:

http://ckan.org/

The current link is broken.

Thanks, Jason

Try and minimise the amount of different caching methodologies, especially for api calls.

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

Rather than polluting the ckan core code base on ckan.net install with ckan.net specific changes these adaptations should be moved out into a dedicated ckan.net theme.

When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key.

The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController? to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...).

This is critical to incorporating any ajax editing into the frontend.

As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author).

A plain javascript library for interfacing with CKAN would be very useful (why? see below!). It would also be nice to have a pure html + javascript web interface to CKAN both for its own sake and to act as a demonstrator for the library.

Why?

• Development of bespoke interfaces -- much easier to edit html + javascript than to change ckan core
  • E.g. for specific communities e.g. geodata, science
  • Specialized tasks - multi-package editing
• Very easy deployment and integration (e.g. can drop in to getthedata or other sites)

Package edit form at hmg.ckan.net has no publishers any more.

Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class.

cf. ​https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16

Should have user listing at /user/ rather than user account page.

• list users, sorted by number of packages contributed/edited
• Move user home page to /user/{user-id}
• Paginated

Revision history information is not accessible, dgu want this.

The current oAuth extension tightly couples repoze.who, ckan, and data.no. These components should be decoupled; they also need more tests

Against ckan-1.3.1, when I create an authz group called "administrators" and visit /authorizationgroup/edit/administrators, I am unable to add more than one user to it.

Each time I add additional users, the existing user on the list is replaced with the new one.

We propose the following push approach using individual packages:

1. New, updated and deleted packages are pushed to a url endpoint on Wordpress (WPURL). This endpoint will be a configuration option on the CKAN side. A POST or PUT http method will get used. The payload will be a json document of the following form:

{

payload: The entity data as available from the REST API entity-type: 'Package', operation-type: 'create'|'update'|'delete' # one of these options

}

It will be sent as the body of the request with content-type set to application/json.

2. The wordpress side will provide a 200 on success. Any other response will be taken as a failure. On failure, submission will be archived and failure logged and notified to system administrator. Submission can be resent later automatically by sysadmin after review.

3. Pushes will happen continuously and approximately simultaneously with updates (a webhooks type model)

4. [optional] CKAN side will support configuring authorization for basic authentication if applied on WP side.

5. [optional] List queue status (including failures) on ckan adminstrative dashboard.

HRI don't like federation, want to login normal way only. Make this a config option and perhaps even mess with runtime repoze config

Schedule worker calls to be ran several times, with certain intervals.

Currently the default authz for a package is hard-coded to:

 <PackageRole user="visitor" role="editor" context="Package">,
 <PackageRole user="logged_in" role="editor" context="Package">,

This should be configurable in the config, so that you can have a more locked down instance etc.

We want to have authz checks on all controller actions so that we can lock down CKAN to a login-only mode.

Errors are caused when harvesting documents. This also makes the count not show up correctly on the ckan search page.

Add an api for searching users. This is needed for any kind of ajax autocomplete (needed for anywhere we want to add users).

• API location: /api/util/user/lookup?q=querystr&limit=10
• Return json objects containing {id: ..., name: ..., fullname: ...}
• Put in a module called controllers/apiv2/user.py

Go to ​http://ckan.net/user/register and fill in all the fields apart from password. On submit you get 500 error.

To counter Cross Site Request Forgery attacks, each form generated is assigned a random number in its url, which must be passed when you submit the form.

(Something to consider for the new form mechanism)

CKAN's harvesting facility is now live on DGU but there are some major improvements that could be made to make it more robust and better fit the generic CKAN harvesting framework proposed in #987.

Some of the key issues:

• Error reports do not currently contain the ID or title of the document with the error.
• We only have "added" and "error" logging on jobs when we really need a report of "added", "updated", "not changed" and "errors" with the items in each referencing a real metadata document for which harvesting was attempted
• We need deletion and editing of sources, without deleting the harvested documents or packages
• We need a more robust harvesting mechanism than a cron job or we need to deal with the case of multiple cron jobs running at once.
• We need to know the last time a list of documents was scheduled for harvest and the last time each one was fetched.

(instead of being hard coded)

It's not good to have ckanext doing lots of different things with different dependencies. Split it off into:

• ckanext-importlib

etc.

And then deprecate the ckanext repo itself.

The problem is that in ckan/controllers/rest.py the BaseApiController? has this method:

    def __before__(self, action, **env):
        BaseController.__before__(self, action, **env)
        if not self.authorizer.am_authorized(c, model.Action.SITE_READ, model.System):
            abort(401, _('Not authorized to see this page'))

which works on the basis of your c.user, rather than your apikey. All API users are treated as visitors (since API users don't get a login cookie) and even a sysadmin's apikey is blocked unless there is a right for a Visitor to SITE_READ.

Also needs tests.

(Also, why is this restriction only on the API, package search, group index and tags and agroup index? I'm guessing SITE_READ is only for places where other authz don't apply, but maybe it should not be called 'SITE_READ' but 'OTHER_READ' or something?)

The stages involved with doing this.

• Convert model objects to standard dict format (DONE)
• Convert standard dicts to current api formats (DONE)
• Make standard dicts savable (DONE)
• Validate standard dict format. (DONE)
• Authorize actions

• Deleting a group changes state to 'deleted' rather than purging it
• Adding authz tests for deleted groups

Refactor ckan/lib/authztool.py so that the relevant methods are independent of the command line interface.

The extracted methods should live in a new file ckan/authz.py. authztool.py should probably move into cli.py and will just do command line parsing and printing and use ckan/authz.py. The updated web gui for authz will also use this code.

Tests should be made. There's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests.

all to go on a branch feature-1050-refactor-authtoolz

Optional extras

• Rename ckan/authz.py to ckan/lib/authz.py or even ckan/logic/authz.py

No authz on:

• Group creation/edit/listing
• Package relationship create/edit/delete

Changing an old resource and creating a new resource on the end results in the old resource moving to the end in the ordering.

This breaks tests:

• (ckanext-dgu) ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadBasic.test_fields
• (ckanext-importlib) ckanext/importlib/tests/test_loader.py:TestLoaderInsertingResources.test_0_reload

You can make a ckan test break with this patch:

diff -r e6643cf1324c ckan/tests/models/test_resource.py
--- a/ckan/tests/models/test_resource.py        Wed Mar 23 13:25:52 2011 +0000
+++ b/ckan/tests/models/test_resource.py        Wed Mar 23 19:22:35 2011 +0000
@@ -297,6 +297,8 @@
                'url':self.urls[1], 'format':u'OTHER FORMAT',
                'description':self.description, 'hash':self.hash,
                'id':original_res_ids[2]},
+            { #new
+                'url':'new'},
            ]
        pkg.update_resources(res_dicts)
        model.repo.commit_and_remove()

There seems to be a problem with vdm creating a replacement Resource for the old resource - because it has a duplicate position it is put to the end by the SQLAlchemy ordering_list function.

Use case:

• Login using OpenID
• Click on 'My account' - results in 404

Solutions:

• User user.id instead of their name
• Escape the URL properly.

Examples which prompt annoying exception emails:

http://ckan.net/locale?locale=ja
Module ckan.i18n:21 in set_session_locale
           assert locale in _KNOWN_LOCALES

A bot has caused these:

http://ca.ckan.net/package/new?package_form=gov
Module ckan.forms.registry:32 in get_fieldset
               raise ValueError('Could not find package_form name %r in those found: \n%r' % (package_form, [en.name for en in entrypoints]))
ValueError: Could not find package_form name u'gov)' in those found: ['gov', 'standard', 'ca']

e.g. we just added notes_rendered and that is read in as an extra field. Tests failing in ckanext-importlib

Also related: we are missing lost metadata_created and metadata_modified in the dumps.

I've been asked if we can do something about the overflow of the Group name in the right hand column on this page:

​http://register.data.overheid.nl/package/europese-aanbestedingen

The reason is that the list display for groups is in the form "group_tltie (group_name)", and of course group_name can't have spaces and so can't wrap nicely.

I was wondering if there's a good reason why we don't only display group_title (if it exists) and group_name only when there's not a title?

Child tickets

• #1198 Publisher hierarchy
• #1050 Authz lib improvement and refactor of ckan/lib/authztool.py
• #1004 Group creation instructions missing
• #1099 Strange interactions between two browsers while playing with authz groups
• #1115 can have two authzgroups with the same name
• #1133 command line rights manipulation doesn't work
• #1138 minor navigations behave inconsistently

Old ticket description:

1. Change name of AuthzGroup? to UserGroup? to reflect what it is for

2. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
   • Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
   • Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

3. Change UserGroups? so that they can have a hierarchical structure,

More info on Hierarchy change

e.g. UserGroup? NHS contains the User nhsysadmin, as well as the UserGroups? SURREY and BERKS, which themselves contain users.

One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not.

Each Group has permissions over various objects.

A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively.

Algorithm:

possible(user, action, package):

if user has permission for action on package

or any of have that permission

or any of his groups group-children (but not user-children), and so on recursively have the permission.

Use 'db dump' and 'db load' for 'pg_dump' and 'psql -f' of a database. Use pylons config to find out database options.

See ​http://wiki.ckan.net/Domain_Model especially section on v2.

• New domain model is planned but not yet finally agreed.
• Layer architecture is complete and implemented

It takes several minutes to print the 'rights' on DGU, which is annoying when you only want to grep for a few lines. Much quicker than grepping is to filter in the query.

Currently repeat the same template and code across Package Authz, Group Authz, and Authz Group authz.

Having now implemented a new, cleaner setup in ckanext-admin we should port this back into core.

• Common template code (checkbox template)
• Logic code (or just common code) for wiring into authz system
• Look for all places thoroughout the system where usernames, authzgroups or groups need to be typed into boxes, and make sure that they auto-complete appropriately.

Will also deliver a significant improvement in the form of ajax user lookup.

Purging Revisions

• Delete button displayed on:
  • /revision/list
  • (/package/history)
    • /package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison.
  • /revision/{id}
• Delete button submits to delete action on revision and changes revision state to 'deleted'.
  • undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?)
• Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: "Empty trash" (irreversible). Click button purges all revisions with deleted state.

Purging Packages

• Put into deleted state.
• Listed on /ckan-admin/trash
• Separate Empty trash button which deletes all associated revisions.
  • Should be separate from Empty trash for revisions

Current system

• Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!)