{22} Trac tickets (2647 matches)

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

Solr is not working for moderation, pending changes are not being indexed.

A bad action name: e.g. curl ​http://test.ckan.net/api/action/user_showaa -d '{}' causes 500 response.

Occurs only on 1.4.3 beta.

We get exceptions on these occasions:

• user/edit if not logged in
• user/edit/bob if bob is not a known user

These are a problem only on 1.4.3b.

Introduced in #1229 - merge of branch feature-1229-db-out-of-controllers

Occasionally see this problem when producing an error page. Here's an example

WebApp Error: <type 'exceptions.TypeError'>: You cannot set the body to a unicode value without a charset
URL: http://at.ckan.net/authorizationgroup/new
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:117 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:284 in __call__
<<              if log_debug:
                       log.debug("Calling Response object to return WSGI data")
                   return response(environ, self.start_response)
               
               if log_debug:
>>  return response(environ, self.start_response)
Module webob.exc:248 in __call__
<<              return []
               if not self.body and not self.empty_body:
                   return self.generate_response(environ, start_response)
               return Response.__call__(self, environ, start_response)
>>  return self.generate_response(environ, start_response)
Module webob.exc:239 in generate_response
<<              status=self.status,
                   headerlist=headerlist,
                   content_type=content_type
               )
               return resp(environ, start_response)
>>  content_type=content_type
Module webob.response:94 in __init__
<<                  if charset is None:
                           raise TypeError(
                               "You cannot set the body to a unicode value without a charset")
                       body = body.encode(charset)
                   self._body = body
>>  "You cannot set the body to a unicode value without a charset")
TypeError: You cannot set the body to a unicode value without a charset

This overlaps with Sean's follow support (for activity streams) and so will be dealt with there.

CORS - ​http://www.w3.org/TR/cors/ - support.

This is what you do in Apache. Should do this in lib/base.py or similar.

    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Methods "POST, PUT, GET, OPTIONS"
    Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type"

    # Respond to all OPTIONS requests with 200 OK
    # This could be done in the webapp
    # This is need for pre-flighted requests (POSTs/PUTs)
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} OPTIONS
    RewriteRule ^(.*)$ $1 [R=200,L]

Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend.

For example:

• name and title should have a higher weight than the notes field

Authorization Groups are going to be deprecated in 1.8

It is still unclear how Authorization Groups fit in the new authz model, so they have been left out of the logic and auth refactoring (Tickets #1229 and #1253)

When you change the locale, it displays a flash message "Language has been set to: English" in the original language, not the one we have just changed to.

A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else.

(But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.)

Same in the Web interface and API.

This ticket ties in with #948 highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can #1076.

Abstract

Package Relationships have not taken off in the 18 months we've had them in the API. There are some issues with them and we need to spend more time improving them or consider getting rid of them.

The Problem

Original use cases are expressed here: #253 Here are comments about how we could handle these specific examples better:

1. groups of packages - maybe better with a custom tag?
2. fragment resources - soon to be covered by 'kind' resource field #957

3&5. derived resource - better to have some sort of resource relationship perhaps?

4. linked resource - again better to have some sort of resource relationship perhaps?

Outstanding issues needing serious effort to fix:

• #256 Editing them in Web UI (not done yet)
• #1288 Package edit/creation can't include 'relationships' field

Specification

Remove relationships from model, API, tests, Web UI. Data migration to remove from db.

Why do it this way

Getting frustrated having problems with the code, when it's not used much. Often asked about what it's for, but rarely used. Seems an overly complicated design.

Backwards Compatibility

n/a

Implementation plan

Deliverables

See Specification

Risks and mitigations

Risk: a customer suddenly wants this, and the new ways to relate resources are not in place yet.

Mitigation: discuss this decision thoroughly to make sure we are confident the use cases are not important. Discuss with team, ckan-discuss and specifically the LOD people who have some related packages on thedatahub.org.

Participants

David Read

Progress

Not yet.

Add tables described in ​http://ckan.okfnpad.org/notifications

Changes

• All templates
• [DONE] Routing: /package/ -> /dataset/ (including API)
• [DONE] Tests: self.app.get('/dataset/...') rather than package ...
  • One test to check redirect works?
• Docs

Leave

• Stay with Package in code e.g. stay with 'PackageController?'
  • also leave stuff like pkg = .... (gradual migration)
• Leave extensions

Notes

• Package -> Dataset
• Data Package -> Dataset
• respect capitalization

Sub ticket of #1294

• Initial screen just title, name, description (notes), license (and tags?).
• State that you can add data and more information on next screen. This takes you to Package 'view' where you can then do more stuff.
  • Flash some instructions? e.g. that resources are missing?
• Name slug should by default look uneditable (cf wordpress way to do the slug)

Subticket of #1294.

Can integrate from working system on ckanjs.

Migrate tables for activity streams

This was removed during the auth refactoring, but it's needed in the old controllers

Only happens if the user does not update the password. Most probably introduced during #1229.

The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one.

Could not send reset link: SMTPRecipientsRefused({u'amercadero@…': (550, 'relay not permitted')},)

As I said, adria.mercader@… works fine. The SMTP server used mail.okfn.org

• Major issue with spam signup on thedatahub.org
• Suggest using recaptcha

Est cost: 1d

Registering a new user directs you to that user's profile page but does not log you in. There is an empty set of "Recent changes" and no ability to modify the page. Users must follow the login link in the top-right corner to continue.

Registering a new user should immediately log you in.

From Florian:

I am having trouble with the characters "=&;".
if they occur e.g. in the title of a package, I get a 400-Bad-Request
when I try to register a package by POSTing to ckan/rest/api/package

However, when I PUT the same package as an edit to
ckan/rest/api/package/existing_package, it works and the title shows
correctly on the front end and API.

Reproduced

(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{"name": "test3", "title": "Test &"}' -H "Authorization:tester" 
HTTP/1.0 400 Bad Request
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 14:09:31 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Length: 87

"Bad request - JSON Error: Unterminated string starting at: line 1 column 27 (char 27)"

We need some documentation for the /api/action. Even though it's beta and subject to change it would be good to have a guide.

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

It is very annoying that you have to wrap links in <...> and most people don't bother. Extend h.markdown in code and markdown in js to fix this.

(We should also support 'package' links etc in javascript?)

*ADDITIONAL*

• Note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
• Move the markdown api to /util/markdown

on thedatahub.org

We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in:

• 'with' statement requires from __future__ import with_statement
• import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5.

Just a tip for developers regarding translations:

• When writing templates, mark <p>, <div>, etc elements with the

i18:msg attribute, otherwise the messages don't get extracted properly.

Good: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40

<p i18n:msg="">More about CKAN, and the background to its creation can
   be found in <a
     href="http://m.okfn.org/files/talks/opentech_20080705/">these
     slides</a>.</p>

gives this: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

<p class="hints">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href="http://opendatacommons.org/licenses/odbl/1.0/">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>

gives this: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the grammar or syntax of the translated language.

To reproduce:

1. Create a package with two tags the same: "bulk bulk"
2. Click 'save'
3. 500 ERROR - 'Server Error'

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

Provide an API to the logic layer at /api/action.

Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'

e.g. ​http://ckan.net/group/edit/history

Module ckan.controllers.group:135 in edit
<<          try:
                   old_data = get_action('group_show')(context, data_dict)
                   c.grouptitle = old_data.get('title')
                   c.groupname = old_data.get('name')
>>  old_data = get_action('group_show')(context, data_dict)
Module ckan.logic.action.get:347 in group_show
<<      if group is None:
               raise NotFound
       
           check_access('group_show',context, data_dict)
>>  raise NotFound

Json decoding error on search, due to date in resources.

This stems from a feature to allow you to prefill fields in the authorizationgroup creation form.

e.g. ​http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it.

Tweaking UIs and polishing the work done in #1294.

## Edit Page ##

• [Done; cset:49e0b059b6f5] Hide upload if storage not enabled
• [Done; cset:81ff13b42dad] Make 'Add resource' links on edit page clearer and more button-like.
• [Done; cset:03904fc50cbf] Move edit form tabs to the left
• [Done; cset:fde26b87d6ce] API placeholder text should not link to a CSV file
• [Done] We used to autocomplete the format field - restore this functionality
  • Works on live. My local db has an empty 'resource_revision' table; maybe 'paster create-test-data' should insert something here?
• [Done; cset:1aab499d73db] Summary should contain 'name' (or, if unavailable, use description. Long term: description is to be deprecated)
• [Done] Remove/move instructions where possible (a lot could just be placeholder text on the elements) (other stuff could only show on hover)
  • NB: reveal_instructions.js already disabled on edit page
  • [RP] have now done this on create -- think we can leave on edit
• [Done; cset:755b1a622199] We have broken 'error' reporting for resource issues. Either need to move that into error summary or somehow not destroy it when we rewrite table on page load ...
• [Done; cset:7ad090cb7855] Add slideDown to Resource expanded view
• [Done; cset:d3595143e506] Need ability to delete resources
  • No thought yet given to uploaded resources. Long term re-think probable.
• [Done; cset:2430a68a59fc] Empty resource table to resemble view page ("no resources" text)

## General ##

• [Done; cset:cc0ad47293e5] Highlight the package tab I'm on (view/edit/history)
  • More generally, the presence of the #minornavigation bar is very inconsistent throughout the site.
• [Done; cset:d5babc3629cf] Remove 'license' filtered search from backend
• [Done] Move script stuff to footer of page

New js code running produced in the work following #1294 contains various string literals, and needs proper internationalisation.

See files:

• application.js
• templates.js

Furthermore, efforts can be made to tidy up strings as described in #1329.

I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org.

If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).

Major UX grip and incredibly simple to do. May also want to have basic autocomplete support within the list (trivial to do as well I believe).

(Talk to RP about this re how we define lists of licenses).

Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install).

Propose:

• ckan.simple_search config option
• If set:
  • query via simple query to database backend
  • do no specialized indexing

Extras

Remove

• TagSearchQuery? from lib/search (just do a search directly)?
• #1360: filter by downloadable and filter by open search options

Possible Extras

(Probably future improvements)

• Re-introduce full-text search indexing where supported in e.g. postgres and use for querying

table: task_status

id
entity_id
entity_type
task_type
key
value
last_updated
state

unique(entity_id, task_type, key)

example keys:
last_task_id

Lots of things rely on the old (i.e. pre-SOLR) search param formats, including ckanclient / DGU. Need a translation layer to convert these params (in api v1&2) to SOLR syntax.

This functionality was broken by the changes in #1277 which went into 1.5a

From Ira: PDF of bugs and suggestions is attached.

This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in #1404.

DONE ?

• [dataset/edit]: Deletions are confirmed with a dialog box
• [dataset/edit]: Long URLs break the edit resources tab. Show name (& 'no name') with a pencil icon.
• [dataset/edit]: Rearrange the delete resources mechanism;
  • Use a little minus icon on the far left, followed by a pencil icon for editing.
  • The 'delete resource' button can become 'hide'
  • Deletions are confirmed with a dialog box
• [dataset/edit]: Rearrange the 'you have unsaved changes' warning;
  • Delete the little green plus sign on the right;
  • Try a flash banner instead (more general purpose & beautiful)
• [dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset.
• [dataset/view]: Subscribe button (& subscribe to history) doesn't work as expected. Modal dialog carrying instructions?
• [dataset/view]: Drop the 'add to a group>>' and 'add tag>>'. Needs inline tag-adder.
• [markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out.
• [dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels.
• [groups/add]: Hide unimportant fields
• [navigation] Add 3D bevel (@see github) to current tab highlight.
• [dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see ​http://groupspaces.com/a/group/
• [groups/view]: Show description and owner. To match dataset section.
• [groups/authz]: Tidy to match admin/authz (pretty buttons & spacing). Add navigation bar.
• [groups]: Entire section to use pretty buttons please.
• [about]: Get the about page working on test.ckan.org. (Works locally and on live).

WONTFIX ?

• [dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold. (wontfix)
• [groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'. (ticket #1403)
• ([groups]: (Investigate) Is it easy to implement a little search-groups feature?) (ticket #1403)
• [*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...) (wontfix: we need to rethink this mechanism; authorization refactor is coming)
• [general]: 'Authorization' refactor to 'Permissions' on nav-bars. (wontfix, further discussion. authorization might be less techy...)
• [search]: (Investigate) Can we quickly & easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets? (wontfix: high cost and maintenance overhead; need compelling user story)

Add a way to share datasets and see mentions of datasets on twitter (and elsewhere)

As a visitor I want to share a link to a dataset I have found. I also want to see how many others have shared this (or mentioned it).

• Sharing platform: simplest option is probably twitter but could generalize to e.g. sharethis system which supports, facebook, twitter etc etc.
  • Will need a config option for relevant API key e.g. ckan.sharethis.apikey
• Location: Put this at top of sidebar on dataset view (a better suggsetion?)
• Should show how many mentions / shares there are. (very important!)

Discussion

• (Tom and Ira and ...): do we really want this, versus e.g. proper follow extension?
• What about bringing conversation back in to CKAN. E.g. show all times this dataset (i.e. its url) was mentioned on twitter. If we do this I think this should be definite +1.
• Also, given our users, I think just doing twitter (identica) may be sufficient (how many people want to share links to datasets on facebook?)

Take the resource archiver (currently part of ckanext-qa) and make it into a standalone extension that works as a celery task.

e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English.

Regression found by updating datacatalogs.