{22} Trac tickets (2647 matches)

To reproduce (from Jonathan Gray):

• I go to add an item to the group
• As I type, I get autocomplete options (cool!)
• When I click on one, it adds this, and copies the same autocomplete option in the box
• When I click submit, it seems to add the item twice
• On the main group page I think it just shows up once, but when I click edit it shows two identical boxes, both ticked - and the count includes two copies of the same item

Basically the problem is allowing the same package associated multiple times with a group.

This is an issue on CKAN 1.5b and probably earlier.

• Review process of releases and updating strings (pot) and translations (po) files
  • c.f. ​http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html
• Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?)
• Review long list of strings for improvement
  • Consolidate any?
  • Get rid of i18n for strings in api
  • Try and remove javascript
  • Add in extensions
• #1374 Fix switching to en from other default language
• #1417 Fix browser language detection

Child tickets:

• #1107 Move package autocomplete from package controller and move to API
• #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

• ​http://wiki.apache.org/solr/SolJSON
• ​http://www.flickr.com/services/api/response.json.html

Give site user system administrative rights.

During registering a user, the user inputs foreign chars into the captcha field.

URL: http://thedatahub.org/user/register
...
Module ckan.lib.captcha:22 in check_recaptcha
<<                                     remoteip=client_ip_address,
                                          challenge=recaptcha_challenge_field,
                                          response=recaptcha_response_field))
           f = urllib2.urlopen(recaptcha_server_name, params)
           data = f.read()
>>  response=recaptcha_response_field))
Module urllib:1267 in urlencode
<<          for k, v in query:
                   k = quote_plus(str(k))
                   v = quote_plus(str(v))
                   l.append(k + '=' + v)
           else:
>>  v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 0: ordinal not in range(128)

Lots of things rely on the old (i.e. pre-SOLR) search param formats, including ckanclient / DGU. Need a translation layer to convert these params (in api v1&2) to SOLR syntax.

This functionality was broken by the changes in #1277 which went into 1.5a

In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory.

Need to ensure config options in the deployment.ini_tmpl match up with doc/configuration.rst. Remove any unused ones too.

Exception received on test.ckan.net at 10.12 on 29/7/11

http://test.ckan.net/package?__cache=2759373&page=4
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:153 in search
<<                          }
       
                   query = get.package_search(context,data_dict)
       
                   c.page = h.Page(
>>  query = get.package_search(context,data_dict)
Module ckan.logic.action.get:514 in package_search
<<      for package in query.results:
               result_dict = table_dictize(package, context)
               result_dict = _extend_package_dict(result_dict,context)
       
               results.append(result_dict)
>>  result_dict = _extend_package_dict(result_dict,context)
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: None'

and a similar one:

 http://test.ckan.net/tag/barbecue
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.tag:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__("logging").getLogger("proxy_cache")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
       
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.tag:64 in read
<<          data_dict = {'id':id}
               try:
                   c.tag = get.tag_show(context,data_dict)
               except NotFound:
                   abort(404, _('Tag not found'))
>>  c.tag = get.tag_show(context,data_dict)
Module ckan.logic.action.get:341 in tag_show
<<      extended_packages = []
           for package in tag_dict['packages']:
               extended_packages.append(_extend_package_dict(package,context))
       
           tag_dict['packages'] = extended_packages
>>  extended_packages.append(_extend_package_dict(package,context))
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: CreativeCommonsAttributionCCBY25'

I couldn't log into theDataHub with OpenID today. I tried both Google ID and MyOpenID. Both times the login on the remote auth server went fine, but when it returns you to theDataHub you get error "Login failed. Bad username or password."

• perhaps we should show the openid as well to distinguish between users with the same name.

• perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name.

• also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it.

Validation error is incorrect arguments, so should be a "400 Bad Request" response. "409 Conflict" is for when there is a conflict creating two resources with the same identifier. Yet we get 409 for all validation errors.

e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English.

Progress can be tracked in this document:

​https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdFQzRnVoUHV6ckoyblI5TUdEcnJLVlE&hl=en_US#gid=0

home page doesn't have c.facets setup.

Only affects 1.5 branch - ok 1.4.3.1. Was broken in cset:05f60f2d26e8.

e.g. cz.ckan.net has lang=cz but when you click on Home page link "English" it flashes message "Language has been set to: English" but the rest of the page is still Czech.

Format should be lowercase. Automatically lower case (for extra points have a bit of javascript to force lower case when entering).

• Even more points: do a update on thedatahub repo to make all format lower case (or script this as an update?)

From Ira: PDF of bugs and suggestions is attached.

This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in #1404.

DONE ?

• [dataset/edit]: Deletions are confirmed with a dialog box
• [dataset/edit]: Long URLs break the edit resources tab. Show name (& 'no name') with a pencil icon.
• [dataset/edit]: Rearrange the delete resources mechanism;
  • Use a little minus icon on the far left, followed by a pencil icon for editing.
  • The 'delete resource' button can become 'hide'
  • Deletions are confirmed with a dialog box
• [dataset/edit]: Rearrange the 'you have unsaved changes' warning;
  • Delete the little green plus sign on the right;
  • Try a flash banner instead (more general purpose & beautiful)
• [dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset.
• [dataset/view]: Subscribe button (& subscribe to history) doesn't work as expected. Modal dialog carrying instructions?
• [dataset/view]: Drop the 'add to a group>>' and 'add tag>>'. Needs inline tag-adder.
• [markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out.
• [dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels.
• [groups/add]: Hide unimportant fields
• [navigation] Add 3D bevel (@see github) to current tab highlight.
• [dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see ​http://groupspaces.com/a/group/
• [groups/view]: Show description and owner. To match dataset section.
• [groups/authz]: Tidy to match admin/authz (pretty buttons & spacing). Add navigation bar.
• [groups]: Entire section to use pretty buttons please.
• [about]: Get the about page working on test.ckan.org. (Works locally and on live).

WONTFIX ?

• [dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold. (wontfix)
• [groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'. (ticket #1403)
• ([groups]: (Investigate) Is it easy to implement a little search-groups feature?) (ticket #1403)
• [*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...) (wontfix: we need to rethink this mechanism; authorization refactor is coming)
• [general]: 'Authorization' refactor to 'Permissions' on nav-bars. (wontfix, further discussion. authorization might be less techy...)
• [search]: (Investigate) Can we quickly & easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets? (wontfix: high cost and maintenance overhead; need compelling user story)

The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows:

• routing_key: Package
• operation: daily
• payload: pkg.as_dict()

Where operation is one of daily, weekly, monthly or any other interval.

Most articles still refer and link to ckan.net, wiki.ckan.net and to packages

Regression found by updating datacatalogs.

If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).

When creating a package, the package_create logic function sends the newly created package object to all plugins implementing the IPackageController interface

​https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64

At this point, the changes have not yet been commited and the id of the package is not set (It has a default value of make_uuid, which is set when commited). For this reason, plugins have no way of knowing the package id. E.g. the spatial query plugin needs to know this id to link the package extent to the actual package.

One possible solution would be to set the package id in the data dict received by package_create (if it's not defined):

from ckan.model.types import make_uuid
if not 'id' in data or not data['id']:
   data['id'] = make_uuid()

table: task_status

id
entity_id
entity_type
task_type
key
value
last_updated
state

unique(entity_id, task_type, key)

example keys:
last_task_id

If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting ​http://localhost:5000/dataset).

Steps to reproduce:

• Change the solr_url parameter in the ini file to something non-existant
• Visit ​http://localhost:5000

Integrating ckanext-admin into core has thrown up a number of problems:

• Look & feel does not match the rest of the site
• Tests are not passing
• On the trash page, clicking 'undelete' triggers a purge
• Using the purge functionality is dangerous; deleting and purging the latest revision will corrupt a dataset (& several corrupt datasets have been found on thedatahub.org)
• Trash page can contain nested form tags in certain cases (breaking test harness & form redirection)

(2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option).

Cost: 1h

Need to be able to build a distribution. Need:

• new 'build' command
• specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution.

It is very annoying that you have to wrap links in <...> and most people don't bother. Extend h.markdown in code and markdown in js to fix this.

(We should also support 'package' links etc in javascript?)

*ADDITIONAL*

• Note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
• Move the markdown api to /util/markdown

Deprecate ckan.net in favour of thedatahub.org

• Announce
• Create thedatahub.org and have it run same data as ckan.net
• Announce closing down of ckan.net with 302
• Implement 302 permanent

Also redirect wiki.ckan.net to wiki.ckan.org

New js code running produced in the work following #1294 contains various string literals, and needs proper internationalisation.

See files:

• application.js
• templates.js

Furthermore, efforts can be made to tidy up strings as described in #1329.

Tweaking UIs and polishing the work done in #1294.

## Edit Page ##

• [Done; cset:49e0b059b6f5] Hide upload if storage not enabled
• [Done; cset:81ff13b42dad] Make 'Add resource' links on edit page clearer and more button-like.
• [Done; cset:03904fc50cbf] Move edit form tabs to the left
• [Done; cset:fde26b87d6ce] API placeholder text should not link to a CSV file
• [Done] We used to autocomplete the format field - restore this functionality
  • Works on live. My local db has an empty 'resource_revision' table; maybe 'paster create-test-data' should insert something here?
• [Done; cset:1aab499d73db] Summary should contain 'name' (or, if unavailable, use description. Long term: description is to be deprecated)
• [Done] Remove/move instructions where possible (a lot could just be placeholder text on the elements) (other stuff could only show on hover)
  • NB: reveal_instructions.js already disabled on edit page
  • [RP] have now done this on create -- think we can leave on edit
• [Done; cset:755b1a622199] We have broken 'error' reporting for resource issues. Either need to move that into error summary or somehow not destroy it when we rewrite table on page load ...
• [Done; cset:7ad090cb7855] Add slideDown to Resource expanded view
• [Done; cset:d3595143e506] Need ability to delete resources
  • No thought yet given to uploaded resources. Long term re-think probable.
• [Done; cset:2430a68a59fc] Empty resource table to resemble view page ("no resources" text)

## General ##

• [Done; cset:cc0ad47293e5] Highlight the package tab I'm on (view/edit/history)
  • More generally, the presence of the #minornavigation bar is very inconsistent throughout the site.
• [Done; cset:d5babc3629cf] Remove 'license' filtered search from backend
• [Done] Move script stuff to footer of page

Subticket of #1294.

Can integrate from working system on ckanjs.

• Major issue with spam signup on thedatahub.org
• Suggest using recaptcha

Est cost: 1d

Just a tip for developers regarding translations:

• When writing templates, mark <p>, <div>, etc elements with the

i18:msg attribute, otherwise the messages don't get extracted properly.

Good: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40

<p i18n:msg="">More about CKAN, and the background to its creation can
   be found in <a
     href="http://m.okfn.org/files/talks/opentech_20080705/">these
     slides</a>.</p>

gives this: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

<p class="hints">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href="http://opendatacommons.org/licenses/odbl/1.0/">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>

gives this: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the grammar or syntax of the translated language.

Consolidate the extra spatial as the field for providing information about the geographic extent of the package. This will integrate automatically with the spatial search and give developers a single point to develop geospatial features.

Abstract

Package Relationships have not taken off in the 18 months we've had them in the API. There are some issues with them and we need to spend more time improving them or consider getting rid of them.

The Problem

Original use cases are expressed here: #253 Here are comments about how we could handle these specific examples better:

1. groups of packages - maybe better with a custom tag?
2. fragment resources - soon to be covered by 'kind' resource field #957

3&5. derived resource - better to have some sort of resource relationship perhaps?

4. linked resource - again better to have some sort of resource relationship perhaps?

Outstanding issues needing serious effort to fix:

• #256 Editing them in Web UI (not done yet)
• #1288 Package edit/creation can't include 'relationships' field

Specification

Remove relationships from model, API, tests, Web UI. Data migration to remove from db.

Why do it this way

Getting frustrated having problems with the code, when it's not used much. Often asked about what it's for, but rarely used. Seems an overly complicated design.

Backwards Compatibility

n/a

Implementation plan

Deliverables

See Specification

Risks and mitigations

Risk: a customer suddenly wants this, and the new ways to relate resources are not in place yet.

Mitigation: discuss this decision thoroughly to make sure we are confident the use cases are not important. Discuss with team, ckan-discuss and specifically the LOD people who have some related packages on thedatahub.org.

Participants

David Read

Progress

Not yet.

Json decoding error on search, due to date in resources.

Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'

Awesome to have the captcha implementation and running on thedatahub.org! Here are some further suggestions concerning the config options, before this goes into a release:

• Document "ckan.recaptcha.publickey" and "ckan.recaptcha.privatekey" in doc/configuration.rst
• Which one config option defines whether recaptcha is enabled? There is disagreement between the template and the lib. Maybe better to have a third option for this. When setting the syntax for this, I can see us wanting to use captcha for other operations, like package edit, package create, group create etc. so perhaps we'd want to be able to turn each one on/off separately, using the name of the logic action. e.g. ckan.recaptcha.enable = user_create, group_create
• We got this exception last night from the template:

  Module ?:37 in <Expression u'g.recaptcha_publickey'>
  <<          <dd><input type="password" name="password2" value="" /></dd>
         
                 <dd py:if="g.recaptcha_publickey">
                   <script type="text/javascript"
                     src="http://www.google.com/recaptcha/api/challenge?k=${g.recaptcha_publickey}">
  >>  <dd py:if="g.recaptcha_publickey">
  Module genshi.template.eval:324 in lookup_attr
  <<                      val = obj[key]
                         except (KeyError, TypeError):
                             val = cls.undefined(key, owner=obj)
                 return val
  >>  val = cls.undefined(key, owner=obj)
  Module genshi.template.eval:410 in undefined
  <<          """Raise an ``UndefinedError`` immediately."""
                 __traceback_hide__ = True
                 raise UndefinedError(key, owner=owner)
  >>  raise UndefinedError(key, owner=owner)
  UndefinedError: <ckan.lib.app_globals.Globals object at 0xb8f1182c> has no member named "recaptcha_publickey"
  

I'm not sure why - might be something to do with the maintenance on the server, but worth double checking, since we don't want exceptions, say if one or other config variable is not defined.

1. Register a new user (/user/register)
2. It redirects you to user page and displays "Welcome back" even though you are a new user.

For Authorization Groups, if you have admin privileges you see view, edit and authz tabs, and if you don't have the necessary privileges you only see the view tab.

For Packages, you see all tabs whatever your permissions, so there's a link you can click on which will redirect you to the login page.

Instead of this:

extras: {
type: ""Primary source""
license_id: ""localauth-withrights""
publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""
}

we should return this:

extras: {
type: "Primary source",
license_id: "localauth-withrights",
publisher_frequency: "2Preño € $ àdrià",
}

Exception caused by bots POSTing 'new group' form, but with bad (old-style) parameters.

<type 'exceptions.ValueError'>: invalid literal for int() with base 10: 'cache'
...
Module ckan.controllers.group:102 in new
<<          if context['save'] and not data:
                   return self._save_new(context)
               
               data = data or {}
>>  return self._save_new(context)
Module ckan.controllers.group:152 in _save_new
<<          try:
                   data_dict = clean_dict(unflatten(
                       tuplize_dict(parse_params(request.params))))
                   context['message'] = data_dict.get('log_message', '')
                   group = create.group_create(data_dict, context)
>>  tuplize_dict(parse_params(request.params))))
Module ckan.logic:57 in tuplize_dict
<<          for num, key in enumerate(key_list):
                   if num % 2 == 1:
                       key_list[num] = int(key)
               tuplized_dict[tuple(key_list)] = value
           return tuplized_dict
>>  key_list[num] = int(key)
ValueError: invalid literal for int() with base 10: 'cache'

I ran into a bug with the size field on resources.

• It would not accept an empty value from form (IMO this clearly equates to null/None)
• This could be fixed via using ignore_empty instead of ignore_missing
• However using this means there was no way to empty the field (e.g. i may just want to set the size field back to null not just change to another value)
• similar issues could arise around other fields (such as last_modified ...)
  • cf cset:645031d07b60

To solve this (cset:58acdcfe6d4e) i created an int_converter temporarily in logic/schema.py (this is almost certainly the wrong place). But I think it raises a bigger issue about the conversion layer and how it works.

e.g. ​http://ckan.net/group/edit/history

Module ckan.controllers.group:135 in edit
<<          try:
                   old_data = get_action('group_show')(context, data_dict)
                   c.grouptitle = old_data.get('title')
                   c.groupname = old_data.get('name')
>>  old_data = get_action('group_show')(context, data_dict)
Module ckan.logic.action.get:347 in group_show
<<      if group is None:
               raise NotFound
       
           check_access('group_show',context, data_dict)
>>  raise NotFound

Sub-tasks / tickets:

• #1335 Action API - access to any logic layer function
• Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
• Add package_create_validate & package_update_validate logic functions
• Allow extensions to provide logic layer functions (IActions)