CKAN: Ticket Query

#1334: "AttributeError: 'NoneType' object has no attribute 'name'" exception when logging in

amercader — Tue, 13 Sep 2011 10:25:08 GMT

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'

#1318: 'Register user' doesn't log you in

dread — Wed, 07 Sep 2011 17:16:23 GMT

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

#1346: 'get_authorization_group_dict not found' exception

dread — Thu, 15 Sep 2011 10:19:34 GMT

This stems from a feature to allow you to prefill fields in the authorizationgroup creation form.

e.g. http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it.

#1253: Authz refactor

amercader — Mon, 01 Aug 2011 11:51:31 GMT

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

#816: Autocomplete for the resource format field

cygri — Wed, 17 Nov 2010 12:22:54 GMT

The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource.

This would help improve data quality and consistency.

#1258: Bring purge revision into ckan repo from vdm

kindly — Tue, 02 Aug 2011 12:52:19 GMT

In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory.

#1271: CORS support

rgrp — Sat, 13 Aug 2011 13:23:59 GMT

CORS - http://www.w3.org/TR/cors/ - support.

This is what you do in Apache. Should do this in lib/base.py or similar.

    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Methods "POST, PUT, GET, OPTIONS"
    Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type"
    # Respond to all OPTIONS requests with 200 OK
    # This could be done in the webapp
    # This is need for pre-flighted requests (POSTs/PUTs)
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} OPTIONS
    RewriteRule ^(.*)$ $1 [R=200,L]

#1218: Colour the History tab icon

minspamboks@… — Mon, 11 Jul 2011 09:16:08 GMT

Change the color of the "History" tab icon to yellowish, like the rest of the icons in the other tabs ("View" and "Edit", "Authorization").

Reasoning

When you view a data package, for instance http://ckan.net/package/thesaurus-w, you will see "View", "Edit", "History" tabs on the top. "History" tab has a black-and-white icon which makes it look like an inactive/disabled tab (since the text is also grayed out when the tab is not selected). This is not a major issue, but it is a little bit confusing for the users. This icon exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net).

The simple solution would be to change the color of the "History" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. "View" and "Edit").

#1192: Convert CKAN Sphinx docs into admin/reference manual

annapowellsmith — Fri, 17 Jun 2011 15:29:53 GMT

As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install.
A Reference Manual which is the primary source of reference for CKAN software - this includes API docs.

The current chapters of the Sphinx docs should be moved as follows:

index.rst - copy some stuff from README.txt, keep short
README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst
[NEW] install.rst - new file on installation
[NEW] theming.rst - move over from wiki
api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
design.rst - REMOVE (can copy some over if you can be bothered to http://wiki.ckan.net/Vision (all philosophical stuff should go!))
loaders.rst: move to wiki.ckan.net/Using_Loaders
feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
importer.rst: remove (don't even copy)
forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) http://wiki.ckan.net/Customizing_Forms
forms-integration.rst: remove
model.rst: leave but move to reference (only for core devs)
load-testing.rst -> move to http://wiki.ckan.net/Load_Testing
distributed.rst -> remove
admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
deb.html -> go into reference (query james gardner on list about moving this to wiki?)

#1270: Default site www.ckan.net is a redirect and loses POSTs

dread — Tue, 09 Aug 2011 17:44:07 GMT

The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters:

In [1]: import ckanclient
In [2]: c = ckanclient.CkanClient()
In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api')
In [4]: c.package_search('', search_options={'groups':'openspending'})
Out[4]:
{u'count': 2102,
 u'results': <generator object _result_generator at 0x10168b1e0>}
In [5]: c2.package_search('', search_options={'groups':'openspending'})
Out[5]: {u'count': 29, u'results': <generator object _result_generator at 0x10168b410>}

Thanks to borior for finding this and raising it.

#1228: Display OpenID login errors

dread — Tue, 19 Jul 2011 14:42:42 GMT

When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again.

#1183: Downloads "Preview" button doesn't preview.

nickstenning — Fri, 10 Jun 2011 10:27:49 GMT

The "Preview" button is a nice idea, but it doesn't seem to actually "preview" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file.

This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of application/x-some-junk-here in order to force a download.

#1340: Editing non-existent group causes Exception

dread — Wed, 14 Sep 2011 09:09:34 GMT

e.g. http://ckan.net/group/edit/history

Module ckan.controllers.group:135 in edit
<<          try:
                   old_data = get_action('group_show')(context, data_dict)
                   c.grouptitle = old_data.get('title')
                   c.groupname = old_data.get('name')
>>  old_data = get_action('group_show')(context, data_dict)
Module ckan.logic.action.get:347 in group_show
<<      if group is None:
               raise NotFound
           check_access('group_show',context, data_dict)
>>  raise NotFound

#1347: Exception caused by New Group form POST with bad formed parameter

dread — Thu, 15 Sep 2011 13:47:46 GMT

Exception caused by bots POSTing 'new group' form, but with bad (old-style) parameters.

<type 'exceptions.ValueError'>: invalid literal for int() with base 10: 'cache'
...
Module ckan.controllers.group:102 in new
<<          if context['save'] and not data:
                   return self._save_new(context)
               data = data or {}
>>  return self._save_new(context)
Module ckan.controllers.group:152 in _save_new
<<          try:
                   data_dict = clean_dict(unflatten(
                       tuplize_dict(parse_params(request.params))))
                   context['message'] = data_dict.get('log_message', '')
                   group = create.group_create(data_dict, context)
>>  tuplize_dict(parse_params(request.params))))
Module ckan.logic:57 in tuplize_dict
<<          for num, key in enumerate(key_list):
                   if num % 2 == 1:
                       key_list[num] = int(key)
               tuplized_dict[tuple(key_list)] = value
           return tuplized_dict
>>  key_list[num] = int(key)
ValueError: invalid literal for int() with base 10: 'cache'

#1338: Exception when trying to use a custom form schema from an extension

amercader — Tue, 13 Sep 2011 14:46:57 GMT

Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'

#78: Extend login cookie lifetime

rgrp — Fri, 17 Jul 2009 11:15:10 GMT

When you login you get given a cookie but it is very short (life of your browser session I think).

Work out how to extend to something reasonable (30 days or perhaps forever with logout unsetting cookie).

A quick investigation to see whether this is possible
If possible do it

Cost: ?

#1254: Feature sheet

dread — Mon, 01 Aug 2011 15:18:45 GMT

Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers.

#1323: GET /authorizationgroup causes Exception when logged in

dread — Thu, 08 Sep 2011 09:54:45 GMT

on thedatahub.org

#1337: Group extras are JSON endoded in the REST API responses

amercader — Tue, 13 Sep 2011 14:38:47 GMT

Instead of this:

extras: {
type: ""Primary source""
license_id: ""localauth-withrights""
publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""
}

we should return this:

extras: {
type: "Primary source",
license_id: "localauth-withrights",
publisher_frequency: "2Preño € $ àdrià",
}

#888: Improvements to the dataproxy and the data API

rgrp — Wed, 29 Dec 2010 19:09:43 GMT

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

Testing for dataproxy
- Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
Google docs style row/column selections
Use the swiss library - https://bitbucket.org/okfn/swiss
- Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
Handle redirects for content-length?
Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}

#318: Insufficient validation of resource URIs

wwaites — Wed, 19 May 2010 20:58:43 GMT

The CKAN instance on data.gov.uk serves invalid URIs out of its API.

For example the following can be found,

http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls]

In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396

Also URIs are not stripped of whitespace at the end.

This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs.

"Be liberal in what you accept and conservative in what you send" would seem apt.

Actions

Validation of urls as part of form entry or data loading
- Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal
This code should support analysis of existing data so we can go through existing database and find invalid urls
- Also useful to have this so we can do out of band validation

#1281: Locale change flash message in wrong language

pudo — Mon, 22 Aug 2011 14:00:56 GMT

When you change the locale, it displays a flash message "Language has been set to: English" in the original language, not the one we have just changed to.

#1222: Mailer has no tests

dread — Tue, 12 Jul 2011 13:58:56 GMT

#1154: Make ckan robust against solr failure

nils.toedtmann — Tue, 24 May 2011 16:27:52 GMT

According to pudo, a ckan with activated solr extension throws a 5xx when solr is unreachable. Instead, it should behave more like a ckan without ckanext-solr when this happens.

#1230: Make sure plugins tables work in tests.

kindly — Tue, 19 Jul 2011 18:40:11 GMT

Dgu tests failed as harvesting table are not created at the correct point.

#1333: Move form converters to ckan core

amercader — Tue, 13 Sep 2011 10:21:08 GMT

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

#1150: Non-ASCII chars prevent data preview

timmcnamara — Sat, 21 May 2011 23:18:34 GMT

Characters outside of ASCII range are not supported within data previews.

Steps to reproduce:

Visit http://ckan.net/package/kivele2010
Click on [preview] for any of the resources

#1221: Password reset tidy up

dread — Tue, 12 Jul 2011 13:01:01 GMT

No tests.
500 error given when user is not found.

#1247: Put docs on ReadTheDocs

dread — Fri, 29 Jul 2011 12:19:08 GMT

Put docs on ReadTheDocs? instead of eu13.

#1172: Remove all try: except: blocks that don't re-raise the original exception

thejimmyg — Wed, 01 Jun 2011 11:00:49 GMT

The current codebase has one or two try: except; blocks that don't catch specific exceptions. Under no circumstances should any broad try: except: blocks be allowed unless the exceptions they catch are immediately re-raised. Uncaught exceptions are wasting us quite a lot of time when trying to track down problems.

#1239: Remove and re-add tag on package not working

rgrp — Sat, 23 Jul 2011 17:10:20 GMT

See reports on list at:

Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix.

#1246: Search results on ckan.net are mistakenly all 'open'

adrian.pohl@… — Thu, 28 Jul 2011 14:29:13 GMT

All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: http://ckan.net/package

#1234: Setting locale error conditions handled badly

dread — Wed, 20 Jul 2011 16:37:51 GMT

Needs tests for error conditions for bad locale and redirect parameters.

#1195: Storage extension to handle incorrect routes better

dread — Mon, 20 Jun 2011 13:32:49 GMT

We've seen exceptions caused by people getting the file storage link wrong, adding a slash to the end.

e.g. http://ckan.net/storage/f/file/e696d857-e997-41c8-be08-408697168ca8/

We should aim to handle all requests better than causing exception. 404 (or even 302 in this case) is better than 500.

#1421: Tag cloud recipe broken

dread — Wed, 26 Oct 2011 14:16:15 GMT

home page doesn't have c.facets setup.

Only affects 1.5 branch - ok 1.4.3.1. Was broken in cset:05f60f2d26e8.

#1427: Tags validation needs to limit length to 100 characters

dread — Thu, 27 Oct 2011 09:44:58 GMT

The tag name is limited in length in the db table to 100 characters. If you try and create a longer one you get an exception. This should cause a ValidationError? and tell the user this sensibly.

#1274: Testing solr search

johnglover — Mon, 15 Aug 2011 13:00:02 GMT

Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend.

#1266: Timestamps without microseconds causing exception

dread — Mon, 08 Aug 2011 09:13:51 GMT

I'm not sure why some timestamps are missing microseconds in the ckan.net data, but plenty seem to. This causes this exception when viewing the package rss feed:

URL: http://ckan.net/package/history/sfk_publishing?format=atom&days=7
In Module ckan.lib.helpers:227 in date_str_to_datetime
WebApp Error: <type 'exceptions.ValueError'>: time data '2008-04-13T20:40:20' does not match format '%Y-%m-%dT%H:%M:%S.%f'

Only seen with ckan 1.4.3b

#1225: Tool for mass tag changes

dread — Thu, 14 Jul 2011 11:16:01 GMT

Using ckanapi

#1128: Upload Scotland gov data

dread — Fri, 06 May 2011 20:04:53 GMT

Upload to ckan.net: https://sites.google.com/site/scotlandsdata/

#1208: Webstore: a datastore with web API

rgrp — Mon, 04 Jul 2011 18:34:20 GMT

SQLite based
Use Cyclone for async http server on top with auth
Designed to not be specific to CKAN or anything else

Current code is here: https://github.com/okfn/webstore.

API Spec

Read

Two basic ways to query::

    GET: /{owner}/{db-name}/?sql=...
    GET: /{owner}/{db-name}/?table=...&attr=value&attr=value&limit=...

Returns::

  {
      u'keys': [u'id', u'name'],
      u'data': [
          [1, u'jones'],
          [u'aaa', u'jones']
          ]
  }

Write

POST to::

/{owner/{database}/{table}

Payload is json data structured as follows::

{

unique_keys: [list of key attributes] data: {dict of values}

}

Authentication and Authorization

Authentication: use basic auth header.

Authorization:

Default: all read, owner can write
Restricted: owner can read and write, everyone can do nothing

Possible future: config file can specify a python method (TODO: method signature)

Integration with Other Systems

TODO: Specify how to delegate authenatication to user database in some other system.

#1256: fix modification extension for moderation

kindly — Tue, 02 Aug 2011 10:13:46 GMT

Solr is not working for moderation, pending changes are not being indexed.

#1329: i18n poor in some templates - missing i18n tags

amercader — Mon, 12 Sep 2011 10:44:48 GMT

Just a tip for developers regarding translations:

When writing templates, mark <p>, <div>, etc elements with the

i18:msg attribute, otherwise the messages don't get extracted properly.

Good: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40

<p i18n:msg="">More about CKAN, and the background to its creation can
   be found in <a
     href="http://m.okfn.org/files/talks/opentech_20080705/">these
     slides</a>.</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

<p class="hints">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href="http://opendatacommons.org/licenses/odbl/1.0/">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the grammar or syntax of the translated language.

#1176: unicode_body exception

dread — Mon, 06 Jun 2011 09:12:42 GMT

We seem to occasionally get this exception on ckan.net.

User agents causing this include:

Alexa ia_archiver (3/6/11)
Maxthon browser (17/5/11)

Module ckan.controllers.error:32 in document
<<              return original_response.body
               # Otherwise, decorate original response with error template.
               c.content = literal(original_response.unicode_body) or cgi.escape(request.GET.get('message', ''))
               c.prefix=request.environ.get('SCRIPT_NAME', ''),
               c.code=cgi.escape(request.GET.get('code', str(original_response.status_int))),
>>  c.content = literal(original_response.unicode_body) or cgi.escape(request.GET.get('message', ''))
AttributeError: 'NoneType' object has no attribute 'unicode_body'

#1265: user/edit error conditions cause exceptions

dread — Fri, 05 Aug 2011 17:36:28 GMT

We get exceptions on these occasions:

user/edit if not logged in
user/edit/bob if bob is not a known user

These are a problem only on 1.4.3b.

Introduced in #1229 - merge of branch feature-1229-db-out-of-controllers

#1312: Particular characters in JSON cause exception creating package on API

florian.marienfeld@… — Tue, 06 Sep 2011 14:21:03 GMT

From Florian:

I am having trouble with the characters "=&;".
if they occur e.g. in the title of a package, I get a 400-Bad-Request
when I try to register a package by POSTing to ckan/rest/api/package
However, when I PUT the same package as an edit to
ckan/rest/api/package/existing_package, it works and the title shows
correctly on the front end and API.

Reproduced

(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{"name": "test3", "title": "Test &"}' -H "Authorization:tester"
HTTP/1.0 400 Bad Request
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 14:09:31 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Length: 87
"Bad request - JSON Error: Unterminated string starting at: line 1 column 27 (char 27)"

#1426: Validation errors cause 409 not 400 response

dread — Thu, 27 Oct 2011 09:43:39 GMT

Validation error is incorrect arguments, so should be a "400 Bad Request" response. "409 Conflict" is for when there is a conflict creating two resources with the same identifier. Yet we get 409 for all validation errors.

#1238: Viewing a package @date misses non-PackageRevisions

dread — Fri, 22 Jul 2011 16:15:03 GMT

There's a problem with getting old versions of packages using @date. The query looks for the PackageRevision? related to the date, but ignores TagRevisions? etc.

So for example if you compare this table: http://ckan.net/package/history/osm with this view: http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830 you actually see the 2010-01-13 11:13 revision - the wrong revision.

#1418: If default language is not 'en' then you can't switch language to 'en'

dread — Tue, 25 Oct 2011 10:39:06 GMT

e.g. cz.ckan.net has lang=cz but when you click on Home page link "English" it flashes message "Language has been set to: English" but the rest of the page is still Czech.