{22} Trac tickets (2647 matches)

Results (1001 - 1100 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#1053 defect dread dread ckan-v1.5 closed fixed Deletion in Model API

Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted.

There is no way to delete objects at the moment - changes to 'state' are ignored in the API.

Do we need an alternative way to purge objects in the API?

1300790039000000 1310126546000000
#1055 defect dread dread ckan-v1.4-sprint-4 closed fixed @search_related tests not running

Tests marked decorated "@search_related" should only be run against postgresql, but in fact they don't get run at all.

1300985228000000 1300992395000000
#1056 defect dread pudo ckan-v1.4-sprint-6 closed fixed User links for OpenID users are broken

Use case:

  • Login using OpenID
  • Click on 'My account' - results in 404

Solutions:

  • User user.id instead of their name
  • Escape the URL properly.
1301060249000000 1302882616000000
#1058 defect dread dread ckan-v1.4-sprint-4 closed fixed Give 400 error (not 500) for invalid locale or package_form

Examples which prompt annoying exception emails:

http://ckan.net/locale?locale=ja
Module ckan.i18n:21 in set_session_locale
           assert locale in _KNOWN_LOCALES

A bot has caused these:

http://ca.ckan.net/package/new?package_form=gov
Module ckan.forms.registry:32 in get_fieldset
               raise ValueError('Could not find package_form name %r in those found: \n%r' % (package_form, [en.name for en in entrypoints]))
ValueError: Could not find package_form name u'gov)' in those found: ['gov', 'standard', 'ca']
1301302303000000 1301303315000000
#1059 defect dread dread ckan-v1.4-sprint-5 closed fixed Loader coping better with poor search indexing

Loader currently checks for same name, but also should check for name_, name etc.

1301310596000000 1301312516000000
#1061 enhancement dread dread ckan-v1.4-sprint-5 closed invalid Orphaned home/license page

No links to home/license and it contains out of date references to knowledgeforge. Remove it.

1301392968000000 1301922350000000
#1066 enhancement dread dread ckan-v1.4-sprint-5 closed fixed Default reader role too permissive

The definition of the 'reader' role includes creating packages, which is too permissive for some CKAN instances (e.g. DGU). 'Reader' suggests only reading, so I think this role should avoid creating and editing.

All projects so far want all roles to be able to create users, so this stays as a Reader action for now, as a convenience.

Implementation:

  • Action.PACKAGE_CREATE removed from reader's default_role_actions
  • Visitor has a new default role, called 'anon_editor' which can edit packages, but not groups / auth groups - you have to log in for that.
  • Migration script not needed?
  • Code comments written, to make clear the suggested policy
1301645250000000 1301932136000000
#1067 enhancement dread dread ckan-v1.4-sprint-5 closed fixed CLI for loading/dumping complete databases

Use 'db dump' and 'db load' for 'pg_dump' and 'psql -f' of a database. Use pylons config to find out database options.

1301645463000000 1302186503000000
#1068 defect dread dread ckan-v1.4-sprint-5 closed fixed metadata_modified problem

This test has been failing since the clocks changed:

======================================================================
FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dread/hgroot/pyenv-ckan2/lib/python2.6/site-packages/nose-0.11.3-py2.6.egg/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/home/dread/hgroot/ckan2/ckan/tests/models/test_package.py", line 283, in test_02_metadata_created_and_modified
    assert out == exp, (out, exp)
AssertionError: (datetime.datetime(2011, 4, 1, 10, 45, 50, 875509), datetime.datetime(2011, 4, 1, 9, 45, 50, 875509))

----------------------------------------------------------------------
1301652085000000 1302109505000000
#1071 defect dread dread ckan-v1.4-sprint-5 closed fixed Package history API moved to /api/rest/package/revisions

api/rest/package_history is not RESTful or follow API naming conventions. Therefore move it to /api/rest/package/revisions

Also, API docs incomplete.

1301937882000000 1301943180000000
#1072 enhancement dread dread ckan-v1.4-sprint-5 closed fixed Add filters to authztool

It takes several minutes to print the 'rights' on DGU, which is annoying when you only want to grep for a few lines. Much quicker than grepping is to filter in the query.

1302106311000000 1302106474000000
#1073 enhancement dread dread ckan-v1.4-sprint-5 closed fixed Search index checker

Tool that checks which packages have not been indexed.

Required for DGU: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/940

1302185444000000 1302185825000000
#1085 defect dread johnlawrenceaspden closed fixed local development copy of ckan depends on existence of ckan.net

ckan.net appears to have either gone down or be running ultra slowly.

this means that ckan copies running locally on my machine run very slowly indeed.

is this behaviour desirable?

This command finds lots of http://~~~ckan.net references in python, html and javascript files:

find ~/pyenv/src \( -name "*.py" -or -name "*.html" -or -name "*.js" \) -print0 | xargs -0 -e grep --color -nH -e "http://.*ckan.net"

output for reference:


/home/okfn/pyenv/src/ckan/ckan/init__.py:5:Network (CKAN) site: http://www.ckan.net. /home/okfn/pyenv/src/ckan/ckan/lib/create_test_data.py:346:<http://ckan.net/> /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:3:DOMAIN = 'http://ckan.net' /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:4:CKAN_NAMESPACE = 'http://ckan.net/#' /home/okfn/pyenv/src/ckan/ckan/lib/talis.py:60: 'ckan':'http://ckan.net/ns#', /home/okfn/pyenv/src/ckan/ckan/public/scripts/bookmarklet.js:2: f='http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&title='+encodeURIComponent(document.title); /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:16: addtockan.src='http://ckan.net/scripts/bookmarklet.js'; /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:27: <p><strong>Proper bookmarklet (compressed -- need to escape &amp;):</strong> <a href="javascript:(function(){f='http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;title='+encodeURIComponent(document.title);if((n=document.getElementsByName('description')[0])&amp;&amp;(d=n.content)){f+='&amp;notes='+encodeURIComponent(d);}a=function(){if(!window.open(f)){location.href=f;}};if(/Firefox/.test(navigator.userAgent)){setTimeout(a,0)}else{a()}})()">Add to CKAN</a> /home/okfn/pyenv/src/ckan/ckan/templates/home/license.html:31: For convenience, all material - including all package, tag and revision information - is available in bulk, in the form of a full dump of the CKAN database. This (gzipped) dump file is updated daily and can be downloaded from <a href="http://www.ckan.net/dump/">http://www.ckan.net/dump/</a>. /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:71: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:137: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:447: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:458: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/functional/api/base.py:178: assert '"ckan_url": "http://test.ckan.net/package/annakarenina"' in msg, msg /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:116: api e.g. http://ckan.net/api rather than http://ckan.net/api/rest) /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:261: :param base_location: default *http://www.ckan.net/api* /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:267: base_location = 'http://www.ckan.net/api'


1302620434000000 1302625314000000
#1089 enhancement dread dread ckan-v1.4-sprint-6 closed fixed Check for "--ckan" when running nosetests

(because if you forget, you get difficult to understand errors, and more than one person has tripped up on this)

1302631189000000 1302631733000000
#1090 defect dread dread ckan-v1.4-sprint-6 closed fixed Visitor can't create packages on new CKAN install

Default visitor roles in default config is reader, not anon_editor.

Problem caused by changes in #1066 (released in 1.3.3)

New installs will be affected, although simple to just increase permissions when the installer realises a visitor can't create packages.

The solution to the config getting out of sync with the code like this is to not have the default_roles in the config - refer to the code in the configuration instructions.

1302635219000000 1302635699000000
#1093 defect dread dread ckan-v1.4-sprint-6 closed fixed 500 errors on GET to api/rest/licenses

CKAN gets its license list from a license service, which can be a local file, but is often the http://licenses.opendefinition.org/2.0/ckan_original server. This server is currently flakey, but I think we only request the list on start up. The problem is we query it much more often than required. It is queried for every request to api/rest/licenses, and we are returning lots of 500 errors when the license server is timing out.

1302862261000000 1302865470000000
#1097 enhancement dread dread ckan-v1.4-sprint-6 closed fixed Sidebar hideable

The web interface has a sidebar (#primary) which should be hidden in some pages. This is for QA extension and useful for package new and edit pages. Must be compatible with DGU theme.

1303293416000000 1303293476000000
#1098 task dread dread ckan-v1.4-sprint-6 closed fixed --ckan-migration tests not initialised correctly

Only tests with failing --ckan-migration fail, due to authz not being initialised.

1303377336000000 1303406017000000
#1104 defect dread johnlawrenceaspden ckan-v1.4-sprint-7 closed fixed create-test-data doesn't index the packages it creates

With the default test data created by

paster db clean paster db init paster create-test-data

going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy

But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.

1303494635000000 1303920791000000
#1112 enhancement dread dread ckan-v1.4-sprint-7 closed fixed Allow searching for all packages
GET api/search/package?q=

returns all packages. This is so you can filter them e.g. by openness, which is not currently possible.

1304007852000000 1304358643000000
#1114 enhancement dread dread ckan-v1.4-sprint-7 closed fixed CLI for viewing search index of a package

To see what lexemes are generated for debug purposes.

1304078353000000 1304085484000000
#1116 defect dread dread ckan-v1.4-sprint-7 closed fixed api search loses boolean q options

filter_by_openness and filter_by_downloadable options don't work when specified as URI parameters. (They do work in qjson parameters)

1304086823000000 1304358664000000
#1123 requirement dread nils.toedtmann closed fixed Please re-package CKAN packages as "noarch"

... or, if the CKAN packages do contain architecture-specific binary code, build packages for i386 too.

Currently, http://apt-alpha.ckan.org/debian only offers packages for amd64, but e.g. "m1.small" EC2 instances are i386.

We would need this in order to migrate the community instances to a packaged based CKAN.

Rufus, pls prioritise.

1304530050000000 1311863806000000
#1125 enhancement dread nils.toedtmann closed fixed Debian package "ckan" should not depend on "postgresql"

The debian package "ckan" with the two scripts "ckan-create-instance" and "ckan-instance-maintenance" depends against "postgresql". But "ckan-create-instance" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs.

Please add a flag "--without-local-db" to "ckan-create-instance" and remove the postgres dependancy from the debain package.

1304538095000000 1310134813000000
#1126 enhancement dread dread ckan-v1.4-sprint-7 closed fixed Exceptions arising from error page

I'm not completely clear what the use case is for loading the error page in this way, but somehow original_request is None and that creates an unnecessary exception with the logic refactor.

http://ckan.net/error/document?__cache=39020485
...
Module ckan.controllers.error:29 in document
<<          original_response = request.environ.get('pylons.original_response')
               # Bypass error template for API operations.
               if original_request.path.startswith('/api'):
                   return original_response.body
               # Otherwise, decorate original response with error template.
>>  if original_request.path.startswith('/api'):
AttributeError: 'NoneType' object has no attribute 'path'
1304586683000000 1304587078000000
#1128 task dread dread ckan-sprint-2011-10-28 closed fixed Upload Scotland gov data

Upload to ckan.net: https://sites.google.com/site/scotlandsdata/

1304712293000000 1310568028000000
#1131 defect dread dread ckan-v1.4-sprint-7 closed fixed Search param validation exception not caught

Example request:

http://nl.ckan.net/api/2/search/package?q=delft&order_by=&offset=&limit=&tags=

Gives 500 error:

<type 'exceptions.ValueError'>: invalid literal for int() with base 10: ''
1304942023000000 1305537897000000
#1143 enhancement dread dread ckan-v1.5-sprint-2 closed fixed Improve stats page
  • Ensure we don't include deleted packages in the stats
  • Some visual improvements:
    • Number of packages:
      • fix x axis to start at first revision
      • fix y axis to start at zero
    • Fix problem with legend to 'Revisions to packages' graph.
    • Hide sidebar in the template, so it isn't right-cropped in the DGU theme
  • Add some testing of the stats lib - results of basic stats
1305734836000000 1307024681000000
#1169 enhancement dread dread ckan-v1.5-sprint-2 closed fixed Link to csv/json dumps on ckan.net 1306749768000000 1306754833000000
#1170 defect dread dread ckan-v1.5-sprint-2 closed fixed Production deployment docs out of date

Need to cover pip-requirements.txt on different branches in the doc/deployment.rst.

Also could do with tidying up upgrade.txt into this document too.

1306864171000000 1306864422000000
#1172 defect dread thejimmyg ckan-sprint-2011-10-28 closed fixed Remove all try: except: blocks that don't re-raise the original exception

The current codebase has one or two try: except; blocks that don't catch specific exceptions. Under no circumstances should any broad try: except: blocks be allowed unless the exceptions they catch are immediately re-raised. Uncaught exceptions are wasting us quite a lot of time when trying to track down problems.

1306926049000000 1314118883000000
#1175 defect dread fccoelho@… closed invalid Stats extension not working

Hi, I get a 500, Internal server error when I enable ckanext-stats. Flavio

1307350823000000 1325355170000000
#1180 defect dread dread ckan-v1.5-sprint-3 closed fixed Links in markdown can be badly formed

User can insert bad anchor tags into the User-About and Package-Notes fields and when you view them (web interface) it causes a 500 error.

Need to improve filtering for anchors in markdown.

<a href="http://xxxsex.com>nasty/website

Also check this related exception:

Module ckan.controllers.user:59 in read
<<          c.is_myself = user.name == c.user
               c.api_key = user.apikey
               c.about_formatted = self._format_about(user.about)
               revisions_q = model.Session.query(model.Revision
                       ).filter_by(author=user.name)
>>  c.about_formatted = self._format_about(user.about)
Module ckan.controllers.user:167 in _format_about
<<      def _format_about(self, about):
               about_formatted = ckan.misc.MarkdownFormat().to_html(about)
               return genshi.HTML(about_formatted) 
       
           def _get_form_password(self):
>>  return genshi.HTML(about_formatted)
WebApp Error: <class 'genshi.input.ParseError'>: junk characters in start tag: u'\u201dhttp://www.settingu': line 1, column 3
1307469037000000 1307544223000000
#1181 defect dread dread closed fixed Link spam vulnerability in Notes and User-About fields

When viewing a user and a package, the about/notes fields contain Markdown, which may have links. These should have rel="nofollow" to discourage link spam.

1307523160000000 1307532816000000
#1187 enhancement dread dread ckan-v1.5-sprint-3 closed fixed Stop spam users with links in the 'about' field

When you edit a user, you should be stopped from putting links in the 'about' field, because this is a common tactic by spammers.

1308142999000000 1308143605000000
#1191 defect dread dread ckan-v1.5-sprint-3 closed fixed Unicode in user password gives 500 error on user registration

Create/register a user with a unicode character in the password. It creates the user, but the redirect to the user page doesn't work and results in 500 error.

1308312895000000 1308650930000000
#1196 defect dread dread ckan-v1.5-sprint-3 closed fixed Markdown logging ignores config settings

Markdown produces Debug logging, even when the pylons logging config is set to level of Info or Warning. This clogs up the logfile.

1308743476000000 1308743857000000
#1207 enhancement dread dread closed fixed ckanclient.package_entity_get should raise more specific exception

When package does not exist in ckan catalogue, ckanclient.package_entity_get should raise more specific exception, such as CkanNotFoundError? instead of generic CkanApiError?.

1309515582000000 1311325343000000
#1210 defect dread dread ckan-v1.5-sprint-4 closed fixed POST application/json error handling with newer WebOb

WebOb? from v1.0.7 has some interesting new behaviour with reading request data for different Content-Types:

  • It seems that looking at request.body mangles request.POST. See fix in ckan cset:e83bad9caa1b (problem with WebOb? v1.0.7 but may go back to v1.0.4?)
  • A particular request caused an AssertionError? during looking at request.body, so we need to enclose this in a try/except. I couldn't recreate it in a test unfortunately (problem with WebOb? v1.0.7 but may go back to v1.0.2?)

Example:

Module ckan.controllers.api:206 in create
<<          log.debug('create: %s' % (context))
               try:
                   request_data = self._get_request_data()
               except ValueError, inst:
                   response.status_int = 400
>>  request_data = self._get_request_data()
Module ckan.lib.base:149 in _get_request_data
<<          cls.log.debug('Retrieving request params: %r' % request.params)
               cls.log.debug('Retrieving request POST: %r' % request.POST)
               cls.log.debug('Retrieving request POST body: %r' % request.body)
               if request.POST:
                   try:
>>  cls.log.debug('Retrieving request POST body: %r' % request.body)
Module paste.registry:137 in __getattr__
<<      
           def __getattr__(self, attr):
               return getattr(self._current_obj(), attr)
           
           def __setattr__(self, attr, value):
>>  return getattr(self._current_obj(), attr)
Module webob.request:470 in _body__get
<<          Return the content of the request body.
               """
               self.make_body_seekable() # we need this to have content_length
               r = self.body_file.read(self.content_length)
               self.body_file.seek(0)
>>  self.make_body_seekable() # we need this to have content_length
Module webob.request:697 in make_body_seekable
<<              self.body_file_raw.seek(0)
               else:
                   self.copy_body()
>>  self.copy_body()
Module webob.request:714 in copy_body
<<                  self.body = self.body_file_raw.read(length)
               elif self.is_body_readable:
                   self.body = self.body_file_raw.read()
                   self._copy_body_tempfile()
               else:
>>  self.body = self.body_file_raw.read()
Module webob.request:1190 in read
<<      def read(self, size=-1):
               body = self._get_body()
               if size < 0:
                   v = body[self.position:]
>>  body = self._get_body()
Module webob.request:1207 in _get_body
<<                  self._body = _encode_multipart(self.vars, self.content_type)
                   else:
                       assert 0, ('Bad content type: %r' % self.content_type)
               return self._body
>>  assert 0, ('Bad content type: %r' % self.content_type)
AssertionError: Bad content type: '; charset=utf-8'
1309949451000000 1309974781000000
#1214 defect dread dread closed worksforme API improvements (following javascript use)

Notes on the CKAN API from Aron:

  • Server returns text/html for errors even when "Accept: application/json" header is set.
  • DELETE package request requires a Content-Length? Should be 405 Method Not Allowed?
  • Tag returned as a JSON object when updating but as a string when requesting. Same with "extras" content.
  • How to remove key from "extras" object? Passing None as described in the docs is invalid JSON and using null doesn't work.
  • Tags return package names rather than ids when querying GET /tag/{id}
  • Doesn't return rendered_text property on package update.
  • Relationships use "object" key rather than id or package_id.
1309975222000000 1314029628000000
#1217 defect dread dread ckan-sprint-2011-11-07 closed invalid Diff exception

Exception raised doing this particular diff.

http://ckan.net/revision/diff/sudocfr?diff_entity=package&oldid=a18b2520-09e0-44be-80dd-4fea94ffe7d1&diff=3934ff16-a3a7-4481-b390-72944903b945

...
Module ckan.controllers.revision:142 in diff
<<          if c.diff_entity == 'package':
                   c.pkg = model.Package.by_name(id)
                   diff = c.pkg.diff(c.revision_to, c.revision_from)
               elif c.diff_entity == 'group':
                   c.group = model.Group.by_name(id)
>>  diff = c.pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:449 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'

Seems to be to do with this particular extra key:

[<PackageExtraRevision id=ba1f3b3e-f61f-4924-bae0-ec3dd2163ae4 package_id=febe4da9-5874-4411-a7da-b497fdebf739 key=namespace value=http://www.sudoc.fr/ state=active revision_id=5ec8e730-c73d-4f1c-989b-2e015eb3d315 continuity_id=ba1f3b3e-f61f-4924-bae0-ec3dd2163ae4 expired_id=None revision_timestamp=2011-07-08 13:48:11.027260 expired_timestamp=9999-12-31 00:00:00 current=True>]

Is it because it was created between the from_revision and to_revision?

1310375498000000 1320663277000000
#1218 enhancement dread minspamboks@… ckan-sprint-2011-10-28 closed fixed Colour the History tab icon

Change the color of the "History" tab icon to yellowish, like the rest of the icons in the other tabs ("View" and "Edit", "Authorization").

Reasoning

When you view a data package, for instance http://ckan.net/package/thesaurus-w, you will see "View", "Edit", "History" tabs on the top. "History" tab has a black-and-white icon which makes it look like an inactive/disabled tab (since the text is also grayed out when the tab is not selected). This is not a major issue, but it is a little bit confusing for the users. This icon exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net).

The simple solution would be to change the color of the "History" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. "View" and "Edit").

1310375768000000 1310389390000000
#1221 defect dread dread ckan-sprint-2011-10-28 closed fixed Password reset tidy up
  • No tests.
  • 500 error given when user is not found.
1310475661000000 1310556607000000
#1222 defect dread dread ckan-sprint-2011-10-28 closed fixed Mailer has no tests 1310479136000000 1310556618000000
#1225 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Tool for mass tag changes

Using ckanapi

1310642161000000 1310642407000000
#1228 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Display OpenID login errors

When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again.

1311086562000000 1311325161000000
#1229 enhancement dread thejimmyg ckan-sprint-2011-10-24 closed fixed Refactor all database code out of the controllers and into the logic layer

Progress can be tracked in this document:

https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdFQzRnVoUHV6ckoyblI5TUdEcnJLVlE&hl=en_US#gid=0

1311093924000000 1319645778000000
#1234 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Setting locale error conditions handled badly

Needs tests for error conditions for bad locale and redirect parameters.

1311179871000000 1311183328000000
#1236 enhancement dread dread ckan-sprint-2011-11-07 closed fixed View a package at a given revision

As a user I want to view a package at a given revision:

  • When I visit /package/osm@yyyy-mm-dd or /package/osm@rrrrrrr I should be shown package at date yyyy-mm-dd or revision rrrrrrr.
  • viewing this page, it should say it is the old revision, just like wikipedia
  • package history page should provide links to these pages
1311269333000000 1330019788000000
#1248 defect dread dread ckan-sprint-2011-11-07 closed fixed License exception

Exception received on test.ckan.net at 10.12 on 29/7/11

http://test.ckan.net/package?__cache=2759373&page=4
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:153 in search
<<                          }
       
                   query = get.package_search(context,data_dict)
       
                   c.page = h.Page(
>>  query = get.package_search(context,data_dict)
Module ckan.logic.action.get:514 in package_search
<<      for package in query.results:
               result_dict = table_dictize(package, context)
               result_dict = _extend_package_dict(result_dict,context)
       
               results.append(result_dict)
>>  result_dict = _extend_package_dict(result_dict,context)
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: None'

and a similar one:

 http://test.ckan.net/tag/barbecue
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.tag:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__("logging").getLogger("proxy_cache")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
       
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.tag:64 in read
<<          data_dict = {'id':id}
               try:
                   c.tag = get.tag_show(context,data_dict)
               except NotFound:
                   abort(404, _('Tag not found'))
>>  c.tag = get.tag_show(context,data_dict)
Module ckan.logic.action.get:341 in tag_show
<<      extended_packages = []
           for package in tag_dict['packages']:
               extended_packages.append(_extend_package_dict(package,context))
       
           tag_dict['packages'] = extended_packages
>>  extended_packages.append(_extend_package_dict(package,context))
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: CreativeCommonsAttributionCCBY25'
1311954136000000 1319797763000000
#1254 task dread dread ckan-sprint-2011-10-28 closed fixed Feature sheet

Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers.

1312211925000000 1313401629000000
#1263 enhancement dread dread closed fixed Bad name for logic action in API causes Exception

A bad action name: e.g. curl http://test.ckan.net/api/action/user_showaa -d '{}' causes 500 response.

Occurs only on 1.4.3 beta.

1312479020000000 1314029279000000
#1265 defect dread dread ckan-sprint-2011-10-28 closed fixed user/edit error conditions cause exceptions

We get exceptions on these occasions:

  • user/edit if not logged in
  • user/edit/bob if bob is not a known user

These are a problem only on 1.4.3b.

Introduced in #1229 - merge of branch feature-1229-db-out-of-controllers

1312565788000000 1312566052000000
#1266 defect dread dread ckan-sprint-2011-10-28 closed fixed Timestamps without microseconds causing exception

I'm not sure why some timestamps are missing microseconds in the ckan.net data, but plenty seem to. This causes this exception when viewing the package rss feed:

URL: http://ckan.net/package/history/sfk_publishing?format=atom&days=7
In Module ckan.lib.helpers:227 in date_str_to_datetime
WebApp Error: <type 'exceptions.ValueError'>: time data '2008-04-13T20:40:20' does not match format '%Y-%m-%dT%H:%M:%S.%f'

Only seen with ckan 1.4.3b

1312794831000000 1312797477000000
#1267 defect dread dread closed fixed Random exception: cannot set the body to a unicode value

Occasionally see this problem when producing an error page. Here's an example

WebApp Error: <type 'exceptions.TypeError'>: You cannot set the body to a unicode value without a charset
URL: http://at.ckan.net/authorizationgroup/new
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:117 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:284 in __call__
<<              if log_debug:
                       log.debug("Calling Response object to return WSGI data")
                   return response(environ, self.start_response)
               
               if log_debug:
>>  return response(environ, self.start_response)
Module webob.exc:248 in __call__
<<              return []
               if not self.body and not self.empty_body:
                   return self.generate_response(environ, start_response)
               return Response.__call__(self, environ, start_response)
>>  return self.generate_response(environ, start_response)
Module webob.exc:239 in generate_response
<<              status=self.status,
                   headerlist=headerlist,
                   content_type=content_type
               )
               return resp(environ, start_response)
>>  content_type=content_type
Module webob.response:94 in __init__
<<                  if charset is None:
                           raise TypeError(
                               "You cannot set the body to a unicode value without a charset")
                       body = body.encode(charset)
                   self._body = body
>>  "You cannot set the body to a unicode value without a charset")
TypeError: You cannot set the body to a unicode value without a charset
1312889084000000 1312897724000000
#1270 defect dread dread ckan-sprint-2011-10-28 closed fixed Default site www.ckan.net is a redirect and loses POSTs

The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters:

In [1]: import ckanclient

In [2]: c = ckanclient.CkanClient()

In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api')

In [4]: c.package_search('', search_options={'groups':'openspending'})
Out[4]: 
{u'count': 2102,
 u'results': <generator object _result_generator at 0x10168b1e0>}

In [5]: c2.package_search('', search_options={'groups':'openspending'})
Out[5]: {u'count': 29, u'results': <generator object _result_generator at 0x10168b410>}

Thanks to borior for finding this and raising it.

1312911847000000 1312912188000000
#1283 enhancement dread dread ckan-backlog closed fixed Deleted packages shouldn't be searchable or browsable

A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else.

(But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.)

Same in the Web interface and API.

This ticket ties in with #948 highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can #1076.

1314112481000000 1330083933000000
#1284 defect dread rgrp closed fixed Lots of revisions being created on datahub.org

Revisions aren't assigned to packages. Where are they coming from? This seems to have been occurring since the upgrade to 1.4.3b a couple of weeks ago?

1314113015000000 1315215626000000
#1290 enhancement dread dread ckan-backlog closed fixed Better error when blank database

When installing CKAN, when doing "paster serve development.ini", lots of users encounter the error for every request:

ProgrammingError: (ProgrammingError) relation "user" does not exist

This is because the database tables have not been created - they have forgotten or missed the "paster db init" step.

Can we provide a better error to say that the database is not initialised yet?

Implementation options

  1. At the start of every request we reflect the database tables and check they are there. This is rather expensive!
  1. Uncached requests to the home page start with a cheap database query. If there is an exception then return this error about database setup. I really like Drupal's page for this that has in large letters that the site is currently off-line. Below a line, in small letters, there are developer suggestions on what is wrong and where to look to fix it.
1314264255000000 1314270894000000
#1292 enhancement dread dread ckan-sprint-2011-11-07 closed fixed Internationalization improvements
  • Review process of releases and updating strings (pot) and translations (po) files
  • Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?)
  • Review long list of strings for improvement
    • Consolidate any?
    • Get rid of i18n for strings in api
    • Try and remove javascript
    • Add in extensions
  • #1374 Fix switching to en from other default language
  • #1417 Fix browser language detection
1314303314000000 1320145732000000
#1310 defect dread dread ckan-sprint-2011-09-12 closed fixed API GET of a revision that has been approved causes exception

e.g. this particular revision on ckan.net: http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c (most others have not been approved) Causes 500 error.

1315234899000000 1315820419000000
#1312 enhancement dread florian.marienfeld@… ckan-sprint-2011-10-28 closed invalid Particular characters in JSON cause exception creating package on API

From Florian:

I am having trouble with the characters "=&;".
if they occur e.g. in the title of a package, I get a 400-Bad-Request
when I try to register a package by POSTing to ckan/rest/api/package

However, when I PUT the same package as an edit to
ckan/rest/api/package/existing_package, it works and the title shows
correctly on the front end and API.

Reproduced

(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{"name": "test3", "title": "Test &"}' -H "Authorization:tester" 
HTTP/1.0 400 Bad Request
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 14:09:31 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Length: 87

"Bad request - JSON Error: Unterminated string starting at: line 1 column 27 (char 27)"
1315318863000000 1315327743000000
#1315 enhancement dread dread ckan-sprint-2011-11-07 closed fixed Document /api/action

We need some documentation for the /api/action. Even though it's beta and subject to change it would be good to have a guide.

1315396722000000 1320173768000000
#1318 defect dread dread ckan-sprint-2011-10-28 closed fixed 'Register user' doesn't log you in

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

1315415783000000 1315569362000000
#1321 defect dread dread ckan-sprint-2011-10-24 closed fixed Flash message deferred

When you reset a password, the flash message saying "check your email" sometimes doesn't display on the next page. You then click on the link in your email and it displays on that page (i.e. too late).

Seen this on thedatahub.org. To do with the proxy caching?

1315474657000000 1319814460000000
#1323 enhancement dread dread ckan-sprint-2011-10-28 closed fixed GET /authorizationgroup causes Exception when logged in

on thedatahub.org

1315475685000000 1315594333000000
#1325 defect dread dread ckan-sprint-2011-09-12 closed fixed python 2.5 incompatibilities

We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in:

  • 'with' statement requires from __future__ import with_statement
  • import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5.
1315580452000000 1315587113000000
#1329 defect dread amercader ckan-sprint-2011-10-28 closed fixed i18n poor in some templates - missing i18n tags

Just a tip for developers regarding translations:

  • When writing templates, mark <p>, <div>, etc elements with the

i18:msg attribute, otherwise the messages don't get extracted properly.

Good: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40

<p i18n:msg="">More about CKAN, and the background to its creation can
   be found in <a
     href="http://m.okfn.org/files/talks/opentech_20080705/">these
     slides</a>.</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

<p class="hints">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href="http://opendatacommons.org/licenses/odbl/1.0/">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the grammar or syntax of the translated language.

1315824288000000 1317404566000000
#1331 defect dread dread closed worksforme Setting a tag twice causes exception

To reproduce:

  1. Create a package with two tags the same: "bulk bulk"
  2. Click 'save'
  3. 500 ERROR - 'Server Error'
1315905959000000 1325355631000000
#1340 defect dread dread ckan-sprint-2011-10-28 closed fixed Editing non-existent group causes Exception

e.g. http://ckan.net/group/edit/history

Module ckan.controllers.group:135 in edit
<<          try:
                   old_data = get_action('group_show')(context, data_dict)
                   c.grouptitle = old_data.get('title')
                   c.groupname = old_data.get('name')
>>  old_data = get_action('group_show')(context, data_dict)
Module ckan.logic.action.get:347 in group_show
<<      if group is None:
               raise NotFound
       
           check_access('group_show',context, data_dict)
>>  raise NotFound
1315991374000000 1315991534000000
#1346 defect dread dread ckan-sprint-2011-10-28 closed fixed 'get_authorization_group_dict not found' exception

This stems from a feature to allow you to prefill fields in the authorizationgroup creation form.

e.g. http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it.

1316081974000000 1316094632000000
#1347 defect dread dread ckan-sprint-2011-10-28 closed fixed Exception caused by New Group form POST with bad formed parameter

Exception caused by bots POSTing 'new group' form, but with bad (old-style) parameters.

<type 'exceptions.ValueError'>: invalid literal for int() with base 10: 'cache'
...
Module ckan.controllers.group:102 in new
<<          if context['save'] and not data:
                   return self._save_new(context)
               
               data = data or {}
>>  return self._save_new(context)
Module ckan.controllers.group:152 in _save_new
<<          try:
                   data_dict = clean_dict(unflatten(
                       tuplize_dict(parse_params(request.params))))
                   context['message'] = data_dict.get('log_message', '')
                   group = create.group_create(data_dict, context)
>>  tuplize_dict(parse_params(request.params))))
Module ckan.logic:57 in tuplize_dict
<<          for num, key in enumerate(key_list):
                   if num % 2 == 1:
                       key_list[num] = int(key)
               tuplized_dict[tuple(key_list)] = value
           return tuplized_dict
>>  key_list[num] = int(key)
ValueError: invalid literal for int() with base 10: 'cache'
1316094466000000 1316094608000000
#1365 defect dread dread ckan-sprint-2011-11-07 closed fixed Support old search formats

Lots of things rely on the old (i.e. pre-SOLR) search param formats, including ckanclient / DGU. Need a translation layer to convert these params (in api v1&2) to SOLR syntax.

This functionality was broken by the changes in #1277 which went into 1.5a

1317233338000000 1319815052000000
#1367 defect dread dread closed fixed PyPI ckan broken

manifest.cfg was not included in the pypi package, causing InvalidRepositoryError? when you do 'paster db init'.

Appears to affect most CKAN PyPI releases up to and including 1.4.3.

1317379931000000 1317394273000000
#1374 defect dread dread ckan-sprint-2011-10-24 closed fixed Can't switch to English if default is non-English

e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English.

1317893975000000 1319648746000000
#1385 enhancement dread dread closed fixed Resolve postgres permissions issues

Currently there is a problem because the docs guide us to set the sqlalchemy url to use 'localhost' i.e. loopback, whereas paster commands don't specify '-h localhost' so use unix sockets (you need to do 'sudo -u postgres'). These should be the same.

Also do we need to tell people to add a line to their postgres authentication config /etc/postgresql/8.4/main/pg_hba.conf to help things? Florian suggests:

local   std         std                          md5

/etc/postgresql/8.4/main/pg_hba.conf

1318418537000000 1326218703000000
#1394 defect dread dread ckan-sprint-2012-01-09 closed fixed Resource validation error messages misleading

(Editing a dataset) If the second resource contains any validation error then it says "Resources: Package resource(s) incomplete" and "Resource 1:".

1318515262000000 1325604784000000
#1405 defect dread jonathan.gray@… ckan-sprint-2011-11-07 closed fixed You can add a package twice to a group

To reproduce (from Jonathan Gray):

  • I go to add an item to the group
  • As I type, I get autocomplete options (cool!)
  • When I click on one, it adds this, and copies the same autocomplete option in the box
  • When I click submit, it seems to add the item twice
  • On the main group page I think it just shows up once, but when I click edit it shows two identical boxes, both ticked - and the count includes two copies of the same item

Basically the problem is allowing the same package associated multiple times with a group.

This is an issue on CKAN 1.5b and probably earlier.

1318856354000000 1320152291000000
#1416 defect dread dread ckan-sprint-2011-11-07 closed fixed Registering with blank email can cause exception

This might well be a bot, but not POSTing the email field should not cause an exception - it should be 400 and then the sysadmin doesn't get these nags.

http://www.publicdata.eu/user/register
...
Module ckan.controllers.user:79 in register
<<              c.login = request.params.getone('login')
                   c.fullname = request.params.getone('fullname')
                   c.email = request.params.getone('email')
                   if not model.User.check_name_available(c.login):
                       h.flash_error(_("That username is not available."))
>>  c.email = request.params.getone('email')
Module webob.multidict:335 in getone
<<          values were found.
               """
               return self._decode_value(self.multi.getone(self._encode_key(key)))
       
           def mixed(self):
>>  return self._decode_value(self.multi.getone(self._encode_key(key)))
Module webob.multidict:99 in getone
<<          v = self.getall(key)
               if not v:
                   raise KeyError('Key not found: %r' % key)
               if len(v) > 1:
                   raise KeyError('Multiple values match %r: %r' % (key, v))
>>  raise KeyError('Key not found: %r' % key)
KeyError: "Key not found: 'email'"
HTTP_USER_AGENT	'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MRA 4.3 (build 01218); .NET CLR 1.1.4322)'
webob._parsed_post_vars	(MultiDict([('login', 'ahusernames9'), ('fullname', 'ahusernames9'), ('password1', '******'), ('password2', '******'), ('s', 'Sign up')]), <FakeCGIBody at 0xb9d7c5ec viewing MultiDict([('lo...p')])>)
1319533989000000 1320153507000000
#1417 defect dread dread ckan-sprint-2011-10-24 closed fixed Browser language detection doesn't work

In Firefox:

  • Edit | Preferences | Content | Languages - put 'de' at top language
  • Clear cookies
  • Load thedatahub.org
  • it should be in German, but it's in the default language - English.
1319539010000000 1319651617000000
#1418 defect dread dread ckan-sprint-2011-10-28 closed duplicate If default language is not 'en' then you can't switch language to 'en'

e.g. cz.ckan.net has lang=cz but when you click on Home page link "English" it flashes message "Language has been set to: English" but the rest of the page is still Czech.

1319539146000000 1319539691000000
#1421 defect dread dread ckan-sprint-2011-10-28 closed fixed Tag cloud recipe broken

home page doesn't have c.facets setup.

Only affects 1.5 branch - ok 1.4.3.1. Was broken in cset:05f60f2d26e8.

1319638575000000 1319639311000000
#1422 enhancement dread dread closed fixed Remove proxy & etag caching

Rufus decided this was more trouble than it was worth.

Removes ckan/lib/cache.py

1319639663000000 1319645722000000
#1426 defect dread dread ckan-sprint-2011-10-28 closed invalid Validation errors cause 409 not 400 response

Validation error is incorrect arguments, so should be a "400 Bad Request" response. "409 Conflict" is for when there is a conflict creating two resources with the same identifier. Yet we get 409 for all validation errors.

1319708619000000 1319709236000000
#1427 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Tags validation needs to limit length to 100 characters

The tag name is limited in length in the db table to 100 characters. If you try and create a longer one you get an exception. This should cause a ValidationError? and tell the user this sensibly.

1319708698000000 1319709925000000
#1428 defect dread dread ckan-sprint-2011-11-07 closed fixed config options not used or documented

Need to ensure config options in the deployment.ini_tmpl match up with doc/configuration.rst. Remove any unused ones too.

1319803594000000 1319804773000000
#1431 defect dread dread ckan-v1.5 closed fixed Captcha field - foreign chars cause exception

During registering a user, the user inputs foreign chars into the captcha field.

URL: http://thedatahub.org/user/register
...
Module ckan.lib.captcha:22 in check_recaptcha
<<                                     remoteip=client_ip_address,
                                          challenge=recaptcha_challenge_field,
                                          response=recaptcha_response_field))
           f = urllib2.urlopen(recaptcha_server_name, params)
           data = f.read()
>>  response=recaptcha_response_field))
Module urllib:1267 in urlencode
<<          for k, v in query:
                   k = quote_plus(str(k))
                   v = quote_plus(str(v))
                   l.append(k + '=' + v)
           else:
>>  v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 0: ordinal not in range(128)
1320078849000000 1320084104000000
#1437 defect dread dread ckan-sprint-2011-11-07 closed fixed JSONP parameter in Action API

Action API needs JSONP support - be able to return responses encapsulated in a function of a supplied name. This is important for remote sites running javascript to interact with a CKAN site.

Specifying the callback parameter is the way we've achieved JSONP with the RESTful and Search APIs. It should work like this:

curl http://test.ckan.net/api/action/package_show?callback=jsoncallback -d '{"id": "fd788e57-dce4-481c-832d-497235bf9f78"}'

Or maybe the callback should be specified in the payload in the context or data_dict?

(My understanding is that CORS is similar - when more browsers support it, can we drop JSONP?)

1320161088000000 1320173795000000
#1440 defect dread dread ckan-sprint-2011-11-07 closed fixed 'paster db load' doesn't update search index

paster db load ckan.pgdump will load in the database but not update the search index, no we're using SOLR.

implementation

We can't tack on 'rebuild' to this command, because it needs a 'db upgrade' first. I'm tempted to make 'db load' do the upgrade and rebuild all in one command.

1320232415000000 1320235084000000
#1441 defect dread dread closed fixed Use local ckan logo image

Using local image rather than the same image at assets.okfn.org reduces dependency on another server that may be down, slow or remote. James agrees.

1320234597000000 1320235062000000
#1442 defect dread dread ckan-sprint-2011-11-07 closed fixed You can't GET then POST a dataset as new

If you try to GET a dataset, purge it then POST it again, it fails checking authz for the group:

Module ckan.controllers.api:278 in create
<<                  (register, subregister))
               try:
                   response_data = action(context, data_dict)
                   location = None
                   if "id" in data_dict:
>>  response_data = action(context, data_dict)
Module ckan.logic.action.create:240 in package_create_rest
<<      api = context.get('api_version') or '1'
       
           check_access('package_create_rest', context, data_dict)
       
           dictized_package = package_api_to_dict(data_dict, context)
>>  check_access('package_create_rest', context, data_dict)
Module ckan.logic:129 in check_access
<<          #    log.debug('Valid API key needed to make changes')
               #    raise NotAuthorized
               logic_authorization = new_authz.is_authorized(action, context, data_dict)
               if not logic_authorization['success']:
                   msg = logic_authorization.get('msg','')
>>  logic_authorization = new_authz.is_authorized(action, context, data_dict)
Module ckan.new_authz:16 in is_authorized
<<      auth_function = _get_auth_function(action)
           if auth_function:
               return auth_function(context, data_dict)
           else:
               raise ValueError(_('Authorization function not found: %s' % action))
>>  return auth_function(context, data_dict)
Module ckan.logic.auth.create:115 in package_create_rest
<<          return {'success': False, 'msg': _('Valid API key needed to create a package')}
       
           return package_create(context, data_dict)
       
       def group_create_rest(context, data_dict):
>>  return package_create(context, data_dict)
Module ckan.logic.auth.create:15 in package_create
<<      else:
               
               check2 = check_group_auth(context,data_dict)
               if not check2:
                   return {'success': False, 'msg': _('User %s not authorized to edit these groups') % str(user)}
>>  check2 = check_group_auth(context,data_dict)
Module ckan.logic.auth.create:90 in check_group_auth
<<      groups = set()
           for group_dict in group_dicts:
               id = group_dict.get('id')
               if not id:
                   continue
>>  id = group_dict.get('id')
AttributeError: 'unicode' object has no attribute 'get'

(using RESTful Model API)

The reason is the POST is expecting the group to be expressed as a dict, not the list of names that GET returns.

1320263318000000 1320275623000000
#1443 defect dread dread ckan-sprint-2011-11-07 closed fixed language detection

If language detected doesn't match any of the offered CKAN languages, you get the default language, but see an error flash up for every page, saying:

Could not change language to 'None': IOError: [Errno 2] No translation file found for domain: 'ckan'

Affects 1.5b only.

1320425588000000 1320432511000000
#1444 defect dread dread ckan-future closed wontfix Language detection defaults to en_us?

Using curl you always get English, even if ckan.locale_default=de. Find out why. (1.5b)

Maybe we should disable locale detection for this release if lots of people's browsers don't have it set correctly (e.g. chinese with us internet explorer)

1320432634000000 1340190852000000
#1452 enhancement dread dread closed wontfix Offer detected browser language, rather than auto-switch

There are issues with setting the site's language according to the browser detection:

  • Many browsers have the language setting incorrectly set. e.g. people who download US versions of Mozilla, pirated US copies in the far east.
  • Although thedatahub.org has a worldwide audience, many CKAN instances are designed for a particular country's citizens. So Open Berlin should probably be in German as default for all.

I suggest the site should have a default language saved in the config. The browser language *should* be detected, and that prompts a flash message offering to change to that language. And if you change language that is saved in the cookie (as we currently do).

Then do some testing to see if this suits people.

1320682652000000 1340190587000000
#1454 enhancement dread dread ckan-sprint-2011-11-21 closed fixed postgres search code/data removal

We made SOLR the main and only supported way to do CKAN search three months ago (#1275), but I notice that we still have remnants of the postgres full-text search hanging around:

  • model code (ckan/model/search_index.py)
  • table in the database (package_search)

The presence of these still create sqlalchemy warnings and add to our db, (& db backups etc), so it would be good to get rid of them if we are not using them. John says this can probably be removed

1320838999000000 1320849527000000
#1470 defect dread amercader ckan-sprint-2011-11-21 closed fixed Check user name in the profile form 1321446143000000 1324473955000000
#1473 defect dread dread ckan-sprint-2011-11-21 closed fixed max name length validators missing

Max length of package name and version field is 100 characters, but the validators seem to enforce this, although in the web interface the javascript limits typing in these fields, so it just affects the API really.

1321631452000000 1321872717000000
#1479 defect dread dread ckan-sprint-2011-12-05 closed fixed Can't edit a user with a unicode email address
  1. Register User with an email address with a unicode char (e.g. u'\u044e')
  2. View the User in the UI (/user/) or with 'user_show' Action API

Exception:

Module ckan.controllers.user:98 in read
<<          try:
                   user_dict = get_action('user_show')(context,data_dict)
               except NotFound:
                   h.redirect_to(controller='user', action='login', id=None)
>>  user_dict = get_action('user_show')(context,data_dict)
Module ckan.logic.action.get:488 in user_show
<<      check_access('user_show',context, data_dict)
       
           user_dict = user_dictize(user_obj,context)
       
           if not (Authorizer().is_sysadmin(unicode(user)) or user == user_obj.name):
>>  user_dict = user_dictize(user_obj,context)
Module ckan.lib.dictization.model_dictize:189 in user_dictize
<<      
           result_dict['display_name'] = user.display_name
           result_dict['email_hash'] = user.email_hash
           result_dict['number_of_edits'] = user.number_of_edits()
           result_dict['number_administered_packages'] = user.number_administered_packages()
>>  result_dict['email_hash'] = user.email_hash
Module ckan.model.user:59 in email_hash
<<          if self.email:
                   e = self.email.strip().lower()
               return hashlib.md5(e).hexdigest()
               
           def get_reference_preferred_for_uri(self):
>>  return hashlib.md5(e).hexdigest()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u044e' in position 17: ordinal not in range(128)
1321960486000000 1321961592000000
#1480 enhancement dread dread ckan-sprint-2011-12-05 closed fixed Munge functions in util

munge_title_to_name, munge_tag v. useful for client software importing things into CKAN. I want it for ONS.

1321965837000000 1321978546000000
#1484 defect dread dread ckan-sprint-2011-12-05 closed fixed Creating package with duplicate tags causes exception

e.g.

curl http://localhost:5000/api/action/package_create -d '{"name":"test5", "tags":[{"name": "tag6"}, {"name": "tag6"}]}' -H 'Authorization:tester'

Also broken in Web UI (adding a package).

Affects API and Web UI in 1.5 and I think all versions from 1.4.1

1322052046000000 1322059006000000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Note: See TracReports for help on using and creating reports.