CKAN: Ticket Query

#1334: "AttributeError: 'NoneType' object has no attribute 'name'" exception when logging in

amercader — Tue, 13 Sep 2011 10:25:08 GMT

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'

#1206: "Content-Type json" header scuppers package POST

dread — Wed, 29 Jun 2011 10:45:48 GMT

Compare these two requests to create a package:

curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'Content-Type: application/json' -H 'X-CKAN-API-KEY: tester'
curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'X-CKAN-API-KEY: tester'

The second one gets the payload through (ckan log):

Retrieving request params: UnicodeMultiDict([('{name:"test"}', u'')])

But the first one causes a ServerError? because the payload (name:"test") doesn't make it to request.POST or request.params:

Retrieving request params: UnicodeMultiDict([])

The only difference is the "ContentType?: application/json" header, which seems a reasonable thing to include. Javascript lib backbone.js (for example) inserts this automatically.

So why does this header cause the payload to not get through to the request object?

#1395: "Import Error: cannot import name UnicodeMultiDict" when installing ckan from source

seanh — Thu, 13 Oct 2011 15:36:23 GMT

At the paster db init command when installing ckan from source I get the error "Import Error: cannot import name UnicodeMultiDict?" (happens with both ckan 1.4.2 and today's latest bleeding edge code, on Ubuntu 10.04.3).

UnicodeMultiDict? has been removed in a recent version of python-webob, and the pip install ... lucid_missing.txt causes a too-new version of python-webob to be installed into ckan's virtualenv (the new webob gets installed as a dependency of formalchemy).

I manually did pip uninstall webob and then ran paster db init again and it worked.

#795: "Match up" GEMINI2 "metadata point of contact" with registered entities

johnbywater — Mon, 08 Nov 2010 14:53:45 GMT

TM wants to know how this will be "matched up". I don't know whether this always the data provider, or sometimes the publisher.

#897: "New package" highlighted in Navigator when searching

dread — Mon, 10 Jan 2011 08:40:15 GMT

On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted.

#1735: "Unsaved changes" warning briefly when you click "Save"

dread — Tue, 31 Jan 2012 11:30:20 GMT

Super ticket #1744

If you edit a dataset, simply add a tag and click on "Save" then you get the message "You have unsaved changed. Make sure to click 'Save Changes' below before leaving this page." inserted whilst the submission takes place. (This is because the click moves focus away from the tag field). This (reasonably common) edge-case could usefully be squashed.

Also note typo: "unsaved changed"

#1194: "Welcome back" message for newly registered user

dread — Mon, 20 Jun 2011 09:49:52 GMT

Register a new user (/user/register)
It redirects you to user page and displays "Welcome back" even though you are a new user.

#1486: "allow_partial_update" option does not work with resource lists of packages

seanh — Wed, 23 Nov 2011 15:15:18 GMT

Call ckan.logic.action.update:package_update()
Pass in a context dict with "allow_partial_update":True
Pass in a partial data dict containing only the package's ID and those fields that you want to update (e.g. title).

=> All of the package's resource will have their statuses changed to "deleted".

The "allow_partial_update" option should apply to the resource lists of packages, but it does not.

#935: "groups" field bug in IATI

pudo — Sat, 29 Jan 2011 12:08:35 GMT

An internal server error was reported when trying to create a new package (not when editing):

WebApp? Error: <class 'sqlalchemy.exceptions.IntegrityError?'>: (IntegrityError?) null value in column "name" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'}

The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the "groups" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server.

Possible causes include:

Stale pyc file
Threading issue
Dependency issue

As the problem is gone for now, it cannot be further investigated atm.

#1626: 'About CKAN' page update

dread — Tue, 10 Jan 2012 14:20:36 GMT

thedatahub.org/about contains info that is very general to the CKAN and really quite technical. The text should be changed to be both specific to thedatahub.org and provide the context in a non-technical way. It should be easy to customise the About page to be appropriate for say new-york.ckan.net - a bit of info about who runs it, plus the general stuff about CKAN powering it and it was written by OKF to further open data.

#1710: 'Announce' email list for thedatahub

dread — Thu, 26 Jan 2012 12:00:59 GMT

We want to make email announcements to users of thedatahub.org.

The most simple way to do this is to get a dump of the email addresses and subscribe/invite them to a mailman list.

Different ways to do it:

'paster dump-users-csv' creates a CSV file. Apache serves up the directory using password protection. Is this secure enough?
CKAN admin pages provides a button to produce the CSV and download it. A config option may be necessary to enable this (disabled by default).

Something more advanced in the future could be:

CKAN tells mailman about new subscribers using the mailman web interface. Could be brittle.
Mailman doesn't have an Internet API, but it does have a python interface. So CKAN could tell mailman about new subscribers using a python proxy. There may be one out there, or maybe easy to do. But annoying to have to maintain another process running on machine.
CKAN sends out the emails itself. A sysadmin is given a form. Users have an opt in/out option in their account. But receiving emails is a sensitive thing so it would need to be right first time, and there are several risks to this approach.

#1042: 'Ckanext' split-up

dread — Wed, 16 Mar 2011 16:45:07 GMT

It's not good to have ckanext doing lots of different things with different dependencies. Split it off into:

ckanext-importlib

etc.

And then deprecate the ckanext repo itself.

#824: 'Getdata' moved out of CKAN

dread — Wed, 24 Nov 2010 10:45:07 GMT

Data importers should use API instead of object model directly:

saves load on server
much better error checking and authentication
eating our own dogfood (tests the API well)

#1318: 'Register user' doesn't log you in

dread — Wed, 07 Sep 2011 17:16:23 GMT

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

#542: 'Show and Tell' page customization using cookie + js

rgrp — Tue, 31 Aug 2010 08:47:12 GMT

To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: http://www.ibm.com/developerworks/web/library/wa-rails2/

In the first place we just need to apply this to:

You are logged in as
Package/group edit/authz tabs

#1586: 'Sort by' field for package search

dread — Wed, 21 Dec 2011 14:59:48 GMT

We default to sort packages search results by "search rank", but we can change it to be "name", "title", "metadata_modified" date, "metadata_created" date. It works in the API now, so let's have (say) a drop-down in the Web UI for this.

Follows on from #191.

#2692: 'add your own data' link in no search results wrong

shevski — Tue, 17 Jul 2012 16:00:03 GMT

the "add your own data" link in "Try another search term, browse the datasets below or add your own data." here http://s031.okserver.org:2375/dataset?q=stuff doesn't take you to add a dataset, just back to the search/datasets page

#1308: 'create package' user case failure

dread — Mon, 05 Sep 2011 10:45:00 GMT

On the CKAN home page there is no suggestion that you can add or register a dataset. The 'Register a dataset' option has been removed at some point! (only visible to users that have registered - why would they do that when the benefits aren't described?)

#983: 'db upgrade' creates system priviledges

dread — Sat, 12 Feb 2011 13:44:25 GMT

(should leave priviledges alone)

#1346: 'get_authorization_group_dict not found' exception

dread — Thu, 15 Sep 2011 10:19:34 GMT

This stems from a feature to allow you to prefill fields in the authorizationgroup creation form.

e.g. http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it.

#1440: 'paster db load' doesn't update search index

dread — Wed, 02 Nov 2011 11:13:35 GMT

paster db load ckan.pgdump will load in the database but not update the search index, no we're using SOLR.

implementation

We can't tack on 'rebuild' to this command, because it needs a 'db upgrade' first. I'm tempted to make 'db load' do the upgrade and rebuild all in one command.

#1493: 'search-index rebuild/clear' doesn't work if no ckan.site_id

dread — Mon, 28 Nov 2011 12:47:02 GMT

You can't delete things from the SOLR search index if the ckan.site_id and ckan.site_url are blank.

Should assert that one of these are set up.

#2847: 'show inactive sources' checkbox displaying incorrectly

shevski — Mon, 13 Aug 2012 13:25:22 GMT

On harvest page: http://s031.okserver.org:2375/harvest the checkbox appears above the text and not clear what it refers to/

Beter to relocate to the left (as on publicdata.eu) see screenshots attached.

#101: (Improved) Download support

rgrp — Tue, 11 Aug 2009 11:07:46 GMT

Given a url download from that url to disk
Given a package use download url to download to disk

Once on disk should uncompress (if necessary).

Details

Should download either to tmp directory (defaulted in config) or a specified path on disk
In download would like to support as many types of target urls as possible:
1. Single files (usually compressed filesets but could be individual files)
2. Index pages listing files to download
3. Revision control repositories
4. ...

This will obviously be a significant amount of work. Suggest:

Start off just supporting the first option.
Have something like a plugin system to handle new target types
Reuse existing work (e.g. easy_install and pip handle standard vcs such as svn, hg, git etc)

Cost: 1d

#1098: --ckan-migration tests not initialised correctly

dread — Thu, 21 Apr 2011 09:15:36 GMT

Only tests with failing --ckan-migration fail, due to authz not being initialised.

#2684: .js leaving forms warning

toby — Tue, 17 Jul 2012 07:34:37 GMT

I think it would be nice to have all form pages monitor their forms - excluding search ones - and if a field has changed then warn the user that they have not saved the form data but are leaving the page.

This would help with things like changing resources on the add dataset resource form etc.

#1316: /api/action uncaught exception if missing necessary parameter

dread — Wed, 07 Sep 2011 12:07:04 GMT

curl http://ckan.net/api/action/package_show -d '{"name": "uk-quango-data"}'

This gives 500 error due to KeyError? (it looked for the 'id' param), rather than 400.

#2868: 1.8 SOLR error

ross — Thu, 16 Aug 2012 10:58:16 GMT

From: <thedatahub.org@…> Date: Wed, Aug 15, 2012 at 8:34 AM Subject: WebApp? Error: <class 'ckan.lib.search.common.SearchError?'>: SOLR returned an error running query: {'sort': 'metadata_modified desc', 'fq': u'groups:"welcome" capacity:"public" +site_id:"www.ckan.net" +state:active', 'facet.mincount': 1, 'rows': 21, 'facet.limit': '50', 'facet': 'true', 'q': '*:*', 'start': -20, 'wt': 'json', 'fl': 'id data_dict'} Error: "'start' parameter cannot be negative" To: ckan-sysadmin@…, kindly@…

URL: http://thedatahub.org/feeds/group/welcome.atom?page=0 Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:155 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:131 in call << r'\1', oldpath)

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:239 in call << try:

res = WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

res = WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.feed:180 in group << data_dictfq? = 'groups:"%s"' % id

item_count, results = _package_search(data_dict)

navigation_urls = self._navigation_urls(params,

item_count, results = _package_search(data_dict)

Module ckan.controllers.feed:57 in _package_search << # package_search action modifies the data_dict, so keep our copy intact.

query = get_action('package_search')(context, data_dict.copy())

return querycount?, queryresults?

query = get_action('package_search')(context, data_dict.copy())

Module ckan.logic.action.get:1130 in package_search << query = search.query_for(model.Package)

query.run(data_dict)

for package in query.results:

query.run(data_dict)

Module ckan.lib.search.query:350 in run << except SolrException?, e:

raise SearchError?('SOLR returned an error running query: %r Error: %r' %

(query, e.reason))

try:

data = json.loads(solr_response)

(query, e.reason))

SearchError?: SOLR returned an error running query: {'sort': 'metadata_modified desc', 'fq': u'groups:"welcome" capacity:"public" +site_id:"www.ckan.net" +state:active', 'facet.mincount': 1, 'rows': 21, 'facet.limit': '50', 'facet': 'true', 'q': '*:*', 'start': -20, 'wt': 'json', 'fl': 'id data_dict'} Error: "'start' parameter cannot be negative" CGI Variables CKAN_CURRENT_URL '/feeds/group/welcome.atom?page%3D0' CKAN_LANG 'ja' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' PATH_INFO '/feeds/group/welcome.atom' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/ja/feeds/group/welcome.atom' QUERY_STRING 'page=0' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '33139' REQUEST_METHOD 'GET' REQUEST_URI '/ja/feeds/group/welcome.atom?page=0' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '<address>Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7f5312a51650> beaker.cache <beaker.cache.CacheManager? object at 0x7f5312a51790> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7f5312a516d0>> beaker.session {'_accessed_time': 1345016040.884951, '_creation_time': 1345016040.884951} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: >, ) paste.parsed_dict_querystring (MultiDict?([('page', '0')]), 'page=0') paste.parsed_querystring ([('page', '0')], 'page=0') paste.registry <paste.registry.Registry object at 0x7f5302eef7d0> paste.throw_errors True pylons.action_method <bound method FeedController?.group of <ckan.controllers.feed.FeedController? object at 0x7f52f0341750>> pylons.controller <ckan.controllers.feed.FeedController? object at 0x7f52f0341750> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7f52f0341450> pylons.routes_dict {'action': u'group', 'controller': u'feed', 'id': u'welcome'} repoze.who.logger <logging.Logger object at 0x7f53127b2b50> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 139994769488336>, 'friendlyform': <FriendlyFormPlugin? 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin? 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>} routes.route <routes.route.Route object at 0x7f5312961e90> routes.url <routes.util.URLGenerator object at 0x7f52f0341210> webob._parsed_query_vars (GET([('page', '0')]), 'page=0') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52ed4873f0> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7f52f0341210>, {'action': u'group', 'controller': u'feed', 'id': u'welcome'})

#2866: 1.8 template error

ross — Thu, 16 Aug 2012 10:57:21 GMT

URL: http://thedatahub.org/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8 Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:155 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:131 in call << r'\1', oldpath)

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:239 in call << try:

res = WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

res = WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.user:409 in perform_reset << c.reset_key = request.params.get('key')

if not mailer.verify_reset_link(user_obj, c.reset_key):

h.flash_error(_('Invalid reset key. Please try again.')) abort(403)

if not mailer.verify_reset_link(user_obj, c.reset_key):

Module ckan.lib.mailer:100 in verify_reset_link << if not user.reset_key or len(user.reset_key) < 5:

return False

return key.strip() == user.reset_key

return key.strip() == user.reset_key

AttributeError?: 'NoneType?' object has no attribute 'strip' CGI Variables CKAN_CURRENT_URL '/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%253Db4c2d03fa8' CKAN_LANG 'it' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' PATH_INFO '/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/it/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d' QUERY_STRING 'key%3Db4c2d03fa8' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '54634' REQUEST_METHOD 'GET' REQUEST_URI '/it/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '<address>Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7f5312a51650> beaker.cache <beaker.cache.CacheManager? object at 0x7f5312a51790> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7f5312a516d0>> beaker.session {'_accessed_time': 1345066159.520708, '_creation_time': 1345066159.520708} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: >, ) paste.parsed_dict_querystring (MultiDict?([('key=b4c2d03fa8', )]), 'key%3Db4c2d03fa8') paste.parsed_querystring ([('key=b4c2d03fa8', )], 'key%3Db4c2d03fa8') paste.registry <paste.registry.Registry object at 0x7f52eba637d0> paste.throw_errors True pylons.action_method <bound method UserController?.perform_reset of <ckan.controllers.user.UserController? object at 0x7f52e49f7a50>> pylons.controller <ckan.controllers.user.UserController? object at 0x7f52e49f7a50> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7f52f5649ad0> pylons.routes_dict {'action': u'perform_reset', 'controller': u'user', 'id': u'3086e91c-fe09-4a98-92e1-19de67a9ac9d'} repoze.who.logger <logging.Logger object at 0x7f53127b2b50> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 139994769488336>, 'friendlyform': <FriendlyFormPlugin? 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin? 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>} routes.route <routes.route.Route object at 0x7f5312961ad0> routes.url <routes.util.URLGenerator object at 0x7f52de7f6390> webob._parsed_query_vars (GET([('key=b4c2d03fa8', )]), 'key%3Db4c2d03fa8') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52fc4f1a08> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7f52de7f6390>, {'action': u'perform_reset', 'controller': u'user', 'id': u'3086e91c-fe09-4a98-92e1-19de67a9ac9d'})

#2865: 1.8 unicode error

ross — Thu, 16 Aug 2012 10:56:53 GMT

URL: http://thedatahub.org/el/tag/jutatt%C3%A1sok Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:155 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:131 in call << r'\1', oldpath)

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:239 in call << try:

res = WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

res = WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.template:30 in view << """

try:

return render(url)

except TemplateNotFound?:

if url.endswith('.html'):

return render(url)

Module ckan.lib.base:153 in render << try:

return cached_template(template_name, render_template,

loader_class=loader_class)

except ckan.exceptions.CkanUrlException?, e:

raise

loader_class=loader_class)

Module pylons.templating:249 in cached_template << return content

else:

return render_func()

return render_func()

Module ckan.lib.base:95 in render_template << template = globsapp_globals?.genshi_loader.load(template_name,

cls=loader_class)

stream = template.generate(globs)

cls=loader_class)

Module genshi.template.loader:223 in load << loadfunc = directory(loadfunc)

try:

filepath, filename, fileobj, uptodate = loadfunc(filename)

except IOError:

continue

filepath, filename, fileobj, uptodate = loadfunc(filename)

Module genshi.template.loader:286 in _load_from_directory << def _load_from_directory(filename):

filepath = os.path.join(path, filename) fileobj = open(filepath, 'U') mtime = os.path.getmtime(filepath) def _uptodate():

fileobj = open(filepath, 'U')

UnicodeEncodeError?: 'ascii' codec can't encode character u'\xe1' in position 73: ordinal not in range(128) CGI Variables CKAN_CURRENT_URL '/el/tag/jutatt%C3%A1sok' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CKAN_PAGE_CACHABLE True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' PATH_INFO '/el/tag/jutatt\xc3\xa1sok' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/el/tag/jutatt\xc3\xa1sok' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '49520' REQUEST_METHOD 'GET' REQUEST_URI '/el/tag/jutatt%c3%a1sok' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '<address>Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7f5312a51650> beaker.cache <beaker.cache.CacheManager? object at 0x7f5312a51790> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7f5312a516d0>> beaker.session {'_accessed_time': 1345113298.778472, '_creation_time': 1345113298.778472} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: >, ) paste.registry <paste.registry.Registry object at 0x7f5309add610> paste.throw_errors True pylons.action_method <bound method TemplateController?.view of <ckan.controllers.template.TemplateController? object at 0x7f52b0ea2d10>> pylons.controller <ckan.controllers.template.TemplateController? object at 0x7f52b0ea2d10> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7f52b0ea2b90> pylons.routes_dict {'url': u'el/tag/jutatt\xe1sok', 'action': u'view', 'controller': u'template'} repoze.who.logger <logging.Logger object at 0x7f53127b2b50> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 139994769488336>, 'friendlyform': <FriendlyFormPlugin? 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin? 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>} routes.route <routes.route.Route object at 0x7f531296ea10> routes.url <routes.util.URLGenerator object at 0x7f5309add810> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52d4004990> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7f5309add810>, {'url': u'el/tag/jutatt\xe1sok', 'action': u'view', 'controller': u'template'})

#1772: 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST]

icmurray — Mon, 06 Feb 2012 14:51:42 GMT

#1773: 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]

icmurray — Mon, 06 Feb 2012 14:55:30 GMT

#1774: 22. Mixed Licenses MUST PHASE 2

icmurray — Mon, 06 Feb 2012 14:55:53 GMT

#1770: 3. Themes/Taxonomy as well as tags WONT PHASE 2

icmurray — Mon, 06 Feb 2012 14:50:34 GMT

#1771: 32. Browse by Tags PHASE 1 MUST

icmurray — Mon, 06 Feb 2012 14:51:02 GMT

#1376: 404 raised when approving package

kindly — Sat, 08 Oct 2011 15:43:09 GMT

Regression found by updating datacatalogs.

#2601: 500 Server Error when editing datasets with organizations plugins enabled

seanh — Tue, 26 Jun 2012 14:59:14 GMT

Add the organizations and organizations_dataset plugins to ini file, run ckan, login, create an organization, create a dataset and put in organization, edit the dataset and set organization to none, try to edit the dataset again (you must be sysadmin to do this) and get a 500:

Error - <class 'genshi.template.eval.UndefinedError'>: [] has no member named "get"
URL: http://127.0.0.1:5000/dataset/edit/fooset
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 162 in __call__
  app_iter = self.application(environ, sr_checker)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__
  return self.wrap_app(environ, session_start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 221 in __call__
  res = WSGIController.__call__(self, environ, start_response)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File '/home/seanh/Projects/ckan/ckan/ckan/controllers/package.py', line 499 in edit
  extra_vars=vars)
File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 148 in render
  return cached_template(template_name, render_template, loader_class=loader_class)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
  return render_func()
File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 101 in render_template
  return literal(stream.render(method=method, encoding=None, strip_whitespace=True))
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/core.py', line 183 in render
  return encode(generator, method=method, encoding=encoding, out=out)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 57 in encode
  return _encode(''.join(list(iterator)))
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 339 in __call__
  for kind, data, pos in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 670 in __call__
  for kind, data, pos in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 771 in __call__
  for kind, data, pos in chain(stream, [(None, None, None)]):
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 586 in __call__
  for ev in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/core.py', line 288 in _ensure
  for event in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 605 in _include
  for event in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/markup.py', line 327 in _match
  for event in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 545 in _flatten
  for kind, data, pos in stream:
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/directives.py', line 169 in _generate
  attrs = _eval_expr(self.expr, ctxt, vars)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 277 in _eval_expr
  retval = expr.evaluate(ctxt)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/eval.py', line 178 in evaluate
  return eval(self.code, _globals, {'__data__': data})
File '/home/seanh/Projects/ckan/ckan/ckanext/organizations/templates/organization_package_form.html', line 110 in <Expression u"{'selected':'selected'} if organization.get('id','') == group['id'] else {}">
  <option value="${group['id']}" py:attrs="{'selected':'selected'} if organization.get('id','') == group['id'] else {}">${group['title']}</option>
File '/home/seanh/Projects/ckan/ckan/ckan/config/environment.py', line 248 in genshi_lookup_attr
  val = cls.undefined(key, owner=obj)
File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/eval.py', line 410 in undefined
  raise UndefinedError(key, owner=owner)
UndefinedError: [] has no member named "get"

#368: 500 Server error when creating package

anonymous — Tue, 20 Jul 2010 10:41:02 GMT

I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response:

"publish: New Zealand Coastline (new-zealand-coastline)" opening connection to nz.ckan.net... opened <- "POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n" <- "{\"name\":\"new-zealand-coastline\",\"title\":\"New Zealand Coastline\"}" -> "HTTP/1.1 500 Internal Server Error\r\n" -> "Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n" -> "Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n" -> "Vary: Accept-Encoding\r\n" -> "Content-Type: text/html; charset=iso-8859-1\r\n" -> "Cache-Control: proxy-revalidate\r\n" -> "Content-Length: 640\r\n" -> "Connection: close\r\n" -> "\r\n" reading 640 bytes... -> "<!DOCTYPE HTML PUBLIC \"-IETFDTD HTML 2.0EN\">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n<hr>\n<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80</address>\n</body></html>\n" read 640 bytes Conn close

The server should not respond with a 500 error if the package was created.

#1093: 500 errors on GET to api/rest/licenses

dread — Fri, 15 Apr 2011 10:11:01 GMT

CKAN gets its license list from a license service, which can be a local file, but is often the http://licenses.opendefinition.org/2.0/ckan_original server. This server is currently flakey, but I think we only request the list on start up. The problem is we query it much more often than required. It is queried for every request to api/rest/licenses, and we are returning lots of 500 errors when the license server is timing out.

#1769: 6. Publisher Registration Improvements MUST PHASE 1

icmurray — Mon, 06 Feb 2012 14:49:49 GMT

#2420: @depreciated decorator

toby — Thu, 24 May 2012 11:15:44 GMT

create a depreciated decorator to wrap functions we intend to remove

logs a warning maybe check that 'depreciated' is included in the docstring too

#1055: @search_related tests not running

dread — Thu, 24 Mar 2011 16:47:08 GMT

Tests marked decorated "@search_related" should only be run against postgresql, but in fact they don't get run at all.

#366: A flash message says why you are redirected to the 'user login' page

dread — Wed, 14 Jul 2010 17:20:58 GMT

You get redirected for a number of different reasons - often confuses me let alone the average user!

Cost: 1h

#26: A registered person creates their own tags for a package

johnbywater — Mon, 10 Jul 2006 17:09:11 GMT

#1157: A simple theme for publicdata.eu, CKAN

pudo — Wed, 25 May 2011 15:32:53 GMT

Change the visual style of CKAN to be more like these sites:

GitHub?
Quora
Google Projects

This does not include major UX work.

#1310: API GET of a revision that has been approved causes exception

dread — Mon, 05 Sep 2011 15:01:39 GMT

e.g. this particular revision on ckan.net: http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c (most others have not been approved) Causes 500 error.

#2942: API POST barfs on interesting Content-Type headers

dread — Tue, 25 Sep 2012 17:12:36 GMT

When POSTing to the API, if specified, the 'Content-Type' header must be blank or 'application/x-www-form-urlencoded'. Otherwise we get an error like: "Bad request - JSON Error: Could not extract request body data: Bad content type: \'; charset=utf-8\'""

The problem is that this is a very reasonable header to send. Indeed requests 0.14 sends this particular header.

This affects all versions of CKAN.

This is due to webob/requests.py:1248 being pretty basic.

#482: API Rate Limiting

johnbywater — Sat, 21 Aug 2010 20:32:36 GMT

#1174: API Representation Registry

pudo — Thu, 02 Jun 2011 21:43:12 GMT

As CKAN grows, more and more optional representations of packages will become used. Besides RDF (which is the motivation for this ticket), support for DSPL, home-grown XML, or CSV listings is imaginable.

To properly support this CKAN should have an extensible representation registry that can be extended when new output types become available (and without changing the API in the process). This needs to integrate in two places:

We need to add support for file format extensions to the package handler and REST API.
We need to add HTTP content type negotiation (Accept headers).

To support this we should have a registry with two registers:

to map format extensions to mime types (e.g. "json" -> "application/json")
to map mime types and entity types to converter functions (e.g. 8"application/json", ckan.model.Package) -> func(obj, mime_type) )

This should be set up on load_environment so that IConfigurer plugins can feed into it.

The registry should then first be added to _finish in the REST API (which needs to be refactored to be passed the {format} part of the URL if one is given. Based on the format part and HTTP headers, an appropriate representation can be generated by the registry and then be returned to the user.

Forwarding of requests to the regular WUI controllers with Accept headers set or a format specified can be implemented in a separate effort.

#377: API Should return JSON in all cases

glen@… — Tue, 27 Jul 2010 22:02:38 GMT

When the API returns any response it should always retunr JSON. For example a 404 response should return something like

{'status': 404, 'message': 'package somename not found'}

When errors are encountered when creating a package:

{'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be
blank', 'description':'license code must be an integer']}

Not authorised:

{'status': 503, 'message': 'You are not authorised to create this package.'}

The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created.

#1663: API call for creating an activity streams event

seanh — Tue, 17 Jan 2012 10:07:24 GMT

The create_activity() logic function already exists, but validation and authorization need to be added before it's hooked up to an API call.

#1495: API call for getting a user's public activity stream

seanh — Mon, 28 Nov 2011 15:52:29 GMT

An API call that returns a user's public activity stream in JSON format.

#1494: API call for getting a user's public activity stream as rendered rtext

seanh — Mon, 28 Nov 2011 15:50:47 GMT

This could be implemented as a separate API call, or the rendered text versions of the activities could be added into the JSON returned by the existing API call.

This requires setting up templates for rendering activity streams items and detail items as nice, human-readable text.

There are some open questions, e.g.: Do we want the entire activity stream rendered as a block of plain text? As HTML? Or do we want a list of JSON objects, where each object contains its textual and/or HTML representations as fields?

Activity stream items and their related detail items are separate objects that each have their own textual representations.

For a mockup of the kind of text messages we want, see:

http://datahub.pudo.org/pudo

but note that this ticket is just for creating the text snippets themselves, not rendering then in an HTML page or RSS feed. Also the mockup only show activity items and not their detail items.

#1637: API call for getting the list of activity detail items for a given activty stream item

seanh — Wed, 11 Jan 2012 18:00:17 GMT

(and add test cases for it)

#2198: API documentation is missing Storage Metadata API info

zydio — Tue, 28 Feb 2012 09:35:43 GMT

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on Storate Metadata API.

#1688: API for changing the permissions on an object

dread — Fri, 20 Jan 2012 12:16:49 GMT

We need an API for changing the authorization (permissions) on a model object.

This was neglected in #1253 because of various changes to authz were looming. But six months later it still hasn't been done.

Suggest this is just moving _add_user_object_role from ckan/lib/base.py to the logic layer, but no doubt it will be more complicated.

Estimate: 3 days

#1214: API improvements (following javascript use)

dread — Wed, 06 Jul 2011 18:00:22 GMT

Notes on the CKAN API from Aron:

Server returns text/html for errors even when "Accept: application/json" header is set.
DELETE package request requires a Content-Length? Should be 405 Method Not Allowed?
Tag returned as a JSON object when updating but as a string when requesting. Same with "extras" content.
How to remove key from "extras" object? Passing None as described in the docs is invalid JSON and using null doesn't work.
Tags return package names rather than ids when querying GET /tag/{id}
Doesn't return rendered_text property on package update.
Relationships use "object" key rather than id or package_id.

#667: API is slow

dread — Thu, 30 Sep 2010 22:20:28 GMT

See message and script: http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html

1500 requests shouldn't take hours.

#1798: API search in non-q fields has exception for unicode characters

dread — Thu, 16 Feb 2012 12:30:20 GMT

You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q.

For example:

http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93

This "N-dash" (Unicode character 2013) causes this exception:

Module ckan.controllers.api:460 in search
<<                      if ver in u'12':
                               # Otherwise, put all unrecognised ones into the q parameter
                               params = convert_legacy_parameters_to_solr(params)
                           query = query_for(model.Package)
                           results = query.run(params)
>>  params = convert_legacy_parameters_to_solr(params)
Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr
<<      for search_key in non_solr_params:
               value_obj = legacy_params[search_key]
               value = str(value_obj).replace('+', ' ')
               if search_key == 'all_fields':
                   if value:
>>  value = str(value_obj).replace('+', ' ')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128)

This problem affects CKAN 1.5 and 1.5.1 only.

#1001: API should use normal user credentials if available

rgrp — Wed, 23 Feb 2011 19:35:05 GMT

When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key.

The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController? to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...).

This is critical to incorporating any ajax editing into the frontend.

As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author).

#788: API user deletes harvest source

johnbywater — Mon, 08 Nov 2010 10:13:16 GMT

Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?).

#473: API user discovers correct header for sending API key

johnbywater — Fri, 20 Aug 2010 13:50:03 GMT

#471: API user sends API key in correctly named header

johnbywater — Fri, 20 Aug 2010 13:48:08 GMT

#472: API user sends API key in incorrectly named header

johnbywater — Fri, 20 Aug 2010 13:48:28 GMT

#123: Ability to edit Group in WUI

dread — Wed, 23 Sep 2009 12:14:01 GMT

Add Group editing page.

If no permissions to change group can't edit group. Also cannot view edit page.

Editable attributes: name, title, description

No preview needed

#848: Ability to hide extras fields on package read

pudo — Tue, 07 Dec 2010 09:31:56 GMT

for helsinki regional infoshare.

#906: Ability to search without accents for accented words

Stiivi — Thu, 13 Jan 2011 17:26:28 GMT

In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results.

This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents.

For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCIITRANSLIT).

#1051: Ability to set custom favicon

sebbacon — Mon, 21 Mar 2011 10:26:00 GMT

Allow deployers to set a URL pointing to their own favicon

#2409: Accept headers not handled correctly

markw — Tue, 22 May 2012 10:51:38 GMT

CKAN returns HTML when an Accept header should make it return RDF/XML:

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002237.html

#116: Access Control - edited in wui

dread — Tue, 15 Sep 2009 17:13:22 GMT

Based on a section of AccessControl design: WUI gives controls to user and administrator to change permissions on a package.

#117: Access Control - group core functionality

dread — Fri, 18 Sep 2009 10:55:33 GMT

Based on a section of AccessControl design: Group reads and edits are controlled by access control. WUI and REST interfaces covered.

#114: Access Control - model

dread — Wed, 09 Sep 2009 11:08:47 GMT

Create in the model basic operation of Access Control.

roles table

name | context | action
-----------------------
admin| package | edit
admin| package | edit-permissions
admin| package | read
editor| package | update
editor| package | read
reader| package | read
This data is set-up on db init and will have no interface.

user-roles table:

username | context_type | objectid | role
rgrp     | system  | n/a | admin
visitor  | package | * | reader
bob      | package | geonames | admin
visitor  | package | geonames | editor
visitor  | package | geonames | reader
john     | group | ukgov | admin
dread    | group | ukgov | editor
visitor  | group | ukgov | reader
This data will be added when someone is given permissions for the system, a package or a group.

Pseudo code:

class Package

def is_allowed(name, action):

is_allowed(name, action, context=self)

class Group

def is_allowed(name, action):

is_allowed(name, action, context=self)

def is_allowed(name, action, context=None): name: string - a username or IP for 'visitor'

action: string - 'read', 'edit', 'delete', 'edit-permissions' context: object - a Group or a Package or None (which means system)

# look up user from name. # look up in user-roles table what roles this user has for this context. # for each roles, look up in roles table what actions are allowed. # return True if action is allowed, else False.

#115: Access Control - wui constrained by model

dread — Tue, 15 Sep 2009 17:06:34 GMT

Based on a section of AccessControl design: Reading and writing to packages in the WUI and REST API are now dependent on the authz tables.

#93: Access control for packages

rgrp — Fri, 31 Jul 2009 15:44:09 GMT

Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: AccessControl.

Cost: 8d

#1517: Accessing resource view sometimes auto-downloads the resource

zephod — Mon, 05 Dec 2011 18:27:27 GMT

Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file.

For example, being here: http://test.ckan.net/dataset/europeana-lod and clicking on any of the links to the first resource.

#614: Acquire local copy of XML schema for remote metadata documents

johnbywater — Sat, 11 Sep 2010 15:31:17 GMT

#615: Acquire local implementation of UKLP schematron

johnbywater — Sat, 11 Sep 2010 15:32:54 GMT

We "just" need a schematron.validate(metadata) method. :-)

#1335: Action API - basic

dread — Tue, 13 Sep 2011 10:33:41 GMT

Provide an API to the logic layer at /api/action.

#1504: Action API given blank data causes exception

dread — Thu, 01 Dec 2011 16:25:48 GMT

This is what caused the exception:

curl http://localhost:5000/api/action/package_search -d='{"q": "osm"}'
The problem is the content-type is formencoding, the '=' is the first character of the request, and so internally the data_dict is set to ' ', which needs to be a dictionary.

#1746: Activity streams pagination

seanh — Sun, 05 Feb 2012 12:54:48 GMT

Currently user, package and group activity streams only return the most recent 15 activities, even though all activities are kept in the db. Do we want to add pagination - to both the API and the HTML pages - to support retrieving older activities?

#1743: Activity streams still link to deleted things

seanh — Thu, 02 Feb 2012 19:46:56 GMT

For example, after deleting a dataset old activity stream events will still hyperlink to that dataset.

#1299: Activity streams table migration

kindly — Tue, 30 Aug 2011 09:30:35 GMT

Migrate tables for activity streams

#2642: Adapt harvest frontend to new theme

amercader — Mon, 09 Jul 2012 15:03:16 GMT

#1727: Add "current" property to harvest objects

amercader — Tue, 31 Jan 2012 10:54:14 GMT

Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new "current" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes:

Model update (See #1726)
Update queries for getting harvest sources' datasets
Update ckanext-csw queries for getting the objects
Update inspire and ckan harvesters
Document new field

#121: Add 'Group' to main menu

dread — Wed, 23 Sep 2009 08:33:47 GMT

And associated page to browse group.

#158: Add 'extras' to REST API

dread — Fri, 16 Oct 2009 14:27:39 GMT

Ensure you can add/change/delete extra fields from packages using the REST API.
Update API docs to cover this.
On Package get, ensure separation of extra attributes from other attributes.

What is the meaning of posting an incomplete dictionary (updating a package)? Option 1 - if field not present, leave field alone

Delete field if field value is None
Option 2 - user must supply all fields

Delete field if field absent
Prefer option 1. Document this.

Follows on from ticket:152

#2754: Add + icon to two buttons

shevski — Wed, 25 Jul 2012 14:06:17 GMT

Specifically the 'add resource' button and the 'add item' button in related section

#2594: Add EU27 to EC ODP geographical coverage field

johnglover — Mon, 25 Jun 2012 15:36:16 GMT

EU27 option seems to have been removed at some point, needs to be added back to geographical coverage vocabulary and tested.

#1409: Add Gravatars to user profiles

zephod — Fri, 21 Oct 2011 12:39:33 GMT

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

[Reopened: Additional] Use gravatars in the HTML generated by helpers.py to create lists of users. Eg. on user/list and dataset/history.

#1410: Add Gravatars to user profiles

zephod — Fri, 21 Oct 2011 12:39:34 GMT

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

#122: Add Group authz page

dread — Wed, 23 Sep 2009 08:34:02 GMT

#1383: Add IResourceChange IResourceUrlChange Interfaces.

kindly — Tue, 11 Oct 2011 14:09:13 GMT

Need to add way for extensions to trigger based on resource change and just resource url change.

#74: Add Is It Open links to package pages

rgrp — Fri, 17 Jul 2009 10:56:58 GMT

On each package page in the openness field if the package is not open provide a link to is it open create enquiry page.

Details:

may want to add to the url a query parameter indicating which package this relates to e.g. ?ckan-package=...
http://isitopen.ckan.net/enquiry/create/?ckan-package=...
do we provide link even if package is open (e.g. so that people can make enquiries about e.g. adding open data/content buttons ...)
Cost: 0.5h

#1197: Add JavaScript guide for CKAN

timmcnamara — Thu, 23 Jun 2011 03:50:32 GMT

A new library, guiders.js, has been open sourced that seems to be great at unobtrusively introducing new users to features of websites.

The library drives Optimizely's website. The GitHub repo is here: https://github.com/jeff-optimizely/Guiders-JS

Some examples:

#1159: Add RDFa to CKAN package pages

pudo — Wed, 25 May 2011 15:38:27 GMT

RDFa can be used as a simple way to expose linked data or at least sameAs the API version of the data but we need to make sure we do not expose anything different from the "official" representation.

#92: Add RDFa to package pages

rgrp — Fri, 31 Jul 2009 14:18:47 GMT

#2604: Add Requests as requirement for tests

amercader — Wed, 27 Jun 2012 09:43:49 GMT

Requests (http://python-requests.org) is really useful to deal with http requests and saves a lot of time.

I propose we include it as a requirement for the tests for the time being, and even for core if necessary at a later stage.

#31: Add a comment to a package

rgrp — Mon, 04 Sep 2006 12:55:05 GMT

As a

Registered user

I want to

Add a comment to a package

Notes

The comment should be tagged with the username of the person making the comment
Should we allow comments per release as well as per package? No for the time being.
Comments will be 'flat' and not threaded (at least for the present)
Comments should be plain text and so html should be escaped (in the future we might look to use some kind of formatter such as markdown)

#956: Add a resource group table.

kindly — Fri, 04 Feb 2011 12:05:12 GMT

We need a way to group resources by various different factors. The most important of which is timeseries.

The model will look like

Package <o2m> resource_group <o2m> resource

The use cases to this are outlined in this email. http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

#1754: Add activities for extras

seanh — Mon, 06 Feb 2012 11:38:38 GMT

When an extra is added to or removed from a dataset or is updated, an activity should go in the dataset's and user's activity streams.