PUT to /rest/api/package of {"name": "name", "resources": ["someurl.com/data"] } (i.e. resource is a string, not a dictionary) gives 500 error, when it should give a 400 error and helpful error message.

Exception:

Module ckan.controllers.rest:154 in create
<<              if register == 'package' and not subregister:
                       fs = ckan.forms.get_standard_fieldset()
                       request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data)
                       fs = fs.bind(model.Package, data=request_fa_dict, session=model.Session)
                   elif register == 'package' and subregister in model.PackageRelationship.get_all_types():
>>  request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data)
Module ckan.forms.package_dict:88 in edit_package_dict
<<                      for res_dict in value:
                               res_dict_str = {}
                               for key, value in res_dict.items():
                                   res_dict_str[str(key)] = value
                               resources.append(res_dict_str)
>>  for key, value in res_dict.items():
AttributeError: 'unicode' object has no attribute 'items'

Add link to RDF representation of a package in our Atom feed.

Several questions to answer (and see file in email). To respond to the Finnish team and copy all Q&A to the ckan-discuss list.

1 ) I have played with two different datasets to get the metadata from those imported to CKAN. The other dataset is in PCAXIS format and other one is in Excel format. Please find the scripts + sample files attached in a zip file that contains instructions too to run the scripts. I looked at import scripts written in object oriented manner and getdata samples in CKAN but it was easier for me too write these from scratch (copied the unicrap function from CKAN samples). If I run the PCAXIS import script I get Unicode decode error (see errors file). When I use the unicrap function then the error does not appear in logs. I assume this is some Python environment setting issue. Of course I would like to run the script without unicrap function to get scandic letter imported correctly to ckan(and displayed in WUI). In Excel case there is no scandic letter issue which is kind of strange but this probably relates how the Excel library handles encoding.

To be honest the whole unicode/encoding/decoding issue has been quite difficult to deal with and debug. I hope I would get proper environment configuration to handle scandic letters.

2 ) It seem in CKAN by default the localisation file contains 462 strings. The transifex link Rufus sent contains 400 strings. Is transifex the one we should use as localisation file ? Where are the rest 62 strings then fetched from or are they needed at all. I have found that as I have been using the .po file coming in CKAN source (which has API translations too) this also causes similar unicode problems when an error in API occurs (the API error messages containing scandic letters provide the similar UnicodeDecoding? errors as mentioned above). So, it seems the unicode issues appear in various cases. Maybe a proper environment settings will be able to prevent also these problems... If transifex is the one to use I can prepare it and maybe then you could upload it to fi.ckan.net as there will be also other people checking the translations. To run the test and see the API translation errors especially in pcaxis case I would recommend to install attached FI translations to your environment and include lang = fi configuration. It looks like the API errors does not come when running default lang = en.

3 ) Is it possible to change the license selection/options available in license field drop-down ? Where are these located and can these be modified locally ?

4) It looks like the values stored in package extra fields are not searchable. Is this intentional ? It looks more or less that all other fields can be searched using free-text search in CKAN.

5) How package dependencies are defined and managed ? For instance we want to import some packages in a manner that the other package is a child package of another. Does API support this and/or is this possible to do in WUI ?

Best Regards, Henri

And as a result timeout on e.g. the datahub ...

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Besides considering the media type of resources, it would also make sense to check for the presence of a SPARQL endpoint. SPARQL endpoints are recorded for more than 300 datasets on the Data Hub using the pseudo-type "api/sparql". A few more are recorded with the format "SPARQL". I suggest that datasets with such resources should also be considered for the fourth star.

Super: #1594

From Richard Cyganiak on the CKAN discuss list:

According to (most of) the definitions of the five-star scheme, the first star already requires an open license, so a dataset shouldn't get any stars at all if it's not under an open license.

manifest.cfg was not included in the pypi package, causing InvalidRepositoryError? when you do 'paster db init'.

Appears to affect most CKAN PyPI releases up to and including 1.4.3.

Currently stored in the python controller file. This is not the right place and putting it in db will allow editing by admins.

May also want to do this for the footer (also in the controller file).

Details:

• Will need to create a new domain object/db table. Suggest called miscellaneous and consist of key value pairs utilizing JSON type for values.

Cost: 2h.

PDEU should be in a mode where:

• Users cannot edit or create packages
• Users cannot sign up on their own
• AuthorizationGroups? and RevisionHistory? is completely gone from the UI
• Groups have been repurposed as Subjects

Spam entries have started to occur on ckan.net in the last few months. It would be useful to be able to purge these revisions -- that is permanently delete all changes to domain objects associated with those revision (one might also permanently delete that revision -- or alternatively simply mark it as purged).

In addition to providing this facility from the shell it would also be useful to be able to do this from the web interface (with associated restrictions on usage via an authorization controller of some sort).

Super #1669

Authz for normal users in publisher profile as per sysadmin

Super #1669

Analysis of all permissions for users given their publishers to ensure that we have covered all of the bases and haven't missed any use-cases.

Super #1669

Editor authz for publisher groups need to be implemented and tested.

Changes to publishers for customer

Super #1669

Authz for the publisher admin (who has permission to administer the publisher group themselves).

Provide some provisional user stories for the CMS/Blogging feature. Should include some of the functionality from wordpresser but be more generic.

1. Find some relationships.
2. Add them via API
3. Notify DGU of completion and point to exemplars

We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app)

1. Use contents from ckan package notes section (rgrp)
2. Update that section to be more guide-like (jwyg)

DOI = digital object identifier = ​http://www.doi.org/

As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way.

Details

• Probably implement as extension rather than core

Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr.

From DR at DGU

Basically get_groups() appears to cache its results, but this falls down when subsequent calls are in a different session. We get this when saving a dataset. get_groups() first gets called in the controller when it is trying to work out what permissions the user has. It later gets called, and by this time c.userobj is detached, so get_groups() fails.

I'm working around this by refreshing c.userobj (from c.user) before calling get_groups(), but I wonder if the get_groups caching could detect the detached session and bypass the cache in this case?

diff at ​https://gist.github.com/3003117

Against ckan-1.3.1, when I create an authz group called "administrators" and visit /authorizationgroup/edit/administrators, I am unable to add more than one user to it.

Each time I add additional users, the existing user on the list is replaced with the new one.

Intended outcome:

See resource url validation error when previewing package with a resources that has no url.

Recurring Error:

URL: http://127.0.0.1:5000/package/new
File 'pyenv-ckan/lib/python2.5/site-packages/weberror/evalexception.py', line 431 in respond
  app_iter = self.application(environ, detect_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/repoze/who/middleware.py', line 107 in __call__
  app_iter = app(environ, wrapper.wrap_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 152 in __call__
  return self.wrap_app(environ, session_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Routes-1.10.3-py2.5.egg/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File 'pyenv-ckan/src/ckan/ckan/lib/base.py', line 50 in __call__
  return WSGIController.__call__(self, environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 216 in new
  clear_session=True)
File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 357 in _render_edit_form
  edit_form_html = fs.render()
File 'pyenv-ckan/src/ckan/ckan/forms/builder.py', line 13 in render
  return render(self.form_template)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render
  format=format, namespace=kargs, **cache_args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render
  **options)
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 533 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate
File 'pyenv-ckan/src/ckan/ckan/templates/package/form_fields.html', line 52 in <Expression u'h.literal(field.render())'>
  ${h.literal(field.render())}
File 'pyenv-ckan/lib/python2.5/site-packages/FormAlchemy-1.3.4-py2.5.egg/formalchemy/fields.py', line 1217 in render
  return self.renderer.render(**opts)
File 'pyenv-ckan/src/ckan/ckan/forms/common.py', line 311 in render
  return render('package/form_resources')
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render
  format=format, namespace=kargs, **cache_args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render
  **options)
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 510 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate
File 'pyenv-ckan/src/ckan/ckan/templates/package/form_resources.html', line 14 in <Expression u"res['id'] if res else ''">
  <td><input name="${c.id}-${i}-id" type="hidden" value="${res['id'] if res else ''}" /></td>
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 335 in lookup_item
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 405 in undefined
UndefinedError: {'url': u'a', 'hash': u'a', 'description': u'a', 'format': u'a'} has no member named "id"

How to reproduce:

1. Go to package edit form, enter a package resource but without a URL.
2. Click preview.

e.g. ​http://datahub.io/dataset/ccc-gistemp/resource/80ebdbd6-d91b-4fef-9db4-d3dfbd7e868e

CKAN attempts to render the PDF as a datastore tabular date even though the mimetype of the PDF is correctly specified.

Follow up with NS and RP.

1. Viewing a dataset, hit 'Edit'
2. Click on the "Basic Information" tab (note: URL has suffix #section-basic-information
3. Click 'Save'
4. URL still has suffix #section-basic-information

Affects 1.5, 1.5.1, 1.5.2a

We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG.

If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button.

It would be nice to add similar popovers for other types of object in CKAN, e.g. resources, groups, organizations, tags...

When using the API, the apikey needs to be supplied in a header called 'Authorization'. Because some proxys / deployments use this header for other things, a configurable header was provided as an alternative, with default "X-CKAN-API-KEY".

Rufus suggests having *one* way for this. a) making this not configurable any more b) making X-CKAN-API-KEY the default

(keep Authorization allowed, but not documented, for backwards compatibility)

See ​http://wiki.ckan.net/Domain_Model especially section on v2.

• New domain model is planned but not yet finally agreed.
• Layer architecture is complete and implemented

When minified files are in use (set debug = false in ini file) the pin images for points don't show up in the data explorer map view when viewing a resource with geolocation data.

set debug = true will fix the problem by using unminified files instead but also turns on a bunch of debug stuff you wouldn't want on a production site.

middleware.py, around line 73, is where the decision to turn minified files on or off based on the debug setting is made. This can be hacked on a production site to run with unminified files but otherwise run on production mode thereby getting around the issue with the pins. But that doesn't fix the underlying bug.

As a sysadmin I should be able to view deleted groups/publishers (publisher profile) but I can't.

In addition, there don't seem to be tests for permissions of deleted publishers.

The pdf preview does not load in IE 9.

With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is:

• the less files should be compiled to css (main.css, not just main.debug.css)

This ticket is to provide a paster command which will compile the .less into the main.css file. The idea of the paster command is that it will be run by developers, and they will check in the resulting .css files. It will also provide a convenient mechanism for production scenarios with node.js installed on the production server, as they will be able to compile the .less there as well.

## Background

The .css files that need to be served are built using less. The css files that are generated *are* checked-in to the repo; but they are a build artifact. So the general workflow is:

• commit changes to the less files
• build the main.css file and commit

This paster command slots into the above workflow for convenience.

## Notes

• doc/frontend-development.rst
• bin/less

to run this paster command, Node will be required. So that dependency should be checked.

## The paster command

This is the proposed behaviour of the paster command:

• ensure custom.less does not exist
• for each colour in{fuchsia,green,maroon,red}:
  • generate a custom.less file for $colour. There's a paster command that does this already: `paster color <color name>.
  • generate the css from the less files. ie - the equivelant of running bin/less --production.
    • this will generate a main.css file, which should be renamed to $color.css.

• ensure custom.less does not exist
• generate the css from the less files. This will create a main.css with the default colour scheme.

• call the paster command that minifies css and js files. (This command does not exist yet, see #2950)

Instead of showing a notice, the password reset page throws an exception if the key parameter is missing:

Module ckan.controllers.user:329 in perform_reset
         c.reset_key = request.params.get('key')
               if not mailer.verify_reset_link(user_obj, c.reset_key):
                   h.flash_error(_('Invalid reset key. Please try again.'))
                   abort(403)
 if not mailer.verify_reset_link(user_obj, c.reset_key):
Module ckan.lib.mailer:100 in verify_reset_link
     if not user.reset_key or len(user.reset_key) < 5:
               return False
           return key.strip() == user.reset_key
 return key.strip() == user.reset_key
AttributeError: 'NoneType' object has no attribute 'strip'

Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param:

​http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8

instead of:

​http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8

You can register a user with password and (optional) email address. But if you forget the password you can't then log in again. We need a password reset facility that sends and email with a new password.

See the related sidebar box

​http://s031.okserver.org:2375/dataset/adur_district_spending

If a package has related items the first (or a random) related item should be passed into the snippet.

When getting a search results page (e.g. ​http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1) and then using the on-page links to page through results, the next link / numbered page links go to ​http://ckan.net/package/search?q=development&page=2 rather than ​http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1&page=2 meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings.

E.g. Pagination links on this page don't include groups=lodcloud ​http://thedatahub.org/dataset?groups=lodcloud

Not sure if related to #1501 (probably not)

1. Tag results could/should show number of associated packages
2. Package results should show title (and be in list)
3. Pagination should be able to be applied to other result sets than simply register listing (e.g. search results ...)
4. Revisions should be listed in a table with full details