This might well be a bot, but not POSTing the email field should not cause an exception - it should be 400 and then the sysadmin doesn't get these nags.

http://www.publicdata.eu/user/register
...
Module ckan.controllers.user:79 in register
<<              c.login = request.params.getone('login')
                   c.fullname = request.params.getone('fullname')
                   c.email = request.params.getone('email')
                   if not model.User.check_name_available(c.login):
                       h.flash_error(_("That username is not available."))
>>  c.email = request.params.getone('email')
Module webob.multidict:335 in getone
<<          values were found.
               """
               return self._decode_value(self.multi.getone(self._encode_key(key)))
       
           def mixed(self):
>>  return self._decode_value(self.multi.getone(self._encode_key(key)))
Module webob.multidict:99 in getone
<<          v = self.getall(key)
               if not v:
                   raise KeyError('Key not found: %r' % key)
               if len(v) > 1:
                   raise KeyError('Multiple values match %r: %r' % (key, v))
>>  raise KeyError('Key not found: %r' % key)
KeyError: "Key not found: 'email'"

HTTP_USER_AGENT	'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MRA 4.3 (build 01218); .NET CLR 1.1.4322)'
webob._parsed_post_vars	(MultiDict([('login', 'ahusernames9'), ('fullname', 'ahusernames9'), ('password1', '******'), ('password2', '******'), ('s', 'Sign up')]), <FakeCGIBody at 0xb9d7c5ec viewing MultiDict([('lo...p')])>)

e.g. cz.ckan.net has lang=cz but when you click on Home page link "English" it flashes message "Language has been set to: English" but the rest of the page is still Czech.

Currently can run as a celery task launched by paster command.

Still have to add a CKAN plugin so that QA will run when a resource is added or URL changed, and then update the extension controller/templates to look for QA data from the task_status table instead of directly on the resource.

Rufus decided this was more trouble than it was worth.

Removes ckan/lib/cache.py

ckan-discuss discussion suggests changes to the 'openness' indicator ( ​http://lists.okfn.org/pipermail/ckan-discuss/2011-October/001786.html )

Dataset view page:

• If there is an explicit but non-OKD compliant license, such as CC-BY-NC, then this should be stated explicitly, perhaps: “This dataset is Not Open. License: Creative Commons Attribution Noncommerical. This is not an open license as it does not meet the Open Knowledge Definition.”
• If the license is marked as “Other::License Not Specified”, then this should be stated explicitly, perhaps: “This dataset is Not Open. It is published without an explicit license, the publisher reserves all rights to the dataset.”
• 3. If the license field was left empty by the contributor of the Data Hub record, then again this should be stated explicitly, perhaps: “This dataset is Not Open. The license of this dataset is unknown or unspecified. Start an enquiry on IsItOpenData? »
• There is a bug so that non-open licenses doesn't have an openness notice.
• If downloadable resources are not available, this should not affect 'openness' - check this has been removed.

Validation error is incorrect arguments, so should be a "400 Bad Request" response. "409 Conflict" is for when there is a conflict creating two resources with the same identifier. Yet we get 409 for all validation errors.

Need to ensure config options in the deployment.ini_tmpl match up with doc/configuration.rst. Remove any unused ones too.

On some occasions (apparently random), the documents indexed in a specific SOLR core get mixed with different site_ids.

E.g: We look for all documents in the testing.iatiregistry.org core, faceted by site_id. We would expect all documents to have site_id = iati_testing, but some of them have site_id = iatiregistry.org

​http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=*:*&fq=+state:active&facet=true&facet.field=site_id

<lst name="facet_fields">
<lst name="site_id">
<int name="iati_testing">265</int>
<int name="iatiregistry.org">255</int>
</lst>
</lst>

If we compare one of the records which disappeared from the "iati_testing" site_id in both the production and testing SOLR cores of the server, the records are exactly the same, including the indexed_ts property:

​http://okfn-solr.fry-it.com:8080/solr/iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id

​http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id

Note that the response from the URLs shown may vary, as the testing site could have been reindexed.

Super ticket: #1190

A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)

Decide which extensions are important.

Many actions in the Action API require parameters. What params are needed should be listed and checked. Because currently, if you get them wrong you simply get a useless 500 error.

Currently they are listed in the docs, extracted from the code manually.

So you could GET /action/api/package_list to receive not only the help text, but a list of arguments.

And if you send an extra or missing argument then an intelligent error message can be returned.

implementation

How about some sort of decorator on the action function:

@logic_params(id, offset, limit)
def get_package_list(context, data_dict):
    ...

This would do the param checking, and is there a way to extract these params from the function? Or do a registration of the logic function?

I'd certainly like to keep the list of the list of params for the function with the function, for ease of reading the code.

Another good thing would be to pass in the params named as themselves, rather than having them contained in the data_dict.

paster db load ckan.pgdump will load in the database but not update the search index, no we're using SOLR.

implementation

We can't tack on 'rebuild' to this command, because it needs a 'db upgrade' first. I'm tempted to make 'db load' do the upgrade and rebuild all in one command.

If you try to GET a dataset, purge it then POST it again, it fails checking authz for the group:

Module ckan.controllers.api:278 in create
<<                  (register, subregister))
               try:
                   response_data = action(context, data_dict)
                   location = None
                   if "id" in data_dict:
>>  response_data = action(context, data_dict)
Module ckan.logic.action.create:240 in package_create_rest
<<      api = context.get('api_version') or '1'
       
           check_access('package_create_rest', context, data_dict)
       
           dictized_package = package_api_to_dict(data_dict, context)
>>  check_access('package_create_rest', context, data_dict)
Module ckan.logic:129 in check_access
<<          #    log.debug('Valid API key needed to make changes')
               #    raise NotAuthorized
               logic_authorization = new_authz.is_authorized(action, context, data_dict)
               if not logic_authorization['success']:
                   msg = logic_authorization.get('msg','')
>>  logic_authorization = new_authz.is_authorized(action, context, data_dict)
Module ckan.new_authz:16 in is_authorized
<<      auth_function = _get_auth_function(action)
           if auth_function:
               return auth_function(context, data_dict)
           else:
               raise ValueError(_('Authorization function not found: %s' % action))
>>  return auth_function(context, data_dict)
Module ckan.logic.auth.create:115 in package_create_rest
<<          return {'success': False, 'msg': _('Valid API key needed to create a package')}
       
           return package_create(context, data_dict)
       
       def group_create_rest(context, data_dict):
>>  return package_create(context, data_dict)
Module ckan.logic.auth.create:15 in package_create
<<      else:
               
               check2 = check_group_auth(context,data_dict)
               if not check2:
                   return {'success': False, 'msg': _('User %s not authorized to edit these groups') % str(user)}
>>  check2 = check_group_auth(context,data_dict)
Module ckan.logic.auth.create:90 in check_group_auth
<<      groups = set()
           for group_dict in group_dicts:
               id = group_dict.get('id')
               if not id:
                   continue
>>  id = group_dict.get('id')
AttributeError: 'unicode' object has no attribute 'get'

(using RESTful Model API)

The reason is the POST is expecting the group to be expressed as a dict, not the list of names that GET returns.

Using curl you always get English, even if ckan.locale_default=de. Find out why. (1.5b)

Maybe we should disable locale detection for this release if lots of people's browsers don't have it set correctly (e.g. chinese with us internet explorer)

Super ticket: #1602 (Data Previewer v2)

We already have first pass of Data Explorer that was released as part of #1357.

• Merging ​https://github.com/okfn/dataexplorer with ​http://github.com/okfn/recline and developing it
  • ​https://github.com/okfn/recline/issues

Tickets include (* indicates improvement over current explorer)

• Core Backbone Models representing Dataset and Tabular data: ​https://github.com/okfn/recline/issues/10
• New theme: ​https://github.com/okfn/recline/issues/22
• Read-only mode: ​https://github.com/okfn/recline/issues/17
• Introduce hash navigation / state support (*): ​https://github.com/okfn/recline/issues/19
• Re-enable editing in DataTable? (*): ​https://github.com/okfn/recline/issues/13
• [super] DataTable? view (in Backbone) (*): ​https://github.com/okfn/recline/issues/14
• DataExplorer? parent view (*): ​https://github.com/okfn/recline/issues/12
• Simple graph widget using flot: ​https://github.com/okfn/recline/issues/11

Est: 10-15d (should be broken down -- partly is in recline issues)

Celery is awkward to deply, need to find a way to do it more simply. i.e using celery-pylons and supervisor. A modified version of celery-pylons may be the best solution. 1d

See ​http://wiki.ckan.org/Dataset_and_Resource_Pages_Proposal

• Create a collapsible section for the README
• Reduce the number of things on the page
• Tidy up groups and tags in the sidebar
• Resource view becomes a list of search results

(additional)

• Resource navbar item links to first resource
• Resource navbar has a dropdown menu of all resources in the dataset
• Edit and Authz navbar items should be in a separate logical group to the right

Very rough summary: create more info section and move stuff out (relayout page e.g. README, tags etc)

There are issues with setting the site's language according to the browser detection:

• Many browsers have the language setting incorrectly set. e.g. people who download US versions of Mozilla, pirated US copies in the far east.

• Although thedatahub.org has a worldwide audience, many CKAN instances are designed for a particular country's citizens. So Open Berlin should probably be in German as default for all.

I suggest the site should have a default language saved in the config. The browser language *should* be detected, and that prompts a flash message offering to change to that language. And if you change language that is saved in the cookie (as we currently do).

Then do some testing to see if this suits people.

We made SOLR the main and only supported way to do CKAN search three months ago (#1275), but I notice that we still have remnants of the postgres full-text search hanging around:

• model code (ckan/model/search_index.py)
• table in the database (package_search)

The presence of these still create sqlalchemy warnings and add to our db, (& db backups etc), so it would be good to get rid of them if we are not using them. John says this can probably be removed

If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those.

Super ticket: #1151 (viewing geo data)

E.g. preview of ​http://thedatahub.org/dataset/louisvillecrime should bring up a map

Current extensions documentation needs some work: ​http://docs.ckan.org/en/latest/plugins.html

• Queue extension section may now be out of date (?)
• Think about how it integrates with ​https://github.com/okfn/ckanext-example (especially tutorial and example extension)
• Document all plugin points (auto-extract from CKAN source??)

With the 1.5 package release, CKAN instances over-write each other's indexes. Adria has a fix.

I'd like to see a new CKAN 1.5.1 release with the Solr changes merged and the Solr site ID set up as part of the install ckan.site_id = dgu-os.okfn.org. I'd also like the ability to search on ID and to have the ID directly visible on the package page.

We need to decide how to do this upgrade without affecting existing CKAN solr indexes.

The current harvesting implementation can only have one instance per server.

We could either:

• Accept this
• Change the RabbitMQ code to support multiple instances
• Switch the entire harvesting to use Celery
• Wait for James's redis based live feedback based system

When I delete a package from the pylons shell I need to remove that package's entry from the Solr search index. That doesn't happen at the moment.

It would be useful if the search-index rebuild command could take a list of package IDs to rebuild.

navl tests are being skipped unskip them!

This is the implementation of the final draft in

​http://wiki.ckan.org/Groups_Refactor

Steps to complete this will follow (somewhat) in the following order.

Change package_group table to membership. (#1478) 3d

• model migration. (done)
• simplify revisioning (done)
• Remove any sqlalchemy releations from groups. (done)
• change logic functions (done)

Single type per group (#1531)

• migration add type to group table. (done) 1d

Add capacity table (#1522)

• model migration. (done)
• add to logic functions.

Allow other objects to be added to groups. (#1531)

• change logic functions (done)

Adapt thedatahub.org to use new model.

• move auth groups over to groups/members
• change user_object roles to use groups? keep user object roles for time being.
• make create update package/group put the correct items in the group/members/capacity tables.
• work out how ui works.

--not doing the below as part of this super ticker now #1669.

Create model for dgu.

• adapt auth functions to use auth.
• adapt package_create/update group_create/update to fill in the correct capacties/members.

Adapt Iati, and potentially other custom instances.

Sub-tickets:

• #1279 Deprecate Authorization Groups
• #1198 Publisher hierarchy

1. Register User with an email address with a unicode char (e.g. u'\u044e')
2. View the User in the UI (/user/) or with 'user_show' Action API

Exception:

Module ckan.controllers.user:98 in read
<<          try:
                   user_dict = get_action('user_show')(context,data_dict)
               except NotFound:
                   h.redirect_to(controller='user', action='login', id=None)
>>  user_dict = get_action('user_show')(context,data_dict)
Module ckan.logic.action.get:488 in user_show
<<      check_access('user_show',context, data_dict)
       
           user_dict = user_dictize(user_obj,context)
       
           if not (Authorizer().is_sysadmin(unicode(user)) or user == user_obj.name):
>>  user_dict = user_dictize(user_obj,context)
Module ckan.lib.dictization.model_dictize:189 in user_dictize
<<      
           result_dict['display_name'] = user.display_name
           result_dict['email_hash'] = user.email_hash
           result_dict['number_of_edits'] = user.number_of_edits()
           result_dict['number_administered_packages'] = user.number_administered_packages()
>>  result_dict['email_hash'] = user.email_hash
Module ckan.model.user:59 in email_hash
<<          if self.email:
                   e = self.email.strip().lower()
               return hashlib.md5(e).hexdigest()
               
           def get_reference_preferred_for_uri(self):
>>  return hashlib.md5(e).hexdigest()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u044e' in position 17: ordinal not in range(128)

The license is not displayed any more, unless there are resources e.g. ​http://test.ckan.net/dataset/dtest#section-basic-information is OGL, but that's not shown.

The error reports do not have a "value". The original task in the task status table is not made and it tries to make a new task, but can't.

Issue: why is the original task_status entry not made?

kindly: there is a chance of course that the task is quicker than the plugin to get to the point where it needs to update the table.

kindly: so we may have to find a way to update the task status before we run the task.

kindly: I think you can supply a task_id to celery. i.e we make our own guid as the task id. So we can use that one to populate the task status.

We want to be able to change form depending on package type or group type.

This is dependent on a type field being added to the Package and the Group.

ordering on homepage by name instead of group count

ckanext-example needs updating for CKAN 1.5:

• theme changes
• new forms

About: 'ckanext-exampletheme' was created in Spring 2011 as an example CKAN extension that showed how to customise the look & operation of CKAN. This moved to github and renamed 'ckanext-example'.

Sean spotted that some strings need internationalising, such as flash messages. Generally all strings that appear on the web front-end should be internationalised.

Particular places that need i18n:

• ckan/controllers/admin
• ckan/lib/base

Note: there are some exceptions, such as in i18n.py, very obscure error messages and stuff that only appears on the API.

NB: there is a cost in making a string internationalisable (all our volunteers have to translate it), so we should not be too zealous. We should also look at the i18n/ckan.pot to see if any of the existing strings can be reused.

You can't delete things from the SOLR search index if the ckan.site_id and ckan.site_url are blank.

Should assert that one of these are set up.

An API call that returns a user's public activity stream in JSON format.

We really shouldn't have lots of benign stuff printed on stderr when we run every command. It confuses users and hides important stuff.

It would be useful to be able to manipulate packages using the paster command-line interface. In particular listing, reading and deleting packages. Seeing which packages were deleted too.

Most of the different pagination links are broken and point to wrong locations. e.g:

• Dataset listings in group page:

​http://test.ckan.net/group/bibliographic

• Tag letters in tag listing:

​http://test.ckan.net/tag

• User listing:

​http://test.ckan.net/user

• Revision listing:

​http://test.ckan.net/revision

This is most probably related with:

​https://github.com/okfn/ckan/commit/1772a5c

Reasons: more people familiar with Git, bigger Open Source community on Github.

Keeping track of the move here: ​https://docs.google.com/spreadsheet/ccc?key=0AvCDqUH8jVN8dGNVNEpXLWk3UTlJZXhFMjBPbXVtRmc&hl=en_GB#gid=0

This query caused ckan to except because ckan/controllers/api.py doesn't catch SearchError? and SearchQueryError?:

curl http://localhost:5000/api/action/package_search -d '{"sort": "metadata_modified"}'

Group edit dataset add form needs some work

• Dataset name is not cleared when you add
• No way to remove item from list of datasets to be added if I make a mistake
• (2nd Apr 2012) It now seems that option to add multiple datasets at once has disappeared (perhaps during the CSS/HTML refactor ...)

e.g. ​http://thedatahub.org/dataset/osm%402011-07-12T12%3A16%3A47.590358 gives:

This is the current revision of this dataset, as edited at 2011-07-12 12:16.

but it should say the date 2011-07-12 (which is the data being displayed).

The problem is looking at PackageRevision?, when it might be a tag or extra.

Add a function to logic.action.get that returns a user's public activity stream as a list of dicts.

Richard Cyganiak: I often see a problem where half the system thinks that I'm logged in, while the other half insists I'm not. Logout then login again may fix it.

David Read: I think I've seen it several times that the "My Account" link is given (suggesting you're logged in) but when you click it it asks you to log in, which means you weren't.

We want to add "activity streams" (chronological lists of activities) to CKAN.

Activity streams standard: ​http://activitystrea.ms/

Super branch where the feature was initially developed (this has now been merged into master): ​https://github.com/okfn/ckan/tree/feature-1515-activity-streams

Etherpad: ​http://ckan.okfnpad.org/notifications (most relevant parts inlined here)

Friedrich's mockup: ​http://datahub.pudo.org/pudo

Tasks

Strikethrough means the task is done.

• New domain objects Activity and ActivityStream? [2d] #1298

• Migrate existing data into activity streams model [3.5d] #1510

• Generate activities when objects are saved [4d] #1298

• Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] #1511

• API call for getting a user's public activity stream as JSON [0.5d] #1495

• API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] #1494

• Render a user's activity stream on her user page [3d] #1496 (needs review/page design work)

• Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.

• Logic functions for getting activity streams for packages, groups, and users.

• API calls for getting activity streams of packages, groups, and users as JSON.

• Rendering activity streams for packages and groups etc. into their pages. (done but disabled pending page design work)

• API calls for inserting events into activity streams. (Still need to handle activity details in the API, ticketed).

• Conform to activitystrea.ms standard

• Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page.

• Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams

• Email notifications

• Extension point for extensions to register themselves as renderers for different types of activity

• Deploy to UAT [0.5d]

• Docs and announce [1d]