{22} Trac tickets (2647 matches)

curl http://ckan.net/api/action/package_show -d '{"name": "uk-quango-data"}'

This gives 500 error due to KeyError? (it looked for the 'id' param), rather than 400.

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

It is very annoying that you have to wrap links in <...> and most people don't bother. Extend h.markdown in code and markdown in js to fix this.

(We should also support 'package' links etc in javascript?)

*ADDITIONAL*

• Note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
• Move the markdown api to /util/markdown

Focusing on improving Action API as the v3 API:

• have an optional parameter of the data_dict called "options". Options would contain items that would get passed into the context. e.g. {"options" : {"ref_package_by": "id"}}.
• instead of using API version to change the way packages are referenced, use the ref_package_by.
• All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'.
• Action API is v3 of api, replacement for v1 & v2. Default for most urls is still v1, but if url is /api/action then default to v3.

Next steps:

• Add search API (package, resource,
• Add Util API
• Clarify JSONP still applies
• Add doc strings, clarifying parameters

add extra fields to resource as outlined in

​http://wiki.ckan.net/Domain_Model/Resource

Specifically (all unicode strings except for those noted):

• name: a name for this resource (could be used in a ckan url)
• type: the type of the resource. One of: file | api | service | listing
• mimetype: standard mimetype (e.g. for zipped csv would be application/zip)
• mimetype_inner: mimetype of innermost object (so for example would be text/csv)
• size: [integer] size of the resource (content length) in bytes. Usually only relevant for resources of type file.
• last_modified: [datetime] the date when this resource's data was last modified (NB: not the date when the metadata was modified).
• cache_url: url for cache of object in ckan storage
• cache_last_updated: [datetime] timestamp when cached version was created
• webstore_url:
• webstore_last_updated: [datetime] timestamp when webstore was last updated

Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor.

A possible bug in CKAN when I tried deleting users using "paster --plugin=ckan user delete" command.

To reproduce the bug do the following:

1. Create a user with an ID (which in my case was a user's full name)

that contains non-unicode caracters like Norwegian "æ", "ø", or "å".

2. Make sure that you can see something like the example below:

(pyenv) rm@mycomputer:$ paster --plugin=ckan user Users: name=Rustæm

3. Then try deleting the user with following command:

(pyenv) rm@mycomputer:$ paster --plugin=ckan user delete "Rustæm"

You should now get a python encoding error. I know that this is quite rare case, but in our case it caused some trouble. Could you guys have a look at this bug?

CKAN ver. 1.3.3.

test_authz.py appears to test in great detail some very specific additional authz (related to total site lock-down it seems -- introduced I think for hri project).

I think there are simpler ways to get total site lockdown (use external auth!) and this test is slow and delicate (e.g. depends on specific words in templates). Suggest removing. If we don't remove we should at least refactor tests for access to certain pages to use a proper method of testing (e.g. agreed html comments in each page) rather than being depending on the presence of absence of specific wording.

We get this i18n error for the URL ​http://no.ckan.net/authorizationgroup (which runs 1.4.2 with Norwegian translation). I'm not sure why it occurs. Fixing it is low importance though, because this site doesn't use Authorization Group feature

Usually caused by bots. The user does not appear to be logged in.

WebApp Error: <type 'exceptions.IndexError'>: list index out of range
...
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'

cset:4b59ab34137d ckan/logic/action/get.py:

-            isopen = model.Package.get_license_register()[license_id].isopen()
-            result_dict['isopen'] = isopen
+            try:
+                isopen = model.Package.get_license_register()[license_id].isopen()
+                result_dict['isopen'] = isopen
+            except KeyError:
+                # TODO: create a log message this error?
+                result_dict['isopen'] = False 

This change hides problems with the license server and returns potentially incorrect values for openness.

This has been noted as 'temporary fix' but seems to be forgotten about, since it has been merged to default and gone into release 1.4.3.

I suggest the licenses are cached (I thought this was already the case when CKAN first requests them after start-up?). I suggest failure would return 503.

Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'

e.g. ​http://ckan.net/group/edit/history

Module ckan.controllers.group:135 in edit
<<          try:
                   old_data = get_action('group_show')(context, data_dict)
                   c.grouptitle = old_data.get('title')
                   c.groupname = old_data.get('name')
>>  old_data = get_action('group_show')(context, data_dict)
Module ckan.logic.action.get:347 in group_show
<<      if group is None:
               raise NotFound
       
           check_access('group_show',context, data_dict)
>>  raise NotFound

If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting ​http://localhost:5000/dataset).

Steps to reproduce:

• Change the solr_url parameter in the ini file to something non-existant
• Visit ​http://localhost:5000

Json decoding error on search, due to date in resources.

This stems from a feature to allow you to prefill fields in the authorizationgroup creation form.

e.g. ​http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it.

Tweaking UIs and polishing the work done in #1294.

## Edit Page ##

• [Done; cset:49e0b059b6f5] Hide upload if storage not enabled
• [Done; cset:81ff13b42dad] Make 'Add resource' links on edit page clearer and more button-like.
• [Done; cset:03904fc50cbf] Move edit form tabs to the left
• [Done; cset:fde26b87d6ce] API placeholder text should not link to a CSV file
• [Done] We used to autocomplete the format field - restore this functionality
  • Works on live. My local db has an empty 'resource_revision' table; maybe 'paster create-test-data' should insert something here?
• [Done; cset:1aab499d73db] Summary should contain 'name' (or, if unavailable, use description. Long term: description is to be deprecated)
• [Done] Remove/move instructions where possible (a lot could just be placeholder text on the elements) (other stuff could only show on hover)
  • NB: reveal_instructions.js already disabled on edit page
  • [RP] have now done this on create -- think we can leave on edit
• [Done; cset:755b1a622199] We have broken 'error' reporting for resource issues. Either need to move that into error summary or somehow not destroy it when we rewrite table on page load ...
• [Done; cset:7ad090cb7855] Add slideDown to Resource expanded view
• [Done; cset:d3595143e506] Need ability to delete resources
  • No thought yet given to uploaded resources. Long term re-think probable.
• [Done; cset:2430a68a59fc] Empty resource table to resemble view page ("no resources" text)

## General ##

• [Done; cset:cc0ad47293e5] Highlight the package tab I'm on (view/edit/history)
  • More generally, the presence of the #minornavigation bar is very inconsistent throughout the site.
• [Done; cset:d5babc3629cf] Remove 'license' filtered search from backend
• [Done] Move script stuff to footer of page

New js code running produced in the work following #1294 contains various string literals, and needs proper internationalisation.

See files:

• application.js
• templates.js

Furthermore, efforts can be made to tidy up strings as described in #1329.

The current search implementation uses the output of the the as_dict method of the domain Package object to update the index

​https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48

It also uses package_to_api1 in the SynchronousSearch? plugin:

​https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93

This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model).

The search should use the logic function to get the package properties:

get_action('package_show')(context,data_dict)

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).

At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs.

Suggest we write a script that automatedly generates reference documentation for the config from the source.

May be obsoleted by #277 (some config in db)

Deliver little user benefit and easier just to search and then see what is listed as open and with resources.

Implementation Details

Areas to fix:

• UI
• lib/search/...
• tests
• ...?

Cost

Est: 2h

When paster db clean is run, the search index should also be cleared.

We should ignore these but log the packages that are causing this to investigate.

SOLR search doesn't support searching for part of an extra field, but it does for other fields.

i.e. title="One Two Three" matches q=one AND q=title:one and geographic_coverage="England Scotland" matches q=England BUT NOT q=geographic_coverage:England

This problem emerged when we went to SOLR in #1275 (CKAN 1.5a). Tests were skipped.

This is could be a problem for DGU and maybe elsewhere.

From Ira: PDF of bugs and suggestions is attached.

This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in #1404.

DONE ?

• [dataset/edit]: Deletions are confirmed with a dialog box
• [dataset/edit]: Long URLs break the edit resources tab. Show name (& 'no name') with a pencil icon.
• [dataset/edit]: Rearrange the delete resources mechanism;
  • Use a little minus icon on the far left, followed by a pencil icon for editing.
  • The 'delete resource' button can become 'hide'
  • Deletions are confirmed with a dialog box
• [dataset/edit]: Rearrange the 'you have unsaved changes' warning;
  • Delete the little green plus sign on the right;
  • Try a flash banner instead (more general purpose & beautiful)
• [dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset.
• [dataset/view]: Subscribe button (& subscribe to history) doesn't work as expected. Modal dialog carrying instructions?
• [dataset/view]: Drop the 'add to a group>>' and 'add tag>>'. Needs inline tag-adder.
• [markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out.
• [dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels.
• [groups/add]: Hide unimportant fields
• [navigation] Add 3D bevel (@see github) to current tab highlight.
• [dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see ​http://groupspaces.com/a/group/
• [groups/view]: Show description and owner. To match dataset section.
• [groups/authz]: Tidy to match admin/authz (pretty buttons & spacing). Add navigation bar.
• [groups]: Entire section to use pretty buttons please.
• [about]: Get the about page working on test.ckan.org. (Works locally and on live).

WONTFIX ?

• [dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold. (wontfix)
• [groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'. (ticket #1403)
• ([groups]: (Investigate) Is it easy to implement a little search-groups feature?) (ticket #1403)
• [*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...) (wontfix: we need to rethink this mechanism; authorization refactor is coming)
• [general]: 'Authorization' refactor to 'Permissions' on nav-bars. (wontfix, further discussion. authorization might be less techy...)
• [search]: (Investigate) Can we quickly & easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets? (wontfix: high cost and maintenance overhead; need compelling user story)

Add a way to share datasets and see mentions of datasets on twitter (and elsewhere)

As a visitor I want to share a link to a dataset I have found. I also want to see how many others have shared this (or mentioned it).

• Sharing platform: simplest option is probably twitter but could generalize to e.g. sharethis system which supports, facebook, twitter etc etc.
  • Will need a config option for relevant API key e.g. ckan.sharethis.apikey
• Location: Put this at top of sidebar on dataset view (a better suggsetion?)
• Should show how many mentions / shares there are. (very important!)

Discussion

• (Tom and Ira and ...): do we really want this, versus e.g. proper follow extension?
• What about bringing conversation back in to CKAN. E.g. show all times this dataset (i.e. its url) was mentioned on twitter. If we do this I think this should be definite +1.
• Also, given our users, I think just doing twitter (identica) may be sufficient (how many people want to share links to datasets on facebook?)

Take the resource archiver (currently part of ckanext-qa) and make it into a standalone extension that works as a celery task.

e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English.

Regression found by updating datacatalogs.

Implement a banner across the top of the screen giving newbies a link to the about page, and making them feel welcome. See stackoverflow for reference?

While integrating the new theme and making a large number of UX tweaks, I've allowed the CSS file to bloat out of control with some very specific rules and subtle hacks, as well as a couple of clearfix <div> objects in the markup which isn't really good practice.

In general we want a logical CSS structure which is as easy to modify as possible. You should not find yourself being overwritten by highly specific rules all the time!

Actions

1. Clean up css
2. Add a config option ckan.template_head_end - arbitrary string inserted at end of <head> tag so you can add custom css etc (may also want to update theming docs to reflect possibility of using this)
3. (While we're at it): Add ckan.template_footer_end config option to deployment.ini_template. (This already exists, we just haven't told anybody)
4. The /users index page isn't linked anywhere. Clean it up and link it in the footer next to Groups etc.
5. package/new, package/edit, group/new and group/edit all include their forms differently. Use h.literal in all cases.
6. Fix the padding on minornavigation.

Background

David Read sent me the following as background:

22. basic theming is described here: ​http://docs.ckan.org/en/latest/theming.html

Deep customisation of the templates is done by creating an extension, with this as a template: ​https://bitbucket.org/okfn/ckanext-exampletheme/overview

That's what Augusto and the Brazilians have done ​http://alpha.dados.gov.br/dados/ and their code is here: hg clone ​http://dev.dados.gov.br/codigo/dev/tema-ckan <snip> Another thing of interest was us becoming compatible with Wordpress themes last December. Richard Pope worked with Rufus on this. I believe the base theme is 'twentyten' and I'm guessing that one would swap that css file with another to change theme, but I don't know. I don't know if this feature has been dropped since then. ​http://lists.okfn.org/pipermail/ckan-dev/2010-December/000073.html

Dave

Perhaps the importer deletes them before re-importing. We shouldn't have deleted resources, so let's investigate.

Most articles still refer and link to ckan.net, wiki.ckan.net and to packages

Superticket: #1343

Creation of accounts from OpenID causes large number of problems:

• No guarantee of other profile info
• Poor username (just from openid)

We therefore will:

1. Require creation of user account via register (no auto-creation of accounts via openid)
   • sidebar of register page should point to login page rather than openid signin

2. Permit association of an openid with an account ...
   • How: paste in your openid url (requires change to edit form to have this)
     • won't work for google with their weird openid urls
   • [future] Nicer way is to have login via openid while logged in (which association then happening in background)

3. Migration (for 1+2):
   • (?) Generate decent usernames for all existing accounts
   • As this is nontrivial suggest instead we allow editing of usernames (by account owner and sysadmins). This is useful in its own right and is a reasonable 80/20 solution.
     • This does present the problem of re-associating commits with the new username. Best solution to this would be to switch revisions to point to userid rather than username. See #1534

4. [optional] Change display_name to always be username
   • Change My Account to Username at top right of all pages (once we know usernames are short ...)
     • could do this straight away by truncating long usernames (e.g. truncate at 20 chars ...)
     • Also should we lose the icon?

Needs to update on:

1. language - It's broken on thedatahub.org if you change to German and then click on the CKAN icon top-left to go to thedatahub.org again.

AND

2. whether signed in or not - #1373

BUT not latest revisions (which is what it was)

Or get rid of etag caching on this page and others?

(Editing a dataset) If the second resource contains any validation error then it says "Resources: Package resource(s) incomplete" and "Resource 1:".

Super ticket: #1506

• Show member since
• Display email (in obfuscated form)
• List all owned datasets (in standard manner)
  • Will need change to logic layer to show datasets on user dict (this is useful for the API as well ...)
• (Re)move list of changes
  • Should have activity (once activity available)

WONTFIX:

• ?? Allow users to provide a home page, organization (requires migration)

• Possible sibling / requirement: #1397 (resource archiving)

• Setup test environment. 1d
• Make task to upload files to webstorer. 3d
  • This includes writing back webstore_url and updated date. 0.5d
  • Type guessing of file format. 1d
  • Make cron to upload all un webstored resources. 0.5d
• Testing for errors by running cron 2d
• Make nice way to deploy celery 1d.
• Documentation. 1d
• Deployment. 1d

• Create page on ​http://wiki.ckan.org/ about setting up a queue (this will merge into core docs when this is stable) - having a queue will now become a generic requirement for a bunch of extensions
• README with install instructions in ckanext-archiver
• Create an entry in ​http://wiki.ckan.org/Extensions
• Write a blog post

Plan to migrate from mercurial git

• Convert from mercurial to git
  • Map named branches to normal branches
  • Also, if possible map usernames for RP: rgrp <​http://rufuspollock.org/> (mercurial) -> rgrp <rufus.pollock@…>
• Place at ​http://github.com/okfn/ckan (can delete current repo there as an out of date mirror)

Process:

• Do trial run
• Announce conversion date / time
• Require everyone to have pushed all outstanding changes at that time
• Do conversion
• Test
• Announce on list and ckan.org/

[Refactored] :: Follows on from #1506 UX changes.

• Declutter the sidebar. No yellow box.
• Facets to go on the left, rather than the right. More logical flow.
• Did you know you can search by author? Probably not. Find a nice way of presenting extended search options.
• Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results).
  • Update that look-and-feel to include the new resource icons created in #1506

RSS 'subscribe' buttons appeared in many places on the site but were not very helpful. They took (confused) users pointed to the raw feed code, and Google Reader could not understand the feed. Safari, however, could interpret it correctly.

Their presentation needs to be clear and consistent; the RSS feed really needs testing in a variety of readers; and we need to decide exactly which items should get a feed. (Package updates? Groups?)

Give site user system administrative rights.

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

Create a new dataset, don't add any resources to the dataset yet, view the dataset in ckan, if you have firebug enabled you'll see the error "resource is undefined" at line 699 of application.js.

Set up tracking for user logins so that we have stats about how many active users of thedatahub exist want to be able to see who logged in the the last x months