{22} Trac tickets (2647 matches)

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Regarding the fifth star (is the dataset linked to others?). This cannot be automatically determined just by looking at the format. It either requires inspection of the actual data, or information about links in the metadata. As you're probably aware, we've established conventions for recording information on data links in CKAN [1], as part of the work of the lodcloud group on the Data Hub. Link information is captured for hundreds of datasets. I would claim that we have the majority of four-star datasets covered there, and hence you can determine if they should get the fifth star by checking for the presence of a links:xxx field.

• Update CKAN on DGU UAT to 1.5.1
• Deploy Celery
• Deploy QA extension

From Mark on the ckan-dev list:

In the table of broken links at <​http://thedatahub.org/qa/dataset/broken_resource_links>, the links as printed should be abridged to a fixed length (but link to the full URL). Otherwise the 'reason' column vanishes off the screen. Also a table would probably be better since as things stand the 'reasons' are not lined up

Some request exceptions are currently not being caught (see the celery log on thedatahub for examples)

• write importer for publisher info (currently in JSON format), add to ckanext-ecportal
• add group translations to translations table
• run on test server

• create importer for geo tags
• add translations to database
• translate tags in WUI
• test
• deploy
• document how to add a geo coverage field in ​http://docs.ckan.org/en/latest/forms.html

• probably needs a new converter, as needs to be usable via API as 'published_by'.

Some recent changes caused things to break in ckanext-ecportal:

• change from blueprint to bootstrap
• other style / css changes (eg: tables, resource view notes field ID, etc. Forms are also broken, but that will be addressed in #2262)
• change to going through db_to_form_schema on package_show

• get up to speed with DataExplorer? code (and dependencies like Backbone)

Finalised metadata model requires several changes to our existing model:

• new extra: alternative title
• new extra: identifier
• new extra: interop level
• new extra: version description
• new extra: dataset status
• dataset type: change to fixed vocab
• resource types: prefix with documentation/distribution/etc for clarity, add new resource types
• rename update frequency to accrual periodicity
• add new fixed vocab field: language coverage
• resources: missing issued date - add as new core field called 'created'

Also add functional tests for full metadata model.

• convert non-free fields to vocab fields (eg: interoperability_level)
• store values as JSON files in the ecportal repo
• translate output on dataset view page
• add tests

• the facet tags should only show free (non-vocab) tags
• decide on what fields to include in search facet list

• Already done for basic publisher forms (index, read, edit), but needs to be done for additional forms (in particular, form paths like 'organization_apply' that are hardcoded into the organizations controller).
• change organization mapping to /publisher

Datasets listed in search results have not gone through schema validation (but on the user page they do), so package_list snippet currently has to display both 'notes' and 'description'.

Quite a few of the current test dataset resources have their format incorrectly specified as text/html. Attempting to preview these results in the browser trying to automatically download the resource, so for now we will not preview this format.

When clicking 'add dataset' on a publisher/group page, the new dataset form should select the appropriate group.

• created repo for final list of instructions: ​http://github.com/okfn/ecportal-server-setup

Add caching config to ecportal-server-setup repository.

Server setup scripts need to have two distinct steps:

1) Download all files that aren't in RHEL repositories, create RPMs where necessary (python virtualenv).

2) Install from local files.

Fixes and updates to Recline / Datapreview for CKAN 2.0.

create a new test file ckan/model/test_user.py

add tests for the following three functions in ckan/model/user.py

number_of_edits, number_administered_packages, search

merged in in changeset 0046f83aedcf

Child tickets

• #1198 Publisher hierarchy
• #1050 Authz lib improvement and refactor of ckan/lib/authztool.py
• #1004 Group creation instructions missing
• #1099 Strange interactions between two browsers while playing with authz groups
• #1115 can have two authzgroups with the same name
• #1133 command line rights manipulation doesn't work
• #1138 minor navigations behave inconsistently

Old ticket description:

1. Change name of AuthzGroup? to UserGroup? to reflect what it is for

2. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
   • Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
   • Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

3. Change UserGroups? so that they can have a hierarchical structure,

More info on Hierarchy change

e.g. UserGroup? NHS contains the User nhsysadmin, as well as the UserGroups? SURREY and BERKS, which themselves contain users.

One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not.

Each Group has permissions over various objects.

A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively.

Algorithm:

possible(user, action, package):

if user has permission for action on package

or any of have that permission

or any of his groups group-children (but not user-children), and so on recursively have the permission.

Set language to Greek, flash message says 'Language set to: English', but page is now about half in Greek.

Set language back to English causes server error:

AttributeError?: 'NoneType?' object has no attribute 'path'

Module ckan.controllers.error:29 in document view

if original_request.path.startswith('/api'):

However going to a new page reveals that it's back to English

ckan.net appears to have either gone down or be running ultra slowly.

this means that ckan copies running locally on my machine run very slowly indeed.

is this behaviour desirable?

This command finds lots of ​http://~~~ckan.net references in python, html and javascript files:

find ~/pyenv/src \( -name "*.py" -or -name "*.html" -or -name "*.js" \) -print0 | xargs -0 -e grep --color -nH -e "​http://.*ckan.net"

output for reference:

/home/okfn/pyenv/src/ckan/ckan/init__.py:5:Network (CKAN) site: ​http://www.ckan.net. /home/okfn/pyenv/src/ckan/ckan/lib/create_test_data.py:346:<​http://ckan.net/> /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:3:DOMAIN = '​http://ckan.net' /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:4:CKAN_NAMESPACE = '​http://ckan.net/#' /home/okfn/pyenv/src/ckan/ckan/lib/talis.py:60: 'ckan':'​http://ckan.net/ns#', /home/okfn/pyenv/src/ckan/ckan/public/scripts/bookmarklet.js:2: f='​http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&title='+encodeURIComponent(document.title); /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:16: addtockan.src='​http://ckan.net/scripts/bookmarklet.js'; /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:27: <p><strong>Proper bookmarklet (compressed -- need to escape &amp;):</strong> <a href="javascript:(function(){f='​http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;title='+encodeURIComponent(document.title);if((n=document.getElementsByName('description')[0])&amp;&amp;(d=n.content)){f+='&amp;notes='+encodeURIComponent(d);}a=function(){if(!window.open(f)){location.href=f;}};if(/Firefox/.test(navigator.userAgent)){setTimeout(a,0)}else{a()}})()">Add to CKAN</a> /home/okfn/pyenv/src/ckan/ckan/templates/home/license.html:31: For convenience, all material - including all package, tag and revision information - is available in bulk, in the form of a full dump of the CKAN database. This (gzipped) dump file is updated daily and can be downloaded from <a href="​http://www.ckan.net/dump/">​http://www.ckan.net/dump/</a>. /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:71: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<​http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:137: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<​http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:447: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<​http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:458: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<​http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/functional/api/base.py:178: assert '"ckan_url": "​http://test.ckan.net/package/annakarenina"' in msg, msg /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:116: api e.g. ​http://ckan.net/api rather than ​http://ckan.net/api/rest) /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:261: :param base_location: default *​http://www.ckan.net/api* /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:267: base_location = '​http://www.ckan.net/api'

If I use my gmail openID to log into a CKAN instance, then my username is:

​https://www.google.com/accounts/o8/id?id=AItOawnduohQ5RgXdPJKHiq-SIPbvCBqUaERuEQ

This seems a bit odd.

With the default test data created by

paster db clean paster db init paster create-test-data

going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy

But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.

With the default test data created by

paster db clean paster db init paster create-test-data

going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy

But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.

I'm finding that if I try to take an action with insufficient credentials from a test then I often (but not always) get a 401 error, whereas in the browser I get redirected to the login page.

It's a bit worrying that the program in its test environment doesn't behave like it does in the browser.

It appears that the command

$ paster rights add russianfan admin warandpeace

has no effect, even though

$ paster rights remove russianfan admin warandpeace

works fine. This may be specific to something I've done, could someone confirm?

If it's the case more generally, then I'm assuming this behaviour is untested? Tests should probably be added.

To reproduce:

1. Enter details for new dataset without being logged in

2. Hit login page for just before step 2

3. Once logged in you end up back at the empty add dataset form (instead of expected 2nd step of dataset edit)

The language dropdown in Internet Explorer 7 behaves oddly. Essentially I think the problem is with the way that the dropdown decides to go into "drop-up mode" (e.g there aint enough screen space below the initial state so it has to drop up).

On the search results page when there are no more formats, groups or tags to display it should not display the show more links.

The user login / avatar with in the header really should be within another topbar, as it goes against convention to have the user information within the normal header.

They seem to be not exactly well thought out. Needs some thought on better display of the information that they contain.

We need a nice way to be able to get to editing the different resources that are associated to a dataset.

From the google doc: When I move between grid, graph and map view mode (or click on filters) the whole data explorer flickers (jumps up & down slightly) for a few moments before settling. This is happening on all resources for me and when you initially open launch the page: ​http://demo.ckan.org/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360. Only in Chrome, working fine in Firefox.

Just a ticket to keep track of a few suggested template changes.

Essentially, the functionality should be as follows:

• Add logout link that has .js-hide attached to it within the header that isn't hidden within a dropdown

See ​http://plus.google.com/ (when logged in) with and without JS to see an example of the actual sign-out working without JS

Create big tag cloud with all CKAN tags - perhaps weighting with size and colour...

David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior.

I have a patch in ​https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default.

This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet).

Things to know about using flush:

• Most of the time nothing will change. A commit implies flush.
• You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query.
• A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated.
• You can use them as a limited form of nested transaction.

It increased the performance on the tests by about 5%. There are less flushes altogether so is faster.

I do not see any downsides.

use pylint

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

• New resource group table. #956
• New kind field in resource. #957
• UI for new kind field. #958
• Resources in REST API ticket:358
• Resources in WUI #1445
• Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

• ​http://ckan.okfnpad.org/resources - discussion etherpad
  • Contains commented version of James Gardner's main email: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html
• UseCasesResources

The new field was outlined in

​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

it says

Can be "data", "metadata", "service", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.

In order to test migrations properly, we need to test the upgraded database against *all* the tests.

To do this a special flag to nosetests should be added. nosetests --migrations

This flag will make sure that the database is recreated from scratch each time using the migrate repository, bypassing init_db.