{22} Trac tickets (2647 matches)

Robust against slightly incorrect URLs (e.g. with trailing slash, or without /csw), and non-CWS service which also return <ows:ExceptionReport ... />.

Both cases would be resolved by checking capabilities (with CswClient's check_get_capabilities() method).

need to make some changes for the links to semantic.ckan.net. it should use ​http://semantic.ckan.net/record/<package_id> now

append .rdf, .ttl, .nt, .dot, .json (even .html for an ugly table) to taste (or just leave off the suffix and let content negotiation take care of it)

the base url is changed, but it now uses id not name.

see for example:

• ​http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33
• ​http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33.ttl

filter_by_openness and filter_by_downloadable options don't work when specified as URI parameters. (They do work in qjson parameters)

The coverage tool (run by buildbot in the ckan build) reports that only 24% of lines of ckan.controllers.authorization_group are run in tests and 38% of ckan.forms.authorization_group. This suggests a need for more tests.

To see what lexemes are generated for debug purposes.

With the default test data created by

paster db clean paster db init paster create-test-data

going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy

But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.

Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception.

With the default test data created by

paster db clean paster db init paster create-test-data

going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy

But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.

.

Need some changes to pip-requirements files in release branches.

The web interface has a sidebar (#primary) which should be hidden in some pages. This is for QA extension and useful for package new and edit pages. Must be compatible with DGU theme.

Purging Revisions

• Delete button displayed on:
  • /revision/list
  • (/package/history)
    • /package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison.
  • /revision/{id}
• Delete button submits to delete action on revision and changes revision state to 'deleted'.
  • undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?)
• Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: "Empty trash" (irreversible). Click button purges all revisions with deleted state.

Purging Packages

• Put into deleted state.
• Listed on /ckan-admin/trash
• Separate Empty trash button which deletes all associated revisions.
  • Should be separate from Empty trash for revisions

Current system

• Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!)

1. Edit a package
2. Hit 'preview'.

The preview contains the section "CKAN API / datapkg" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the "CKAN API / datapkg" section too?

This has once again been ruled out for UKLP for the timebeing.

in ckan/controllers/package.py around line 130 it does some strange things...

perhaps replace with ​https://github.com/wwaites/autoneg

and handle redirection of these content types:

application/rdf+xml
application/turtle
text/plain
text/x-graphviz

Use case:

• Login using OpenID
• Click on 'My account' - results in 404

Solutions:

• User user.id instead of their name
• Escape the URL properly.

The stages involved with doing this.

• Convert model objects to standard dict format (DONE)
• Convert standard dicts to current api formats (DONE)
• Make standard dicts savable (DONE)
• Validate standard dict format. (DONE)
• Authorize actions

When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK.

Default visitor roles in default config is reader, not anon_editor.

Problem caused by changes in #1066 (released in 1.3.3)

New installs will be affected, although simple to just increase permissions when the installer realises a visitor can't create packages.

The solution to the config getting out of sync with the code like this is to not have the default_roles in the config - refer to the code in the configuration instructions.

ckan.net appears to have either gone down or be running ultra slowly.

this means that ckan copies running locally on my machine run very slowly indeed.

is this behaviour desirable?

This command finds lots of ​http://~~~ckan.net references in python, html and javascript files:

find ~/pyenv/src \( -name "*.py" -or -name "*.html" -or -name "*.js" \) -print0 | xargs -0 -e grep --color -nH -e "​http://.*ckan.net"

output for reference:

/home/okfn/pyenv/src/ckan/ckan/init__.py:5:Network (CKAN) site: ​http://www.ckan.net. /home/okfn/pyenv/src/ckan/ckan/lib/create_test_data.py:346:<​http://ckan.net/> /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:3:DOMAIN = '​http://ckan.net' /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:4:CKAN_NAMESPACE = '​http://ckan.net/#' /home/okfn/pyenv/src/ckan/ckan/lib/talis.py:60: 'ckan':'​http://ckan.net/ns#', /home/okfn/pyenv/src/ckan/ckan/public/scripts/bookmarklet.js:2: f='​http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&title='+encodeURIComponent(document.title); /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:16: addtockan.src='​http://ckan.net/scripts/bookmarklet.js'; /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:27: <p><strong>Proper bookmarklet (compressed -- need to escape &amp;):</strong> <a href="javascript:(function(){f='​http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;title='+encodeURIComponent(document.title);if((n=document.getElementsByName('description')[0])&amp;&amp;(d=n.content)){f+='&amp;notes='+encodeURIComponent(d);}a=function(){if(!window.open(f)){location.href=f;}};if(/Firefox/.test(navigator.userAgent)){setTimeout(a,0)}else{a()}})()">Add to CKAN</a> /home/okfn/pyenv/src/ckan/ckan/templates/home/license.html:31: For convenience, all material - including all package, tag and revision information - is available in bulk, in the form of a full dump of the CKAN database. This (gzipped) dump file is updated daily and can be downloaded from <a href="​http://www.ckan.net/dump/">​http://www.ckan.net/dump/</a>. /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:71: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<​http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:137: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<​http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:447: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<​http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:458: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<​http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/functional/api/base.py:178: assert '"ckan_url": "​http://test.ckan.net/package/annakarenina"' in msg, msg /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:116: api e.g. ​http://ckan.net/api rather than ​http://ckan.net/api/rest) /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:261: :param base_location: default *​http://www.ckan.net/api* /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:267: base_location = '​http://www.ckan.net/api'

As a User (especially as a Package Owner/Maintainer?) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

Implementation

• Use the existing support for this feature in google analytics or piwiki
• For google analytics see:
  • Integrate to record: ​http://www.google.com/support/analytics/bin/answer.py?answer=55529
  • Accessing analytics data via API (for showing download counts): ​http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html
• Limitations only record downloads by browsers with js turned on.
  • This is OK I think (in any case machine download via e.g. datapkg does not get downloaded)

Old Spec (do it in CKAN)

• Record info of form: resource id (or url?), timestamp
• Do this via javascript capturing of onclick event talking to an api
• API: /api/resource/{id}/download
  • POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
  • GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

Questions

• Do we record ip addresses (to handle de-botting etc)?
• Do we count preview clicks as well?

Tool that checks which packages have not been indexed.

Required for DGU: ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/940

It takes several minutes to print the 'rights' on DGU, which is annoying when you only want to grep for a few lines. Much quicker than grepping is to filter in the query.

api/rest/package_history is not RESTful or follow API naming conventions. Therefore move it to /api/rest/package/revisions

Also, API docs incomplete.

The definition of the 'reader' role includes creating packages, which is too permissive for some CKAN instances (e.g. DGU). 'Reader' suggests only reading, so I think this role should avoid creating and editing.

All projects so far want all roles to be able to create users, so this stays as a Reader action for now, as a convenience.

Implementation:

• Action.PACKAGE_CREATE removed from reader's default_role_actions
• Visitor has a new default role, called 'anon_editor' which can edit packages, but not groups / auth groups - you have to log in for that.
• Migration script not needed?
• Code comments written, to make clear the suggested policy

The Wordpresser extension seems to be working well for DataGM, but needs caching and tests.

Loader currently checks for same name, but also should check for name_, name etc.

Errors are caused when harvesting documents. This also makes the count not show up correctly on the ckan search page.

Changing an old resource and creating a new resource on the end results in the old resource moving to the end in the ordering.

This breaks tests:

• (ckanext-dgu) ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadBasic.test_fields
• (ckanext-importlib) ckanext/importlib/tests/test_loader.py:TestLoaderInsertingResources.test_0_reload

You can make a ckan test break with this patch:

diff -r e6643cf1324c ckan/tests/models/test_resource.py
--- a/ckan/tests/models/test_resource.py        Wed Mar 23 13:25:52 2011 +0000
+++ b/ckan/tests/models/test_resource.py        Wed Mar 23 19:22:35 2011 +0000
@@ -297,6 +297,8 @@
                'url':self.urls[1], 'format':u'OTHER FORMAT',
                'description':self.description, 'hash':self.hash,
                'id':original_res_ids[2]},
+            { #new
+                'url':'new'},
            ]
        pkg.update_resources(res_dicts)
        model.repo.commit_and_remove()

There seems to be a problem with vdm creating a replacement Resource for the old resource - because it has a duplicate position it is put to the end by the SQLAlchemy ordering_list function.

create a new test file ckan/model/test_user.py

add tests for the following three functions in ckan/model/user.py

number_of_edits, number_administered_packages, search

merged in in changeset 0046f83aedcf

It's because Package carries several out-of-band values, which are snagged on the way back out. Entity get response also can't be posted.

However, post response can be re-posted (because it isn't the same as the register-post/entity-get responses.

An issue for CKAN too.

Sub-ticket of #961 (form, validation, model sync meta-ticket) and depends on that work.

It's not good to have ckanext doing lots of different things with different dependencies. Split it off into:

• ckanext-importlib

etc.

And then deprecate the ckanext repo itself.

http://ckan.net/api/2/rest/group

returns group IDs but I can only reach a group by name:

http://ckan.net/api/2/rest/group/economics

when I also want to get a group by ID:

http://ckan.net/api/2/rest/group/04fb43d2-8ddf-4485-9bf5-66d47d3672f3

• Deleting a group changes state to 'deleted' rather than purging it
• Adding authz tests for deleted groups

Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time).

Cost: 10m (giving high priority because of low cost)

(Assigning to dread as he has been managing the stable branch).

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

Documentation article on caching / improving performance. (To complement configuration docs.)

• Different sorts of cache - beaker style, etags, package_dict in search results(?)
• How each one affects performance
• How to turn them on/off and configure them
• Is it possible to bypass each of them in the browser or with wget/curl?

​http://ckan.net/tag/linguistsic doesn't have a single entry, but is still kept and displayed.

It should be purged !!!

Requested by DGU.

(instead of being hard coded)

You get redirected for a number of different reasons - often confuses me let alone the average user!

Cost: 1h

Implies that last harvested time must be stored on source by harvesting job (see #802).

in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat

add revisioning to harvested document table

Add an api for searching users. This is needed for any kind of ajax autocomplete (needed for anywhere we want to add users).

• API location: /api/util/user/lookup?q=querystr&limit=10
• Return json objects containing {id: ..., name: ..., fullname: ...}
• Put in a module called controllers/apiv2/user.py

Package edit form at hmg.ckan.net has no publishers any more.

I'm using v 1.3.1a and get a lot of "WebApp? Error: <type 'exceptions.Exception'>: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>" error messages.

Module ckan.forms.package:87 in build_package_form
<<      # Options/settings
           builder.set_field_option('name', 'validate', package_name_validator)
           builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
           builder.set_field_option('state', 'dropdown', {'options':model.State.all})
           builder.set_field_option('notes', 'textarea', {'size':'60x15'})
>>  builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
Module ckan.model.package:283 in get_license_options
<<      @classmethod
           def get_license_options(self):
               register = self.get_license_register()
               return [(l.title, l.id) for l in register.values()]
>>  register = self.get_license_register()
Module ckan.model.package:278 in get_license_register
<<      def get_license_register(self):
               if not hasattr(self, '_license_register'):
                   self._license_register = LicenseRegister()
               return self._license_register
>>  self._license_register = LicenseRegister()
Module ckan.model.license:46 in __init__
<<              from licenses.service import LicensesService2
                   self.service = LicensesService2(group_url)
                   entity_list = self.service.get_licenses()
               else:
                   from licenses import Licenses
>>  entity_list = self.service.get_licenses()
Module licenses.service:44 in get_licenses
<<          except Exception, inst:
                   msg = "Couldn't connect to licenses service: %s" % inst
                   raise Exception, msg
               try:
                   licenses = loads(response_body)
>>  raise Exception, msg
Exception: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>

Editing a group properties on ckan.net gives an exception when you submit the form. (Is this related to db migration issues?)

Example page: ​http://ckan.net/group/edit/civil-society

Exception:

WebApp Error: <class 'sqlalchemy.exc.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "group_revision_pkey" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'description': u"A group for open data related to civil society supported by members of the interest group at: http://okfn.org/groups/civil-society\r\n\r\nCivil society is composed of the totality of voluntary civic and social organizations and institutions that form the basis of a functioning society, as distinct from the force-backed structures of a state (regardless of that state's political system) and commercial institutions of the market. \r\n\r\nhttp://en.wikipedia.org/wiki/Civil_society", 'created': datetime.datetime(2011, 2, 24, 14, 30, 53, 334842), 'title': u'Civil Society', 'state': u'active', 'continuity_id': u'f4f9f09