{22} Trac tickets (2647 matches)

Investigate exception occured occasionally in last couple of days on ckan.net:

WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'key' 					

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'zh-cn,zh-tw'
HTTP_CONNECTION	'close'
HTTP_HOST	'ckan.net'
HTTP_USER_AGENT	'Baiduspider+(+http://www.baidu.com/search/spider.htm)'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'123.125.66.32'
REMOTE_PORT	'63767'
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x9f603ec>
beaker.cache	<beaker.cache.CacheManager object at 0x9f6042c>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x9f602ac>>
beaker.session	{'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: >, '')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0x104552ec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xfb17aec>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xfb17aec>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x10286d4c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.logger	<logging.Logger instance at 0xa16e0cc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 167584972>, 'auth_tkt': <AuthTktCookiePlugin 169253516>}
routes.route	<routes.route.Route object at 0x9f3690c>
routes.url	<routes.util.URLGenerator object at 0xfd8d7cc>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x103a5bf0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xfd8d7cc>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})

Genshi exception when rendering the page whilst logged in to sl.ckan.net.

Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page.

Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish.

Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat.

Anja is reporting this, severe bug, I think.

Discussion between John and Peter:

Given we can identify a document, does the disappearance of a document from a registered source imply the disappearance of the metadata (such that we delete packages once the documents disappear from the registered source)?

I would expect a more explicit 'delete'. The UKLP Use Case Library describes this as "withdraw a dataset or service from publication" (part of UCD03 Maintain resources).

Whilse using CKAN web interface, you are not tempted to edit stuff:

• You know at all times this CKAN is read-only
• All editing facilities are still seen but greyed-out with an indication why it is.

Should return help for the harvester (currently crashes).

(Just like resource table, but extra fields don't need up/down buttons)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

Implies that last harvested time must be stored on source by harvesting job (see #802).

The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page.

The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import.

Based on the ideas in:

• DistributingChanges
• SyncingInstances

Merging the tickets #296, #297, #298, #299 into this one.

Related ideas include:

• Using dcat to exchange data
• Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance

This should be considered a wishlist item for the timebeing.

We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG.

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

I need to completely munge the rendered data coming from the app. There's not currently a plugin hook for this, where I get access to the entire environment. The most generic form of plugin would be WSGI Middleware. I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system.

We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include:

• CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
• Catalogue scraping for LOD2 experiments (yields RDF graphs)
• Atom/DCat for LOD2 production (yields RDF graphs)
• OAI-PMH for ​http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios.

In general, such a system would have the following stages:

• Source selection: find what to download/scrape/harvest/parse
• Index retrieval (i.e. package index)
• Item retrieval (i.e. package entity)
• (Optional: Serialization)
• Normalisation
• Loading/Merging? into CKAN

Exisiting harvesters are at:

• CSW: ​https://bitbucket.org/okfn/ckanext-csw/src/
• Scraper+CKAN: ​https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/

Package edit form at hmg.ckan.net has no publishers any more.

1. Login (shows ckan home page with flash message "Welcome back xyz")
2. Click away, to "Add package" say
3. Click back to ckan "Home". It shows "Welcome back xyz" again - it shouldn't, I assume?

Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787.

Two man reasons for the changes are:

• We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer.

• Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able.

The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package).

1. The first thing the is_authorized method is a hook to a plugin

which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!)

Reason: authorization can be completely delegated to another system (or partially)

2. is_authorized method currently takes (username, action, object)

but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model.

3. Rename these two classes to better reflect what they are
   • AuthorizationGroup? -> UserGroup?
   • Group -> PackageGroup?

4. Rename the Editor role to PriveledgeUser? since Editors sometimes can't edit.

Although this sounds a bit radical we already have auth extensions.

Read-only CKAN Web UI

(Additional requirement from #764)

Whilse using CKAN web interface, you are not tempted to edit stuff:

• You know at all times this CKAN is read-only
• All editing facilities are still seen but greyed-out with an indication why it is.

With the default test data created by

paster db clean paster db init paster create-test-data

going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy

But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.

PDEU has a newer layout which could be adapted to also be used on ckan.net. For this, things should be cleaned up, copied to the ckanext-ckan.net repo as needed and re-colored the core CKAN color scheme.

Maybe we could have this coincide with a CKAN rename?

Registering a new user directs you to that user's profile page but does not log you in. There is an empty set of "Recent changes" and no ability to modify the page. Users must follow the login link in the top-right corner to continue.

Registering a new user should immediately log you in.

Split out of #852. Automated archiving of datasets (related to QA).

Automated archiving using worker process

• #890 - Timed actions in ckanext-queue
• #891 - Resource download worker daemon
• #892 - Make stored data available in WUI

e.g. cz.ckan.net has lang=cz but when you click on Home page link "English" it flashes message "Language has been set to: English" but the rest of the page is still Czech.

Can this be released as a standard CKAN feature?

Super ticket: #1745

Ability to build up search query using a nice javascript-y interface.

• Add facets by selecting attribute and adding -> search facet options in dropdown -> added to search (with 'x' to remove -- as we currently do).
  • (a bit like the data.hri.fi)
• Some improvements to css
  • See ​http://eutr.ep.io/
• Improvements to faceting
  • Ability to configure faceting and number of items to show (?)
• Pure JS search implementation to make it easy to reuse across site

As part of #1574 we decided that it would be better for webstore (and future external services) to be able to authenticate simply with CKAN-Core.

Currently webstore access the CKAN database to obtain the key for the user but it would be better if this connection was not so tightly bound and that webstore used an API as any other external service might.

Need to discuss further with dread

Provide configuration for reverse proxying that will correctly handle the mapping of a URL to a sub-folder (using X-SCRIPT-NAME)

• Analysis of the best solution [1d]

• Implement reverse proxying in tandem with #1653 [2d]

• Document and store the configuration files. [1d]

We need to be able to translate:

• EC extension (templates, form)
  • Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
• Field values in taxonomy (e.g. country names - Eurovoc)
  • #1665 Research into Eurovoc
  • Display Taxonomy in different languages
• Field values not in taxonomy (e.g. title & description)
  • Use extra fields e.g. _i18n_title_fr = Le data.
  • (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)

Timescale to complete this - Mar/Apr?.

Related work to interface with:

• Eurovoc - Sean
• i18n in search index - Adria

Enable ckanext's to add new vocabularies to default_package_schema(), use the new vocabularies in templates for dataset view, create, edit pages, helper function for extensions to create a vocabulary schema for a given vocabulary name (must match a vocab already in the db) and add it to default package schema.

Add a helper function to make it easy for extensions to add new vocabularies to default_package_schema(). The helper function should take the name or ID of a vocabulary (which should already exist in the db) and add the necessary key: [schema] entry to default_package_schema(), with the necessary validation, authorisation, transformation.

Changes to publishers for customer