{22} Trac tickets (2647 matches)

Results (2101 - 2200 of 2647)

Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#548 task johnbywater johnbywater ckan-v1.2 closed fixed Functional test for getting package create form from the API 1283339598000000 1283351141000000
#547 story johnbywater johnbywater ckan-v1.2 closed Submit package create form response to the API 1283339487000000 1283458645000000
#546 story johnbywater johnbywater ckan-v1.2 closed Get the package create form from the API 1283339414000000 1286896814000000
#545 requirement johnbywater johnbywater ckan-v1.2 closed The system shall support creating packages via the Form API 1283339310000000 1283458648000000
#544 requirement pudo ckan-v1.3 closed duplicate Backport facet browsing to CKAN 1.2

This is in IATI, would be nice to have in generic CKAN.

1283267292000000 1291638966000000
#543 task wwaites rgrp closed duplicate Investigate partial page caching and edge-side includes

Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page.

Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish.

Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat.

1283244784000000 1311178918000000
#542 task wwaites rgrp closed fixed 'Show and Tell' page customization using cookie + js

To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: http://www.ibm.com/developerworks/web/library/wa-rails2/

In the first place we just need to apply this to:

  • You are logged in as
  • Package/group edit/authz tabs
1283244432000000 1283278942000000
#541 enhancement wwaites rgrp closed fixed Full-page caching even for logged in users

The simplest way to improve performance on specific pages is to implement full page caching.

This, however, has challenges on pages with user-specific content. For example, on CKAN when a user is logged at the top of the page we have "You are logged in as: {username}". Similarly we change the visibility of some material depending on the status of users:

  • Visibility of Edit or Authz link on packages and groups
1283243139000000 1294924872000000
#540 story wwaites rgrp closed fixed Implement caching in a systematic manner 1283242705000000 1302694845000000
#539 requirement wwaites rgrp ckan-v1.4 closed invalid The system shall improve performance 1283242447000000 1303118486000000
#538 task rgrp johnbywater closed wontfix Update CKAN Trac ticket report queries

Update report queries with more recent versions (below are report queries shipped with trac 0.11.7):

NB: Report 1 has already been done.

id | query


1 |

: SELECT p.value AS color, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY CAST(p.value AS integer), milestone, t.type, time :

2 |

: SELECT p.value AS color, : version AS group, : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (version IS NULL),version, CAST(p.value AS integer), t.type, time :

3 |

: SELECT p.value AS color,

: 'Milestone '
milestone AS group,

: id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (milestone IS NULL),milestone, CAST(p.value AS integer), t.type, time :

4 |

: : SELECT p.value AS color, : owner AS group, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time :

5 |

: SELECT p.value AS color, : owner AS group, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : description AS _description_, : changetime AS _changetime, reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time :

6 |

: SELECT p.value AS color, : t.milestone AS group, : (CASE status : WHEN 'closed' THEN 'color: #777; background: #ddd; border-color: #ccc;' : ELSE : (CASE owner WHEN $USER THEN 'font-weight: bold' END) : END) AS style, : id AS ticket, summary, component, status, : resolution,version, t.type AS type, priority, owner, : changetime AS modified, : time AS _time,reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : ORDER BY (milestone IS NULL), milestone DESC, (status = 'closed'), : (CASE status WHEN 'closed' THEN changetime ELSE (-1) * CAST(p.value AS integer) END) DESC :

7 |

: SELECT p.value AS color, : (CASE status WHEN 'accepted' THEN 'Accepted' ELSE 'Owned' END) AS group, : id AS ticket, summary, component, version, milestone, : t.type AS type, priority, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE t.status <> 'closed' AND owner = $USER : ORDER BY (status = 'accepted') DESC, CAST(p.value AS integer), milestone, t.type, time :

8 |

: SELECT p.value AS color, : (CASE owner : WHEN $USER THEN 'My Tickets' : ELSE 'Active Tickets' : END) AS group, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (COALESCE(owner, ) = $USER) DESC, CAST(p.value AS integer), milestone, t.type, time

1283190081000000 1294414537000000
#537 task wwaites wwaites closed duplicate Caching and Performance improvement

There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests.

Introducing caching has to be done carefully and should be done in a standards compliant manner.

General strategy

  • Where possible, cache output within the pylons app (beaker).
  • Facilitate external caching in an end-user's web browser or a caching proxy
  • Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes).
  • When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data.

Tasks

These tasks should be broken into sub-tickets:

  • caching of parts of templates that are expensive to render (package list, tag list, group list)
  • caching of entire output using beaker particularly for API read operations.
  • need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches.
  • cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: http://www.mnot.net/cache_docs/#CACHE-CONTROL
    • Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well
    • Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner

Future Work

  • Investigate ckanclient library maintaining a local cache as a web browser would
  • Investigate using a CDN like Google Storage or Amazon for serving cached data.
1283184362000000 1311178929000000
#536 task dread dread ckan-v1.2 closed fixed Extend base importer merging in resources

Multiple ONS records can go into one package e.g. Stats for Jan, Stats for Feb, Stats for Mar etc.

We already identify an existing package by 'external_id'. Now need to find package by title and an extra field (department in ONS case) and allow it to merge in resource and tags. Alert if rest of meta data is different.

1283173313000000 1286376029000000
#535 defect dread ckan-v1.2 closed duplicate genshi error when logged into sl.ckan.net

Genshi exception when rendering the page whilst logged in to sl.ckan.net.

1283165774000000 1283167040000000
#534 defect johnbywater dread ckan-v1.2 closed fixed buildbot config - database machine moved

Configure buildbot to use postgres db which has been moved to eu5.okfn.org

pudo said: Could you perhaps also comment on http://knowledgeforge.net/okfn/tasks/ticket/466 with a hint on how to update this properly? I only edited /home/buildslave/okfn/full/build/buildandsmoke/buildandsmoke.ini which was the only occurence of the DB credentials I could find there.

1283165568000000 1288002762000000
#533 requirement johnbywater ckan-v1.3 closed invalid The performance of CKAN shall scale

CKAN uses a database. We can put the database on another machine and then clone the CKAN machine. Then there are two machines calling the database and converting SQL results into HTTP responses.

How many machines can you have before the database becomes the bottleneck? At that point, what QoS can be obtained with normal hardware?

Afterwards, can CKAN be made to scale further than this with a little development? Can its Postgres database be mirrored in realtime? Or can we change SQL writes (create/updates) to write to many databases?

Are there any other options for scaling the persistence mechanism?

1282910499000000 1292957374000000
#532 defect johnbywater johnbywater ckan-v1.2 closed invalid Just a test - please ignore 1282906842000000 1282907197000000
#531 task pudo rgrp closed fixed Load sample IATI metadata
  • Parse data
  • Load via API
  • Develop test for load via API (can be local or remote -- with paste webfixture may be easier)
1282899244000000 1283536676000000
#530 task pudo pudo closed fixed Map IATI attributes to CKAN metadata schema

From lopad:

Publishing Entity (often/always? equal to the logged in user)
    * Domain logic: only can add entries for "yourself"
Donor
Country
Activity period: period to which this record relates
Verification status: enumeration of statuses (checked, not checked etc)
Resource links: to the actual IATI record
Number of activities: ...
Date record updated:
Date data updated: 
License: Need this field even if it may be a standard license
1282899187000000 1287392210000000
#529 task pudo rgrp closed fixed Obtain sample metadata 1282899161000000 1283536554000000
#528 task pudo rgrp closed fixed Setup demonstration instance at iati.ckan.net 1282899110000000 1283536475000000
#527 story pudo rgrp closed fixed Create domain model and load sample data 1282899025000000 1293097531000000
#526 requirement pudo pudo closed wontfix Integrate validation tool into the registry

Assumes 8 is done:

  1. Tool for validation an IATI XML record (referenced by a URL) for compliance with the IATI specification
  • Outside of scope of registry
  • Multiple validators with different levels of rigour
1282894300000000 1340626152000000
#525 requirement pudo pudo iati-2 closed fixed Confirm and approve publishing entity account

  • So some kind of confirmation or approval by someone more senior?
  • Alternative: just have an account request.
  • Priority: 3
1282894040000000 1285595152000000
#524 requirement pudo pudo iati-2 closed fixed Limit registering data to authenticated users either via API or manually.

Limit registering data to authenticated users either via API or manually.

  • Edits may only be made by authenticated entities
  • Priority: 5
1282893904000000 1285594971000000
#523 requirement pudo pudo iati-1 closed fixed Read/write APIs for users and publishing entities

Read/write APIs for users and publishing entities to access and register data

  • Users have read access
  • Publishing entities have both (though can only write to 'their' records)
  • Priority: 4 for write (read access is 3)
1282893743000000 1283897688000000
#522 requirement pudo pudo iati-2 closed fixed Create publishing entity account
  • Publishing entity emails system editors
  • System editor creates publishing entity account and emails publishing entity with details
  • Priority: 4/5
1282893668000000 1287392999000000
#521 requirement pudo pudo iati-1 closed fixed IATI Customized search 1282893401000000 1283897124000000
#520 requirement pudo pudo iati-1 closed fixed Browseable web interface onto the data
  • e.g. find/browse by country and by publishing entity and by donor
1282893270000000 1283538080000000
#519 requirement pudo pudo iati-1 closed fixed IATI look and feel
  • Use what is on iatiregistry.org at the moment.
1282893224000000 1283536828000000
#518 requirement pudo pudo iati-1 closed fixed IATI WUI Editing
  • Create custom form
1282893167000000 1283896718000000
#517 requirement pudo pudo iati-1 closed fixed Standard domain model for entries

cf. Tariqs mail

Updated version: https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&hl=en#gid=0

Original Version

Publishing Entity:
Publishing Entity Type: (Donor, Recipient, Community Data..)
Donor
Country
Activity period:
Verification status: enumeration of statuses (checked, not checked etc)
Resource links: to the actual IATI record
Number of activities: ...
Date record updated:
Date data updated:
License: Need this field even if it may be a standard license

So naively mashing these together, we get something like:

Identifier
Title
Abstract
Donor
Country
Publisher
Publisher Type
Verification Status
Department
Contact
Contact e-mail
Licence
Resource format
Resource URL
Resource ID
Activity period
Number of activities
Date record updated
Date data updated
1282893081000000 1284041561000000
#516 defect dread ckan-v1.2 closed fixed Double-escaped text visible - revision page

At: http://ckan.net/revision you can see: "Next &raquo;" on pager This doesn't seem to be a problem with earlier (1.0.1) version.

1282829539000000 1288002933000000
#515 defect kindly dread ckan-v1.4-sprint-6 closed fixed Inconsistent use of 'location' header in API

When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all.

I've removed the info about the header in the docs in the meantime.

1282757360000000 1302774268000000
#514 defect dread ckan-v1.2 closed duplicate Inconsistent use of 'location' header in API

When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all.

I've removed the info about the header in the docs in the meantime.

1282757357000000 1282757391000000
#513 task dread closed wontfix Creating a package with API should warn if there are incorrect fields

When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK.

1282754802000000 1302774329000000
#512 story dread closed duplicate User creates package via API with incorrect core fields specified 1282754750000000 1294917121000000
#511 requirement dread ckan-v1.3 closed worksforme Better warnings and errors when using API 1282754677000000 1297075354000000
#510 enhancement dread dread closed fixed Setup cron for ons import 1282665939000000 1294138332000000
#509 story dread dread closed invalid ONS data imported daily 1282665870000000 1291734435000000
#508 task dread dread ckan-v1.2 closed fixed Create branch ultrastable
  • Create a branch named ultrastable that represents latest version of 1.0 as found on dgu
    • Going forward ultrastable will act as stable - 1 (and either ultrastable or stable will be on the rock-solid production systems such as dgu)
  • Do not deprecate the stable or metastable branch for the present
  • Ask for recommendations on mailing list about best practice for branching (e.g. versioned based versus a simple HEAD versus stable) and how this affects standard operations such as applying bugfixes.
    • For useful discussion of how to do backporting and applying bugfix patches see: http://www.python.org/dev/peps/pep-0374/#backport
    • NB: in general do not always have to apply in stable/ultrastable and merge upwards for bugfixes -- can use transplant extension as well.
1282663663000000 1282908795000000
#507 story johnbywater closed fixed Upgrade a CKAN service 1282661646000000 1282909852000000
#506 task dread rgrp closed fixed Plan transition to DGU metadata-fields v3 1282660490000000 1287080989000000
#505 enhancement dread rgrp closed fixed DGU fieldset supports v3 of DGU metadata specification 1282660416000000 1298368280000000
#504 requirement dread rgrp ckan-v1.3 closed invalid The DGU fieldset shall conform to v3 of DGU metadata specification

No need for requirement - closing in favour of #505

1282660246000000 1292586686000000
#503 task dread rgrp closed fixed Provide exemplar package relationships for some packages
  1. Find some relationships.
  2. Add them via API
  3. Notify DGU of completion and point to exemplars
1282659631000000 1314031851000000
#502 requirement dread rgrp ckan-v1.3 closed wontfix Data4nr uploading script shall remove deleted packages

Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed.

1282659304000000 1292586466000000
#501 requirement pudo ckan-v1.2 closed duplicate Read-only maintenance mode

CKAN should have a read-only maintenance mode with a nice little banner on all pages, appropriate REST messages etc. Bonus points if this is triggered via an environment variable and thus can be triggered by the surrounding apache.

1282554617000000 1282724566000000
#500 defect dread ckan-v1.2 closed duplicate Exception from diff

Investigate exception occured occasionally in last couple of days on ckan.net:

WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'key' 					

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'zh-cn,zh-tw'
HTTP_CONNECTION	'close'
HTTP_HOST	'ckan.net'
HTTP_USER_AGENT	'Baiduspider+(+http://www.baidu.com/search/spider.htm)'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'123.125.66.32'
REMOTE_PORT	'63767'
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x9f603ec>
beaker.cache	<beaker.cache.CacheManager object at 0x9f6042c>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x9f602ac>>
beaker.session	{'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: >, '')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0x104552ec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xfb17aec>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xfb17aec>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x10286d4c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.logger	<logging.Logger instance at 0xa16e0cc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 167584972>, 'auth_tkt': <AuthTktCookiePlugin 169253516>}
routes.route	<routes.route.Route object at 0x9f3690c>
routes.url	<routes.util.URLGenerator object at 0xfd8d7cc>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x103a5bf0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xfd8d7cc>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})



1282553033000000 1287747652000000
#499 story thejimmyg johnbywater closed wontfix Registration and management of UKLII related code lists, controlled vocabularies and other forms of reference/static data 1282432571000000 1296593038000000
#498 story johnbywater closed invalid Search packages within location "bounding box" 1282427412000000 1294412520000000
#497 story johnbywater johnbywater closed duplicate Respond to CSW "GetRecords" request 1282427334000000 1294407718000000
#496 requirement wwaites johnbywater ckan-v1.4-sprint-2 closed fixed Provide a mechanism for exporting GEMINII metadata via a CSW interface 1282427307000000 1299164106000000
#495 story johnbywater ckan-v1.2 closed Put form for updating remote metadata entity 1282427150000000 1285198898000000
#494 story johnbywater ckan-v1.2 closed Get form for updating remote metadata entity 1282427106000000 1285198894000000
#493 story johnbywater ckan-v1.2 closed Get harvest source entity 1282427083000000 1284493130000000
#492 story johnbywater ckan-v1.2 closed Submit harvest source create form response to the API 1282427042000000 1284493145000000
#491 story johnbywater ckan-v1.2 closed Get form for creating harvest source entity 1282427008000000 1284493173000000
#490 requirement johnbywater closed fixed The system shall support registering a harvest source via the CKAN API 1282426875000000 1286200804000000
#489 story wwaites johnbywater closed fixed Notify RDF service of model events 1282426051000000 1294416189000000
#488 requirement johnbywater closed wontfix Catalogue service shall notify RDF service 1282426021000000 1320930240000000
#487 story johnbywater closed duplicate Notify SOLR service of model events 1282425910000000 1291639404000000
#486 requirement johnbywater ckan-v1.3 closed duplicate Catalogue service shall notify and query SOLR service 1282425790000000 1291639321000000
#485 story johnbywater closed duplicate Performance beats QoS criteria 1282425219000000 1294411946000000
#484 story johnbywater closed invalid Restricted partial read-write access to catalogue Web UI 1282422858000000 1294417248000000
#483 story johnbywater closed invalid Restricted total read-write access to catalogue Web UI 1282422801000000 1294417216000000
#482 story thejimmyg johnbywater ckan-backlog closed wontfix API Rate Limiting 1282422756000000 1298284158000000
#481 story johnbywater ckan-v1.3 closed fixed Unrestricted total read-only access to catalogue API 1282422697000000 1294248359000000
#480 requirement thejimmyg johnbywater ckan-v1.4 closed fixed Catalogue service shall conform to specification

Common requirements for running CKAN behind a (e.g Wordpress or Drupal) front-end:

  1. Unrestricted total read-only access to catalogue API for general public (e.g. voluntary organisation).
    • monitored by API key
    • not monitored by API key
  2. Restricted total read-write access to catalogue API for authorized clients (e.g. front-end system, bulk upload clients).
    • restricted by CKAN access controller
    • restricted by HTTP Auth
    • restricted by IP address
  3. Restricted total read-write access to catalogue Web UI for authorized users (e.g. site admins).
    • restricted by CKAN access controller
    • restricted by HTTP Auth
  4. Restricted partial read-write access to catalogue Web UI for authorized users (e.g. group admins).
    • restricted by CKAN access controller
    • restricted by HTTP Auth

CKAN as a catalogue service

1282422612000000 1300281551000000
#479 task dread johnbywater ckan-v1.2 closed fixed Suggest DGU presents database dumps through front end 1282324365000000 1288004211000000
#478 task johnbywater ckan-v1.2 closed fixed Unit test for "interface resource" in API

We don't want /api to be an HTML page, but rather provide information about the API in a JSON structure.

1282315067000000 1283248713000000
#477 story johnbywater ckan-v1.3 closed duplicate Discover location of the daily database dumps 1282313788000000 1294411761000000
#476 requirement johnbywater ckan-v1.3 closed wontfix Daily database dumps shall be discoverable and downloadable 1282313433000000 1294411741000000
#475 task johnbywater ckan-v1.2 closed fixed Unit test to check API key in incorrectly named HTTP header is not accepted by server 1282312802000000 1282932705000000
#474 task dread dread closed wontfix Questions related to fi.ckan.net

Several questions to answer (and see file in email). To respond to the Finnish team and copy all Q&A to the ckan-discuss list.

1 ) I have played with two different datasets to get the metadata from those imported to CKAN. The other dataset is in PCAXIS format and other one is in Excel format. Please find the scripts + sample files attached in a zip file that contains instructions too to run the scripts. I looked at import scripts written in object oriented manner and getdata samples in CKAN but it was easier for me too write these from scratch (copied the unicrap function from CKAN samples). If I run the PCAXIS import script I get Unicode decode error (see errors file). When I use the unicrap function then the error does not appear in logs. I assume this is some Python environment setting issue. Of course I would like to run the script without unicrap function to get scandic letter imported correctly to ckan(and displayed in WUI). In Excel case there is no scandic letter issue which is kind of strange but this probably relates how the Excel library handles encoding.

To be honest the whole unicode/encoding/decoding issue has been quite difficult to deal with and debug. I hope I would get proper environment configuration to handle scandic letters.

2 ) It seem in CKAN by default the localisation file contains 462 strings. The transifex link Rufus sent contains 400 strings. Is transifex the one we should use as localisation file ? Where are the rest 62 strings then fetched from or are they needed at all. I have found that as I have been using the .po file coming in CKAN source (which has API translations too) this also causes similar unicode problems when an error in API occurs (the API error messages containing scandic letters provide the similar UnicodeDecoding? errors as mentioned above). So, it seems the unicode issues appear in various cases. Maybe a proper environment settings will be able to prevent also these problems... If transifex is the one to use I can prepare it and maybe then you could upload it to fi.ckan.net as there will be also other people checking the translations. To run the test and see the API translation errors especially in pcaxis case I would recommend to install attached FI translations to your environment and include lang = fi configuration. It looks like the API errors does not come when running default lang = en.

3 ) Is it possible to change the license selection/options available in license field drop-down ? Where are these located and can these be modified locally ?

4) It looks like the values stored in package extra fields are not searchable. Is this intentional ? It looks more or less that all other fields can be searched using free-text search in CKAN.

5) How package dependencies are defined and managed ? For instance we want to import some packages in a manner that the other package is a child package of another. Does API support this and/or is this possible to do in WUI ?

Best Regards, Henri

1282312255000000 1294916760000000
#473 story johnbywater ckan-v1.2 closed API user discovers correct header for sending API key 1282312203000000 1283248736000000
#472 story johnbywater ckan-v1.2 closed API user sends API key in incorrectly named header 1282312108000000 1282932740000000
#471 story johnbywater ckan-v1.2 closed API user sends API key in correctly named header 1282312088000000 1283248729000000
#470 task dread johnbywater ckan-v1.3 closed fixed Adjust API key's HTTP header name on DGU "live" and "test" 1282310854000000 1292587187000000
#469 task johnbywater ckan-v1.2 closed fixed Unit test to check API key in correctly named HTTP header is accepted by server 1282310760000000 1282932682000000
#468 task johnbywater johnbywater ckan-v1.2 closed fixed Default config file has commented out configuration variable 1282310659000000 1283250799000000
#467 story johnbywater ckan-v1.3 closed duplicate Admin configures CKAN to expect API key in named HTTP header 1282310562000000 1294411681000000
#466 requirement dread johnbywater ckan-v1.3 closed wontfix Name of HTTP header for API key shall be configurable 1282308995000000 1294835610000000
#465 requirement johnbywater ckan-v1.3 closed wontfix Require API key for all API requests. 1282308904000000 1294411534000000
#464 task rgrp dread closed duplicate Request dgu db server access 1282306104000000 1282325194000000
#463 defect dread ckan-v1.3 closed worksforme package diff exception

Investigate exception:

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
AUTH_TYPE	'cookie'
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'en-US,en;q=0.8'
HTTP_CACHE_CONTROL	'max-age=0'
HTTP_COOKIE	'__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173'
HTTP_HOST	'ckan.net'
HTTP_REFERER	'http://ckan.net/package/history/dbtune-audioscrobbler'
HTTP_USER_AGENT	'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'92.226.62.168'
REMOTE_PORT	'55102'
REMOTE_USER	'http://pudo.org/'
REMOTE_USER_TOKENS	['']
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0xbb528cc>
beaker.cache	<beaker.cache.CacheManager object at 0xbb528ec>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0xbb5286c>>
beaker.session	{'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: __utma='27730403.1381387877.1281962773.1282248244.1282250173.11' __utmb='27730403.6.10.1282250173' __utmc='27730403' __utmz='27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan' auth_tkt='e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!'>, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0xf702aec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xf6eaf0c>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xf6eaf0c>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0xf6eaa8c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.identity	<repoze.who identity (hidden, dict-like) at 280815340>
repoze.who.logger	<logging.Logger instance at 0xbcdfccc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 196612236>, 'auth_tkt': <AuthTktCookiePlugin 198048140>}
routes.route	<routes.route.Route object at 0xbbc44cc>
routes.url	<routes.util.URLGenerator object at 0xf5e55ec>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0xfe0b0b0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xf5e55ec>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})
1282306037000000 1294916148000000
#462 story dread johnbywater ckan-v1.2 closed fixed Visitor downloads a daily database dump 1282305305000000 1285757238000000
#461 task dread johnbywater ckan-v1.2 closed fixed Change ONS data importing to work via API
  • Move script out to ckandgu repo
  • Change script to convert xml into package dicts
  • Test (against test.ckan.net, hmg.test.ckan.net)
  • Deploy
1282303411000000 1283250478000000
#460 defect pudo johnbywater closed fixed State field changed by non-sysadmin

This package:

http://ckan.net/package/dbtune-audioscrobbler

was:

  1. created by Richard (logged-in)
  2. edited by Richard (logged-in)

(According to the logs, at this point the state was changed from 'active' to 'deleted') -- RP was it set to 'deleted' or just ?

  1. pudo changed the state back to active

Similarly an incident with bibbase package where field set to (see http://ckan.net/revision/diff/bibbase?diff=702bb0a3-03b7-49ac-87ad-e489c414962f&oldid=5447842d-b6ed-41d9-9cfd-8bb73b85c409)

Need to investigate how this got changed, fix if necessary and report back to Richard. Note that package 'admins' as well as sysadmins can change the state of a package (though note that bibbase did not appear to have an owner).

Suggested solution (for setting to ):

  • Ensure in ckan/forms.py that there is a validator for state field that ensures only set to valid values.
  • Check that we do not allow state to be changed in the api except by package owner or sysadmin
1282300002000000 1311176868000000
#459 bug johnbywater ckan-v1.2 closed fixed Versions on branches are broken 1282299973000000 1282921783000000
#458 task rgrp johnbywater closed wontfix Request to DGU for access to database machine 1282299950000000 1294415537000000
#457 story dread dread ckan-v1.2 closed fixed Dump database on a daily basis 1282299921000000 1282909051000000
#456 story johnbywater dread ckan-v1.2 closed duplicate Daily dump 1282299917000000 1282665858000000
#455 task johnbywater closed invalid SOLR - suggest 1 pager about how system would work

Either CKAN writes to SOLR and Drupal reads from SOLR, or CKAN writes to SOLR and Drupal reads SOLR via CKAN API (so search resource locations are unaffected).

1282299913000000 1291637172000000
#454 story dread dread ckan-v1.2 closed fixed Import ONS data via API 1282299851000000 1286376044000000
#453 task thejimmyg johnbywater closed invalid Analysis and planning for UKLII 1282299725000000 1294411374000000
#452 task dread closed fixed Assist with using package update error form 1282299647000000 1287997540000000
#451 task dread dread ckan-v1.3 closed invalid Show/Allow only departments associated with user

Use drupal api

1282299641000000 1292587233000000
#450 task dread closed fixed Assist with proxying of static media for forms

John did this a while ago.

1282299573000000 1287997487000000
#449 task wwaites johnbywater closed invalid Start email thread about RDF in CKAN with JF and WW 1282299482000000 1311182945000000
Note: See TracReports for help on using and creating reports.