Try and minimise the amount of different caching methodologies, especially for api calls.

Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page.

Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish.

Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat.

Not so keen on URLS such as /package/edit/mypackagename. These seem to be task-oriented rather than resource oriented (which would be /package/mypackagename/edit), and unstable: there is a common root for /package/edit/mypackage and /package/history/mypackage which is not reflected in the URL structure.

Sort out how we deal with dashes versus underscores in package names.

Test coverage for ckan.lib.cache is only 40%. Can that be usefully improved?

Rationale

The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview).

Implementation

A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues.

Steps to test:

1. Modify a dataset on datadotgc.ca, redirects to CKAN

2. On save, redirects to ​http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to ​http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100

3. You can see that the Dataset has not updated correctly. Run a check on the API v1 - ​http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present

4. Check the v2 of the API - ​http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present.

5. Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either.

On ckan.net there are a few packages which cause a 403 exception when you try to rate them: ​http://www.ckan.net/package/rate/coins-data?rating=3

The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() InvalidRequestError?: Multiple rows returned for one()

It looks like this package got rated twice in quick succession, creating two similar PackageRating? objects. This race condition needs to be solved and these particular rating objects repaired.

(2008-09-10) at present when setup.py is edited info command may not give up to date information if pkg-info (in egg.info) is not rebuilt.

Suggest: info command needs to rebuilds pkg-info in python-type distributions.

Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787.

Two man reasons for the changes are:

• We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer.

• Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able.

The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package).

1. The first thing the is_authorized method is a hook to a plugin

which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!)

Reason: authorization can be completely delegated to another system (or partially)

2. is_authorized method currently takes (username, action, object)

but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model.

3. Rename these two classes to better reflect what they are
   • AuthorizationGroup? -> UserGroup?
   • Group -> PackageGroup?

4. Rename the Editor role to PriveledgeUser? since Editors sometimes can't edit.

Although this sounds a bit radical we already have auth extensions.

Read-only CKAN Web UI

(Additional requirement from #764)

Whilse using CKAN web interface, you are not tempted to edit stuff:

• You know at all times this CKAN is read-only
• All editing facilities are still seen but greyed-out with an indication why it is.

​http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d

As reported on ​ckan-dev:

items in the footer of CKAN ("Packages", "Groups & Tags", "About", "Language", etc.) are shown vertically instead of horizontally in IE7. This works fine in later browsers like IE8, IE9, FF4, and latest Opera and Chrome.

This seems to exist in all recent CKAN versions up to 1.4.1a.

StaticURLParser can have caching - use it

When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below).

cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan.

In the meantime I've reverted the db to the old version (pre-upgrade).

cz.ckan.net is not working properly until this is fixed.

okfn@eu19:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://<user>:<pass>@<host>/<dbname>
  module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects
2011-06-29 09:07:41,294 INFO  [migrate.versioning.api] 28 -> 29... 
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname
  ret = fn(self, con, *args, **kw)
Traceback (most recent call last):
  File "/usr/bin/paster", line 18, in <module>
    command.run()
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 218, in run
    result = self.command()
  File "/usr/lib/pymodules/python2.6/ckan/lib/cli.py", line 84, in command
    model.repo.upgrade_db()
  File "/usr/lib/pymodules/python2.6/ckan/model/__init__.py", line 163, in upgrade_db
    mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 185, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 364, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py", line 83, in runchange
    change.run(self.engine, step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py", line 140, in run
    script_func(engine)
  File "/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py", line 152, in upgrade
    migrate_engine.execute(q)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1788, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1191, in execute
    params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
    return self.__execute_context(context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1302, in __execute_context
    context.parameters[0], context=context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py", line 299, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "package_group_revision" violates foreign key constraint "package_group_revision_continuity_id_fkey"
DETAIL:  Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table "package_group".
 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'}

You can register a user with password and (optional) email address. But if you forget the password you can't then log in again. We need a password reset facility that sends and email with a new password.

Change the color of the "History" tab icon to yellowish, like the rest of the icons in the other tabs ("View" and "Edit", "Authorization").

Reasoning

When you view a data package, for instance ​http://ckan.net/package/thesaurus-w, you will see "View", "Edit", "History" tabs on the top. "History" tab has a black-and-white icon which makes it look like an inactive/disabled tab (since the text is also grayed out when the tab is not selected). This is not a major issue, but it is a little bit confusing for the users. This icon exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net).

The simple solution would be to change the color of the "History" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. "View" and "Edit").

We need a way to pass in schemas to the logic layer to deal with edge cases.

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

Storage is now working but there are

• Integrate with Resources (e.g. create a resource for each file upload and give option to associate with a package)
  • Should we introduce rule that files *not* associated with a Resource are periodically deleted?
• Allow setting of a file name/path before upload
• Allow for file overwriting/deleting etc (how should this work -- do we want to allow this sort of thing)
• Integrate local file upload stuff in api/auth/*

Different Backend Issues

Local file store is rather different from 'remote' storage in various ways:

• For remote you don't want to use many buckets as there are bucket limits while for local you want to. Should we there have a single path that users provide which we then partition differently for different backends.

(spun out of ticket:336)

Resource added to model API at:

api/rest/resource

Example model request

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028

returns:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Authorization

1. Have it generic (ie. not per resource) and use an action/role on system
2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
3. Introduce Resource in authorization system (requires migration)

Mixed model

Create / Edit:

if resource associated to package:
    check_permissions(package, update)
else:
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)

Extra fields should be represented in some manner.

Lookup is needed to add a package but won't show up at some times.

Originally reported by Anja Jentzsch, re questions.

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: ​http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From ​http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

1. Generating (or just filling) a form template with 'form data' (and errors)
2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

• #926 - Pick a simpler form framework
• #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
  • #1079 Refactor API to use new logic layer and dictization
  • #1078 Refactor WUI controllers and forms to use logic layer
  • cf existing dumper and importer code
  • This will fix #662
• [not ticketed yet] - validation layer (should work on serialized objects?)
• #662 - Can't put entity that is returned by posting to package register (Defect)
• #972 - Merge 'extras' into main package dict rather than having separate key
• #1035 - Form impressions are given IDs
• #810 - Move "add packages" field up in group form (easier to do this once forms are done)

second part of #908

Simplify the logic in the atom feed an make all feeds use logic layer to return lists.

WebOb? from v1.0.7 has some interesting new behaviour with reading request data for different Content-Types:

• It seems that looking at request.body mangles request.POST. See fix in ckan cset:e83bad9caa1b (problem with WebOb? v1.0.7 but may go back to v1.0.4?)

• A particular request caused an AssertionError? during looking at request.body, so we need to enclose this in a try/except. I couldn't recreate it in a test unfortunately (problem with WebOb? v1.0.7 but may go back to v1.0.2?)

Example:

Module ckan.controllers.api:206 in create
<<          log.debug('create: %s' % (context))
               try:
                   request_data = self._get_request_data()
               except ValueError, inst:
                   response.status_int = 400
>>  request_data = self._get_request_data()
Module ckan.lib.base:149 in _get_request_data
<<          cls.log.debug('Retrieving request params: %r' % request.params)
               cls.log.debug('Retrieving request POST: %r' % request.POST)
               cls.log.debug('Retrieving request POST body: %r' % request.body)
               if request.POST:
                   try:
>>  cls.log.debug('Retrieving request POST body: %r' % request.body)
Module paste.registry:137 in __getattr__
<<      
           def __getattr__(self, attr):
               return getattr(self._current_obj(), attr)
           
           def __setattr__(self, attr, value):
>>  return getattr(self._current_obj(), attr)
Module webob.request:470 in _body__get
<<          Return the content of the request body.
               """
               self.make_body_seekable() # we need this to have content_length
               r = self.body_file.read(self.content_length)
               self.body_file.seek(0)
>>  self.make_body_seekable() # we need this to have content_length
Module webob.request:697 in make_body_seekable
<<              self.body_file_raw.seek(0)
               else:
                   self.copy_body()
>>  self.copy_body()
Module webob.request:714 in copy_body
<<                  self.body = self.body_file_raw.read(length)
               elif self.is_body_readable:
                   self.body = self.body_file_raw.read()
                   self._copy_body_tempfile()
               else:
>>  self.body = self.body_file_raw.read()
Module webob.request:1190 in read
<<      def read(self, size=-1):
               body = self._get_body()
               if size < 0:
                   v = body[self.position:]
>>  body = self._get_body()
Module webob.request:1207 in _get_body
<<                  self._body = _encode_multipart(self.vars, self.content_type)
                   else:
                       assert 0, ('Bad content type: %r' % self.content_type)
               return self._body
>>  assert 0, ('Bad content type: %r' % self.content_type)
AssertionError: Bad content type: '; charset=utf-8'

It could be interesting for the publicdata.eu project to add some CSW datasources. Thanks to the INSPIRE directive, there are quite a few national catalogs of geodata that could be worth harvesting. Work on this should include:

• Compose Endpoint List (1d)
• Harvest on staging system and evaluate which sources have useful material (2d)

First point is being performed in this GDoc: ​https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&authkey=CNu4hsEB&hl=en_US#gid=0

The new formalchemy stores dict in a new format.

Markdown produces Debug logging, even when the pylons logging config is set to level of Info or Warning. This clogs up the logfile.

A nice map in the homepage showing the availability of data across Europe

Go to ​http://ckan.net/user/register and fill in all the fields apart from password. On submit you get 500 error.

I have followed the instruction for installation a configuration of ckanext-storage, but I am getting this error when I try to access the /storage/upload url:

URL: ​http://ckan.emap.fgv.br/storage/upload Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:152 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:130 in call << environSCRIPT_NAME? = environSCRIPT_NAME?[:-1]

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:124 in call << return % paste.registry.restorer.get_request_id(environ)?

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

controller = self.resolve(environ, start_response)

Module pylons.wsgiapp:263 in resolve << if self.log_debug:

log.debug("Resolved URL to controller: %r", controller)

return self.find_controller(controller)

def find_controller(self, controller):

return self.find_controller(controller)

Module ckan.config.environment:39 in find_controller << # Check to see if its a dotted name

if '.' in controller or ':' in controller:

mycontroller = pkg_resources.EntryPoint?.parse('x=%s' % controller).load(False) self.controller_classes[controller] = mycontroller return mycontroller

mycontroller = pkg_resources.EntryPoint?.parse('x=%s' % controller).load(False)

Module pkg_resources:1954 in load << def load(self, require=True, env=None, installer=None):

if require: self.require(env, installer) entry = import(self.module_name, globals(),globals(), __name__?) for attr in self.attrs:

try:

entry = import(self.module_name, globals(),globals(), __name__?)

Module ?:87 in <module> << class StorageAPIController(BaseController?):

ofs = get_ofs()

class StorageAPIController(BaseController?):

Module ?:88 in StorageAPIController << class StorageAPIController(BaseController?):

ofs = get_ofs()

@jsonpify

ofs = get_ofs()

Module ?:69 in get_ofs << continue

kw[k[4:]] = v

ofs = get_impl(storage_backend)(kw) return ofs

ofs = get_impl(storage_backend)(kw)

Module ofs.local.pairtreestore:26 in init << self.hashing_type = hashing_type

self.shorty_length = shorty_length self._open_store()

def _open_store(self):

self._open_store()

Module ofs.local.pairtreestore:30 in _open_store << def _open_store(self):

if self.hashing_type:

self._store = PairtreeStorageClient?(self.uri_base, self.storage_dir, shorty_length=self.shorty_length, hashing_type=self.hashing_type)

else:

self._store = PairtreeStorageClient?(self.uri_base, self.storage_dir, shorty_length=shorty_length)

self._store = PairtreeStorageClient?(self.uri_base, self.storage_dir, shorty_length=self.shorty_length, hashing_type=self.hashing_type)

Module pairtree.pairtree_client:91 in init << self._decode = re.compile(r"\^{(..)", re.U)}

self._init_store()

def char2hex(self, m):

self._init_store()

Module pairtree.pairtree_client:264 in _init_store << self.uri_base = prefix

else:

raise NotAPairtreeStoreException

if not os.path.isdir(self.store_dir):

raise NotAPairtreeStoreException

NotAPairtreeStoreException: CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'utma=4669863.1260802706.1295364409.1306935693.1306960483.86; utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; utmc=4669863; hide_welcome_message=1' HTTP_HOST 'ckan.emap.fgv.br' HTTP_KEEP_ALIVE '115' HTTP_USER_AGENT 'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1' PATH_INFO '/storage/upload' PATH_TRANSLATED '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/storage/upload' REMOTE_ADDR '10.250.48.110' REMOTE_PORT '57426' REMOTE_USER u'flavio' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [] REQUEST_METHOD 'GET' REQUEST_URI '/storage/upload' SCRIPT_FILENAME '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py' SERVER_ADDR '10.252.2.60' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.emap.fgv.br' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.16 (Debian)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7fc2999a6cd0> beaker.cache <beaker.cache.CacheManager? object at 0x7fc2999a6d90> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7fc2999a6d10>> beaker.session {'locale': u'pt_BR', '_accessed_time': 1307373458.4790759, '_creation_time': 1302605470.0245121} mod_wsgi.application_group 'dck093|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio' hide_welcome_message='1'>, 'utma=4669863.1260802706.1295364409.1306935693.1306960483.86; utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid _type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221fla...lcome_message=1') paste.registry <paste.registry.Registry object at 0x7fc299de4f50> paste.throw_errors True pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7fc299e27d50> pylons.routes_dict {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'} repoze.who.identity <repoze.who identity (hidden, dict-like) at 140473773131872> repoze.who.logger <logging.Logger instance at 0x7fc2999a3b48> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 140473779397008>, 'friendlyform': <FriendlyFormPlugin? 140473777417040>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7fc299b93a50>, 'auth_tkt': <AuthTktCookiePlugin? 140473779397136>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7fc299b934d0>} routes.route <routes.route.Route object at 0x7fc299864090> routes.url <routes.util.URLGenerator object at 0x7fc299e27ed0> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fc299dae4e0> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7fc299e27ed0>, {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'}) Reply Reply to all Forward Reply to all paste@localhost to fccoelho show details 10:17 (2 hours ago)

• Show quoted text -

CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'utma=4669863.1260802706.1295364409.1306935693.1306960483.86; utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; utmc=4669863; hide_welcome_message=1' HTTP_HOST 'ckan.emap.fgv.br' HTTP_KEEP_ALIVE '115' HTTP_USER_AGENT 'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1' PATH_INFO '/storage/upload' PATH_TRANSLATED '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/storage/upload' REMOTE_ADDR '10.250.48.110' REMOTE_PORT '57429' REMOTE_USER u'flavio' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [] REQUEST_METHOD 'GET' REQUEST_URI '/storage/upload' SCRIPT_FILENAME '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py' SERVER_ADDR '10.252.2.60' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.emap.fgv.br' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.16 (Debian)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7fc2999a6cd0> beaker.cache <beaker.cache.CacheManager? object at 0x7fc2999a6d90> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7fc2999a6d10>> beaker.session {'locale': u'pt_BR', '_accessed_time': 1307373466.085686, '_creation_time': 1302605470.0245121} mod_wsgi.application_group 'dck093|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio' hide_welcome_message='1'>, 'utma=4669863.1260802706.1295364409.1306935693.1306960483.86; utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid _type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221fla...lcome_message=1') paste.registry <paste.registry.Registry object at 0x7fc29a9da850> paste.throw_errors True pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7fc29a9ef950> pylons.routes_dict {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'} repoze.who.identity <repoze.who identity (hidden, dict-like) at 140473790084352> repoze.who.logger <logging.Logger instance at 0x7fc2999a3b48> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 140473779397008>, 'friendlyform': <FriendlyFormPlugin? 140473777417040>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7fc299b93a50>, 'auth_tkt': <AuthTktCookiePlugin? 140473779397136>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7fc299b934d0>} routes.route <routes.route.Route object at 0x7fc299864090> routes.url <routes.util.URLGenerator object at 0x7fc29a9efcd0> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fc29a9d7d50> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7fc29a9efcd0>, {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'})

Logic layer should not use any vdm defined state and should manage it itself.

As CKAN grows, more and more optional representations of packages will become used. Besides RDF (which is the motivation for this ticket), support for DSPL, home-grown XML, or CSV listings is imaginable.

To properly support this CKAN should have an extensible representation registry that can be extended when new output types become available (and without changing the API in the process). This needs to integrate in two places:

• We need to add support for file format extensions to the package handler and REST API.
• We need to add HTTP content type negotiation (Accept headers).

To support this we should have a registry with two registers:

• to map format extensions to mime types (e.g. "json" -> "application/json")
• to map mime types and entity types to converter functions (e.g. 8"application/json", ckan.model.Package) -> func(obj, mime_type) )

This should be set up on load_environment so that IConfigurer plugins can feed into it.

The registry should then first be added to _finish in the REST API (which needs to be refactored to be passed the {format} part of the URL if one is given. Based on the format part and HTTP headers, an appropriate representation can be generated by the registry and then be returned to the user.

Forwarding of requests to the regular WUI controllers with Accept headers set or a format specified can be implemented in a separate effort.

User can insert bad anchor tags into the User-About and Package-Notes fields and when you view them (web interface) it causes a 500 error.

Need to improve filtering for anchors in markdown.

<a href="http://xxxsex.com>nasty/website

Also check this related exception:

Module ckan.controllers.user:59 in read
<<          c.is_myself = user.name == c.user
               c.api_key = user.apikey
               c.about_formatted = self._format_about(user.about)
               revisions_q = model.Session.query(model.Revision
                       ).filter_by(author=user.name)
>>  c.about_formatted = self._format_about(user.about)
Module ckan.controllers.user:167 in _format_about
<<      def _format_about(self, about):
               about_formatted = ckan.misc.MarkdownFormat().to_html(about)
               return genshi.HTML(about_formatted) 
       
           def _get_form_password(self):
>>  return genshi.HTML(about_formatted)
WebApp Error: <class 'genshi.input.ParseError'>: junk characters in start tag: u'\u201dhttp://www.settingu': line 1, column 3

Hello,

I have done a recent update to mu ckan install, and I am now getting this error for all my packages:

URL: http://ckan.emap.fgv.br/package/dengue-net
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
          
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
              
               response = self.app(environ, start_response)
              
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<         
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
              
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
          
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']   
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
              
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                  
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
          
           def _inspect_call(self, func):
>>  return func(**args)
Module ?:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__("logging").getLogger("proxy_cache")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
      
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.package:208 in read
<<          #render the package
               PackageSaver().render_package(c.pkg)
               return render('package/read.html')
      
           def comments(self, id):
>>  return render('package/read.html')
Module ckan.lib.base:74 in render
<<     
           return cached_template(template_name, render_template, cache_key=cache_key,
                                  cache_type=cache_type, cache_expire=cache_expire)
                                  #, ns_options=('method'), method=method)
>>  cache_type=cache_type, cache_expire=cache_expire)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:63 in render_template
<<              stream = item.filter(stream)
              
               return literal(stream.render(method=method, encoding=None))
          
           if 'Pragma' in response.headers:
>>  return literal(stream.render(method=method, encoding=None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
      
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out=out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:339 in __call__
<<          for filter_ in self.filters:
                   stream = filter_(stream)
               for kind, data, pos in stream:
                   cached = cache_get((kind, data))
                   if cached is not None:
>>  for kind, data, pos in stream:
Module genshi.output:670 in __call__
<<          _gen_prefix = _gen_prefix().next
      
               for kind, data, pos in stream:
                   output = cache_get((kind, data))
                   if output is not None:
>>  for kind, data, pos in stream:
Module genshi.output:771 in __call__
<<          push_text = textbuf.append
               pop_text = textbuf.pop
               for kind, data, pos in chain(stream, [(None, None, None)]):
      
                   if kind is TEXT:
>>  for kind, data, pos in chain(stream, [(None, None, None)]):
Module genshi.output:586 in __call__
<<      def __call__(self, stream):
               prev = (None, None, None)
               for ev in stream:
                   if prev[0] is START:
                       if ev[0] is END:
>>  for ev in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
      
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.markup:378 in _match
<<                      for event in self._match(self._flatten(template, ctxt,
                                                                  **vars),
                                                    ctxt, start=idx + 1, **vars):
                               yield event
>>  ctxt, start=idx + 1, **vars):
Module genshi.template.markup:327 in _match
<<                      break
      
               for event in stream:
      
                   # We (currently) only care about start and end events for matching
>>  for event in stream:
Module genshi.template.base:565 in _flatten
<<                  elif kind is EXPR:
                           result = _eval_expr(data, ctxt, vars)
                           if result is not None:
                               # First check for a string, otherwise the iterable test
>>  result = _eval_expr(data, ctxt, vars)
Module genshi.template.base:277 in _eval_expr
<<      if vars:
               ctxt.push(vars)
           retval = expr.evaluate(ctxt)
           if vars:
               ctxt.pop()
>>  retval = expr.evaluate(ctxt)
Module genshi.template.eval:178 in evaluate
<<          __traceback_hide__ = 'before_and_this'
               _globals = self._globals(data)
               return eval(self.code, _globals, {'__data__': data})
>>  return eval(self.code, _globals, {'__data__': data})
Module ?:16 in <Expression u"h.subnav_link(c, h.icon('comments') + _('Comments & Questions'), controller='package', action='comments', id=c.pkg.name)">
<<        </li>
             <li py:if="g.has_commenting">
                 ${h.subnav_link(c, h.icon('comments') + _('Comments &amp; Questions'), controller='package', action='comments', id=c.pkg.name)}
             </li>
             <li>${h.subnav_link(c, h.icon('page_white_stack') + _('History'), controller='package', action='history', id=c.pkg.name)}</li>
>>  ${h.subnav_link(c, h.icon('comments') + _('Comments &amp; Questions'), controller='package', action='comments', id=c.pkg.name)}
Module ckan.lib.helpers:126 in subnav_link
<<      return link_to(
               text,
               url_for(action=action, **kwargs),
               class_=('active' if c.action == action else '')
           )
>>  url_for(action=action, **kwargs),
Module routes.util:280 in url_for
<<          raise GenerationException(
                   "url_for could not generate URL. Called with args: %s %s" % \
                   (args, kargs))
           return url
>>  (args, kargs))
GenerationException: url_for could not generate URL. Called with args: () {'action': 'comments', 'controller': 'package', 'id': 'dengue-net'}
CGI Variables
AUTH_TYPE 	'cookie'
DOCUMENT_ROOT 	'/etc/apache2/htdocs'
GATEWAY_INTERFACE 	'CGI/1.1'
HTTP_ACCEPT 	'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
HTTP_ACCEPT_CHARSET 	'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING 	'gzip, deflate'
HTTP_ACCEPT_LANGUAGE 	'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3'
HTTP_CONNECTION 	'keep-alive'
HTTP_COOKIE 	'__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; __utmc=4669863'
HTTP_HOST 	'ckan.emap.fgv.br'
HTTP_KEEP_ALIVE 	'115'
HTTP_REFERER 	'http://ckan.emap.fgv.br/'
HTTP_USER_AGENT 	'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1'
PATH_INFO 	'/package/dengue-net'
PATH_TRANSLATED 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/package/dengue-net'
REMOTE_ADDR 	'10.250.48.110'
REMOTE_PORT 	'47624'
REMOTE_USER 	u'flavio'
REMOTE_USER_DATA 	'userid_type:unicode'
REMOTE_USER_TOKENS 	['']
REQUEST_METHOD 	'GET'
REQUEST_URI 	'/package/dengue-net'
SCRIPT_FILENAME 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py'
SERVER_ADDR 	'10.252.2.60'
SERVER_ADMIN 	'[no address given]'
SERVER_NAME 	'ckan.emap.fgv.br'
SERVER_PORT 	'80'
SERVER_PROTOCOL 	'HTTP/1.1'
SERVER_SIGNATURE 	'<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n'
SERVER_SOFTWARE 	'Apache/2.2.16 (Debian)'
WSGI Variables
application 	<beaker.middleware.CacheMiddleware object at 0x7f051c802690>
beaker.cache 	<beaker.cache.CacheManager object at 0x7f051c802750>
beaker.get_session 	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f051c8026d0>>
beaker.session 	{'locale': u'pt_BR', '_accessed_time': 1307358102.6266389, '_creation_time': 1302605470.0245121}
mod_wsgi.application_group 	'dck093|'
mod_wsgi.callable_object 	'application'
mod_wsgi.handler_script 	''
mod_wsgi.input_chunked 	'0'
mod_wsgi.listener_host 	''
mod_wsgi.listener_port 	'80'
mod_wsgi.process_group 	'ckan'
mod_wsgi.request_handler 	'wsgi-script'
mod_wsgi.script_reloading 	'1'
mod_wsgi.version 	(3, 3)
paste.cookies 	(<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio'>, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode";... __utmc=4669863')
paste.registry 	<paste.registry.Registry object at 0x7f051d7e5710>
paste.throw_errors 	True
pylons.action_method 	<bound method PackageController.read of <ckan.controllers.package.PackageController object at 0x7f051d7e5b10>>
pylons.controller 	<ckan.controllers.package.PackageController object at 0x7f051d7e5b10>
pylons.environ_config 	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons 	<pylons.util.PylonsContext object at 0x7f051d7e5a50>
pylons.routes_dict 	{'action': u'read', 'controller': u'package', 'id': u'dengue-net'}
repoze.who.identity 	<repoze.who identity (hidden, dict-like) at 139659946193344>
repoze.who.logger 	<logging.Logger instance at 0x7f051c801128>
repoze.who.plugins 	{'openid': <OpenIdIdentificationPlugin 139659931643152>, 'friendlyform': <FriendlyFormPlugin 139659929725584>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f051c9e7610>, 'auth_tkt': <AuthTktCookiePlugin 139659931643280>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f051c9dcfd0>}
routes.route 	<routes.route.Route object at 0x7f051c69fd50>
routes.url 	<routes.util.URLGenerator object at 0x7f051d7e57d0>
webob._parsed_cookies 	({'ckan': 'd4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06', '__utmz': '4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'auth_tkt': 'c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode', 'ckan_display_name': 'Fl\xc3\xa1vio Code\xc3\xa7o Coelho', 'ckan_apikey': 'c139718d-918f-4a2b-b219-e33cb05cbe23', 'ckan_user': 'flavio', '__utma': '4669863.1260802706.1295364409.1306935693.1306960483.86', '__utmc': '4669863', 'SESS61faf7f7d2406929b2f9eb4cbfff17d2': 'a3306f5d1194cf0d8a51dbf7b42c1259'}, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"< wbr>; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!... __utmc=4669863')
webob._parsed_query_vars 	(GET([]), '')
webob.adhoc_attrs 	{'language': 'en-us'}
wsgi process 	'Multi process AND threads (?)'
wsgi.file_wrapper 	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f051cc1ad50>
wsgi.version 	(1, 1)
wsgiorg.routing_args 	(<routes.util.URLGenerator object at 0x7f051d7e57d0>, {'action': u'read', 'controller': u'package', 'id': u'dengue-net'})

Trying to run the tests in test_authz.py with

$ nosetests --ckan ckan/tests/functional/test_authz.py results in no tests being run:

Ran 0 tests in 0.840s

OK (SKIP=3)

Update CKAN wiki front page - a la OpenSpending?: ​http://wiki.openspending.org/Main_Page

Sections should relate to different types of people using the site:

Developers, Users etc..

Need to cover pip-requirements.txt on different branches in the doc/deployment.rst.

Also could do with tidying up upgrade.txt into this document too.

An internal server error was reported when trying to create a new package (not when editing):

WebApp? Error: <class 'sqlalchemy.exceptions.IntegrityError?'>: (IntegrityError?) null value in column "name" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'}

The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the "groups" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server.

Possible causes include:

• Stale pyc file
• Threading issue
• Dependency issue

As the problem is gone for now, it cannot be further investigated atm.

• Should license go in the bottom line?
• Formats should be styled as in CKAN

Write a harvester for data.london.gov.uk to import catalogue metadata into PDEU. API (or at least documentation) is available at: ​http://sourceforge.net/projects/londondatastore/files/

• data.london.gov.uk (1d, ping DERI)
• catalogues from Paris and Vienna (1d)

Others?

• digitaliser.dk (1d)
• data-publica.com (2d)
• data.suomi.fi (2d)
• dati.piedmonte.it (1d)
• portalu.de (2d)

This should make it more efficient as it currently does a lot of repeating work. i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1.

Write CKAN FAQ (Basis can be: ​http://wiki.ckan.net/FAQ)

Post preliminary questions on:

​http://notebook.okfn.org/

An extension that provides a todo list feature on CKAN so that people can register and find things to do.

Extension name: ckanext-todo

User Story

Package page

As a user I come to a package:

• Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
• At the bottom is a section of the package display titled "ToDo?" where I see a list of all toDos for the package most recent at the top
  • If I am logged in
    • See a form for "Add to do" at the top of the todo section and can add one straight away
    • I see a "now resolved" button next to each which goes green when you hover.

When clicked the todo fades away.

• Not logged in: I see a button that says "login to add todo"

Todo list page

When a user comes to todo overview at /todo

At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category.

Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo.

Implementation

The Todo form

• One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed)
• Add a description
• Submit, the todo gets added via AJAX to the list at the top as the most recent todo

Model:

todo table

  id (autoincrement integer)
  package_id
  todo_category_id (required)
  description (required)
  created=NOW()
  resolved=null (unresolved) or a datetime (datetime of resolution)
  creator=user
  resolver=user

todo_category table
  id
  name

Prepopulate with: broken-resource-link, no-author, bad-format, add-description

API at /api/2/todo

• GET / POST / PUT ...

/api/2/todo?package=package_id_or_name&category=...&resolved=0/1

• support limit (?)

/api/2/todo/category -> return list of todo categories

• No GET / PUT / POST (these are auto-created by creation of todo)

Optional Extras (Will not be done atm)

• Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ...

The logic layer is a bit too api centric. Make the reusable parts separate in preparation for the wui refactor.

the add_user_to_role/remove_user_from_role functions are asymmetrical in that the add function is happy to add the same role twice but the remove asserts that it's only in the table once and crashes if that's not true.

an attempt has been made to guard against this, but fails, I think because the add functions rely on the caller committing the change to the db.

same problem affects corresponding authorization_group functions

I'll try to sort this out. Making a note here.