{22} Trac tickets (2647 matches)

When you change the locale, it displays a flash message "Language has been set to: English" in the original language, not the one we have just changed to.

IndexError? exception in genshi.filters.i18n keeps popping up daily.

e.g.

...
Module ckan.controllers.authorization_group:28 in index
<<              items_per_page=20
               )
               return render('authorization_group/index.html')
       
           def read(self, id):
>>  return render('authorization_group/index.html')
Module ckan.lib.base:74 in render
<<      
           return cached_template(template_name, render_template, cache_key=cache_key, 
                                  cache_type=cache_type, cache_expire=cache_expire)
                                  #, ns_options=('method'), method=method)
>>  cache_type=cache_type, cache_expire=cache_expire)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:63 in render_template
<<              stream = item.filter(stream)
               
               return literal(stream.render(method=method, encoding=None))
           
           if 'Pragma' in response.headers:
>>  return literal(stream.render(method=method, encoding=None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
       
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out=out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:339 in __call__
<<          for filter_ in self.filters:
                   stream = filter_(stream)
               for kind, data, pos in stream:
                   cached = cache_get((kind, data))
                   if cached is not None:
>>  for kind, data, pos in stream:
Module genshi.output:670 in __call__
<<          _gen_prefix = _gen_prefix().next
       
               for kind, data, pos in stream:
                   output = cache_get((kind, data))
                   if output is not None:
>>  for kind, data, pos in stream:
Module genshi.output:771 in __call__
<<          push_text = textbuf.append
               pop_text = textbuf.pop
               for kind, data, pos in chain(stream, [(None, None, None)]):
       
                   if kind is TEXT:
>>  for kind, data, pos in chain(stream, [(None, None, None)]):
Module genshi.output:586 in __call__
<<      def __call__(self, stream):
               prev = (None, None, None)
               for ev in stream:
                   if prev[0] is START:
                       if ev[0] is END:
>>  for ev in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module ckanext.googleanalytics.plugin:62 in download_adder
<<              (downloaded %s times)</span>'''
                   count = None
                   for mark, (kind, data, pos) in stream:
                       if mark and kind == START:
                           href = data[1].get('href')
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1175 in __call__
<<          """
               callable_value = hasattr(self.value, '__call__')
               for mark, (kind, data, pos) in stream:
                   if mark is ENTER:
                       if callable_value:
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
       
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.markup:378 in _match
<<                      for event in self._match(self._flatten(template, ctxt,
                                                                  **vars),
                                                    ctxt, start=idx + 1, **vars):
                               yield event
>>  ctxt, start=idx + 1, **vars):
Module genshi.template.markup:327 in _match
<<                      break
       
               for event in stream:
       
                   # We (currently) only care about start and end events for matching
>>  for event in stream:
Module genshi.template.base:545 in _flatten
<<          while 1:
                   for kind, data, pos in stream:
       
                       if kind is START and data[1]:
>>  for kind, data, pos in stream:
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range

I can't add a package (e.g. ​http://ckan.net/package/ub-konstanz) to a group (e.g. ​http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is "history") nor on the group page.

As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually.

The current codebase has one or two try: except; blocks that don't catch specific exceptions. Under no circumstances should any broad try: except: blocks be allowed unless the exceptions they catch are immediately re-raised. Uncaught exceptions are wasting us quite a lot of time when trying to track down problems.

When installing CKAN, when doing "paster serve development.ini", lots of users encounter the error for every request:

ProgrammingError: (ProgrammingError) relation "user" does not exist

This is because the database tables have not been created - they have forgotten or missed the "paster db init" step.

Can we provide a better error to say that the database is not initialised yet?

Implementation options

1. At the start of every request we reflect the database tables and check they are there. This is rather expensive!

2. Uncached requests to the home page start with a cheap database query. If there is an exception then return this error about database setup. I really like Drupal's page for this that has in large letters that the site is currently off-line. Below a line, in small letters, there are developer suggestions on what is wrong and where to look to fix it.

According to pudo, a ckan with activated solr extension throws a 5xx when solr is unreachable. Instead, it should behave more like a ckan without ckanext-solr when this happens.

Consolidate js. Remove cruft.

• Move stuff to vendor
  • Have all vendor libs local (?)
  • jquery, jquery-ui etc
• Consolidate autocomplete to use jquery-ui
  • package autocomplete (groups)
  • tag autocomplete (package)
  • user autocomplete (authz group create)
    • turns out that authz group edit had autocomplete disabled so refactoring meant enabling ...
• Move forms.css into main
• Remove tag cloud from front page (can be example going forward)
• Remove bookmarklet (since in wiki at: ​http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F)
• Move name slug generation into standard js and clean up

Possible:

1. JS unit tests for the given pages ...
2. Option to use external site to serve js and css (e.g. CDN)

This was removed during the auth refactoring, but it's needed in the old controllers

Only happens if the user does not update the password. Most probably introduced during #1229.

Revisions aren't assigned to packages. Where are they coming from? This seems to have been occurring since the upgrade to 1.4.3b a couple of weeks ago?

Subticket of: #1294

Rather than having to visit a dedicated page it would be good if registration and login could be done from a modal form (separate or combined ...).

Why

• It could be used from dataset creation page in situations where user needs to be registered / logged in to create a dataset (so we could allow someone to start creating a dataset and only get them to login at the end ...)
• It allows for quicker and easier logging in

Implementation

• See Friedrich's work on the datahub

From Florian:

I am having trouble with the characters "=&;".
if they occur e.g. in the title of a package, I get a 400-Bad-Request
when I try to register a package by POSTing to ckan/rest/api/package

However, when I PUT the same package as an edit to
ckan/rest/api/package/existing_package, it works and the title shows
correctly on the front end and API.

Reproduced

(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{"name": "test3", "title": "Test &"}' -H "Authorization:tester" 
HTTP/1.0 400 Bad Request
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 14:09:31 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Length: 87

"Bad request - JSON Error: Unterminated string starting at: line 1 column 27 (char 27)"

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in:

• 'with' statement requires from __future__ import with_statement
• import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5.

e.g. this particular revision on ckan.net: ​http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c (most others have not been approved) Causes 500 error.

Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor.

3 (4) use cases: (With and without access to auth keys.)

1. upload form on ckan (by default local system has auth key for storage)
2. datapkg upload (client) (by default without key) (plumbing done)
3. background job archiving downloads (by queue client)

Most important are:

1. Upload form on ckan.net (with access to keys)
2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

• #826 - Resource 'extra' fields
• #358 - Resources in REST API
• #235 - Resource format normalization and detection

User upload via Web UI

• #877 - File upload in WUI
• #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

• #853 - plumbing for credential-less rest upload
• #879 - Storage auth API

Also related:

• #405 - Retrieval options for package resources

More Info

• Upload/storage discussion: ​http://ckan.okfnpad.org/storage
  • For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/

Provide an API to the logic layer at /api/action.

Set language to Greek, flash message says 'Language set to: English', but page is now about half in Greek.

Set language back to English causes server error:

AttributeError?: 'NoneType?' object has no attribute 'path'

Module ckan.controllers.error:29 in document view

if original_request.path.startswith('/api'):

However going to a new page reveals that it's back to English

Subticket of #1294.

• Nice way to edit all resource attributes
• Nice way to add new resources
  • Including file upload (cf #878).
    • Cf work in ckanjs (this can be a straight integration).

add extra fields to resource as outlined in

​http://wiki.ckan.net/Domain_Model/Resource

Specifically (all unicode strings except for those noted):

• name: a name for this resource (could be used in a ckan url)
• type: the type of the resource. One of: file | api | service | listing
• mimetype: standard mimetype (e.g. for zipped csv would be application/zip)
• mimetype_inner: mimetype of innermost object (so for example would be text/csv)
• size: [integer] size of the resource (content length) in bytes. Usually only relevant for resources of type file.
• last_modified: [datetime] the date when this resource's data was last modified (NB: not the date when the metadata was modified).
• cache_url: url for cache of object in ckan storage
• cache_last_updated: [datetime] timestamp when cached version was created
• webstore_url:
• webstore_last_updated: [datetime] timestamp when webstore was last updated

See ​http://ckan.okfnpad.org/search for details/discussion

At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas:

• #191 Searching by modification date
• #193 Searching by time-related field
• #301 Package discussion pages
• #1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages
• #1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page
• Other improvements as per my word doc.

Add tables described in ​http://ckan.okfnpad.org/notifications

We get this i18n error for the URL ​http://no.ckan.net/authorizationgroup (which runs 1.4.2 with Norwegian translation). I'm not sure why it occurs. Fixing it is low importance though, because this site doesn't use Authorization Group feature

Usually caused by bots. The user does not appear to be logged in.

WebApp Error: <type 'exceptions.IndexError'>: list index out of range
...
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

This stems from a feature to allow you to prefill fields in the authorizationgroup creation form.

e.g. ​http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it.

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org.

Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install).

Propose:

• ckan.simple_search config option
• If set:
  • query via simple query to database backend
  • do no specialized indexing

Extras

Remove

• TagSearchQuery? from lib/search (just do a search directly)?
• #1360: filter by downloadable and filter by open search options

Possible Extras

(Probably future improvements)

• Re-introduce full-text search indexing where supported in e.g. postgres and use for querying

Resource and Tag search is currently still using the old ckan query parser which is unnecessary. Refactor this out and move to the logic layer.

Inline with general naming policy going forward.

Right now, for the "View available WMS layers" link to be displayed, the package needs to have an "INSPIRE=True" extra, a "resource-type=service" extra and a resource of type "WMS".

This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough.

(The repository for this is ckanext-spatial, but it's not on the list)

manifest.cfg was not included in the pypi package, causing InvalidRepositoryError? when you do 'paster db init'.

Appears to affect most CKAN PyPI releases up to and including 1.4.3.

Here is Rufus' plan for ckan.org reorganisation from ​http://ckan.okfnpad.org/documentation (22/7/11) (ticket made by dread):

/solutions
  /overview <--- current 'about' page (the software)
  /customers <-- say "in addition to our clients..."
  /features <-- needs most work (pictures etc)
/pricing/  <-- simplest thing possible ... talk with Ira (cover pricing and support)
/developers <--- links to http://docs.ckan.org/, getting the software, the wiki, the mailing list /contact
/blog

Tasks:

• add tagline in header - "the data portal software"
• make and add glossy PDF
• change "data hub" to "data portal" passim (if only using one term)
• add info@…
• image banner on home page - redo screenshots without menus

Changes

• All templates
• [DONE] Routing: /package/ -> /dataset/ (including API)
• [DONE] Tests: self.app.get('/dataset/...') rather than package ...
  • One test to check redirect works?
• Docs

Leave

• Stay with Package in code e.g. stay with 'PackageController?'
  • also leave stuff like pkg = .... (gradual migration)
• Leave extensions

Notes

• Package -> Dataset
• Data Package -> Dataset
• respect capitalization

Deliver little user benefit and easier just to search and then see what is listed as open and with resources.

Implementation Details

Areas to fix:

• UI
• lib/search/...
• tests
• ...?

Cost

Est: 2h

Various things in templates can be tidied up after work has finished on #1294:

• remove styling in page in group/index.html (put in css main)
  • ditto for package/search.html
  • ditto for home/index.html
• Forms.css - to merge into style.css; remove explicit imports elsewhere
  • @import at the top of style.css @import url(forms.css)
• Make a note: Success/failure controllers clash CSS with success/failure flash banners

The template for editing a user's profile seems to output the same form twice in one page. Furthermore, the page is made needlessly complex by the preview functionality. The neat inline preview used in package will make this page a lot neater.

• Fix the duplicate form
• Strip the preview functionality from the front-end, replacing it with js editor
• Strip the preview functionality from the back-end and test

We have an all new and much improved data preview system developed as part of QA in July. Need to integrate this.

• Is pure javascript
• Will have this in CKAN core but considered to be a plugin (so need datapreview in ckan.plugins list to work -- will enable this by default)
• Will depend on the ​http://wiki.ckan.org/DataExplorer

ckan.js can be rearchitected to avoid instantiating showdown.js in the root namespace. That way, layout_base.html doesn't need to import the script (because we don't use the showdown-dependant parts of ckan.js).

(2008-10-29) convert existing directory on disk to a datapkg distribution (PythonDistribution? by default). This would be presented in the CLI as a convert command.

Details:

• Add a metadata file (setup.py), perhaps prompting for input
• List all existing data in that directory

Cost: 4h

This extension should be in core now -- functionality is basic and very useful.

Copy across from here: ​http://bitbucket.org/okfn/ckanext-admin

Suggest this no longer even be an extension.

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN

## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png

## Site tagline / description (used on front page)
ckan.site_description = 

## Used in creating some absolute urls (such as rss feeds, css files) and 
## dump filenames
ckan.site_url =

## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico


## An 'id' for the site (using, for example, when creating entries in a common search index) 
## If not specified derived from the site_url
# ckan.site_id = ckan.net

## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net

## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end = 

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

• [namespace]: ? only if KeyValues? (for settings this would then always be settings)
• key
• label
• value (json)
• type (e.g. date and to specify in advance what type should be)
• description
• tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

• /ckan-admin/settings
• Show label, plus description plus text field

Depends

• Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)

When paster db clean is run, the search index should also be cleared.

At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs.

Suggest we write a script that automatedly generates reference documentation for the config from the source.

May be obsoleted by #277 (some config in db)

At the moment this appears to be slow (kindly suggests because missing index in db). Page is /user on a ckan instance.

Would this also speed up user page where we recent changes?

We should ignore these but log the packages that are causing this to investigate.