{22} Trac tickets (2647 matches)

Check to see that all the strings are getting translated, i.e. that all user-visible strings are being passed to gettext. Fix any strings that are not getting translated.

Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words.

Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.:

'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible')

​http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting

We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated.

User story #035 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

You get a UnicodeDecodeError?, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example.

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

Solr install instructions install the wrong version

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

We want a much quicker and easier way of deploying multiple CKAN instances to the same or multiple servers and managing them (e.g. configuration, themes, extensions, upgrades, and adding new instances). Currently we just deploy and manage each instance separately, which doesn't scale.. We'd also want to have (as far as possible) one single way of deploying CKAN that works across different situations, e.g. deployment server, demo server, jenkins, local deployment for development, these should share the same tools for installing CKAN and its dependencies.

Components that this might include:

Virtualenv bootstrap script ​https://gist.github.com/2206132

Ansible ​http://ansible.github.com/

CKAN deb package

Check that lists of strings are being correctly translated. See ​https://github.com/okfn/ckan/commit/f1d68c3d2d4d25a0c0f8a89a68940643fc19b156

If you change the language to one where the word for 'Search' is longer than the English word (e.g. fr) then the search button wraps onto the next line, below the text field, which looks bad.

There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either:

ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled

There are also some options that are in the default deployment.ini even though they're deprecated:

ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously

See email to ckan-dev from David Read: ​http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html

It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358

Looks like recent CKAN cleanups broke some imports:

​https://gist.github.com/2935639

Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command.

May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command.

When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once.

It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests.

If you're not logged in and you click this button, it takes you to the add group page not the login page. Anyway, the default behaviour everywhere else in CKAN is to hide buttons that you don't have permission to use (e.g. add dataset, edit dataset, etc.)

Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered.

CKAN's internal tracking seems to count each page view twice, the problem appears to be with the SQL in the update_tracking() method in ckan/lib/cli.py.

The internal tracking feature needs some tests, and some of the code could maybe do with some more explanatory comments, e.g. what is the intended difference between count and running_total?

It doesn't pass extras_as_string

Luca De Santis reported these strings not marked in 1.7.1:

• /user/login.html ("Remember me" & "Sign in")
• /package/search.html and template/package/group/read.html (Tags, Resource Formats, Groups)
• /package/resource_read.html (the "No description" message)
• the "Add related item" dialog (some strings are i18n savvy, others not)
• the stats page (labels on the graphs)
• the history page (as far as the log messages are concerned).

lets check that they are marked in 1.8

Upload the new pot file/resource on transifex, put out the call for translations. Some recent commits on master need to be merged into 1.8 first

If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates.

On the organization read page there is only an 'Add Dataset' button, not a drop-down with add new dataset or add existing dataset as on the default group page.

On the organization edit page you do have the drop-down, but if you click add existing dataset nothing happens except it appends #datasets to the URL

From: <thedatahub.org@…> Date: Thu, Aug 16, 2012 at 3:20 AM Subject: WebApp? Error: <class 'genshi.template.eval.UndefinedError?'>: "tag_list" not defined To: ckan-sysadmin@…, kindly@…

URL: ​http://ckan.net/dataset/uk-postboxes Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:155 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:131 in call << r'\1', oldpath)

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:239 in call << try:

res = WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

res = WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.package:322 in read << template = template[:template.index('.') + 1] + format

return render(template, loader_class=loader)

def comments(self, id):

return render(template, loader_class=loader)

Module ckan.lib.base:153 in render << try:

return cached_template(template_name, render_template,

loader_class=loader_class)

except ckan.exceptions.CkanUrlException?, e:

raise

loader_class=loader_class)

Module pylons.templating:249 in cached_template << return content

else:

return render_func()

return render_func()

Module ckan.lib.base:102 in render_template << if loader_class == NewTextTemplate?:

return literal(stream.render(method="text", encoding=None))

return literal(stream.render(method=method, encoding=None,

return literal(stream.render(method="text", encoding =None))

Module genshi.core:183 in render << method = self.serializer or 'xml'

generator = self.serialize(method=method, kwargs) return encode(generator, method=method, encoding=encoding, out=out)

def select(self, path, namespaces=None, variables=None):

return encode(generator, method=method, encoding=encoding, out =out)

Module genshi.output:57 in encode << _encode = lambda string: string

if out is None:

return _encode(.join(list(iterator)))

for chunk in iterator:

out.write(_encode(chunk))

return _encode(.join(list(iterator)))

Module genshi.output:569 in call << def call(self, stream):

strip_markup = self.strip_markup for event in stream:

if event[0] is TEXT:

data = event[1]

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module ckanext.googleanalytics.plugin:93 in download_adder << [downloaded %s times]</span>

count = None for mark, (kind, data, pos) in stream:

if mark and kind == START:

href = data[1].get('href')

for mark, (kind, data, pos) in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1175 in call << """

callable_value = hasattr(self.value, 'call') for mark, (kind, data, pos) in stream:

if mark is ENTER:

if callable_value:

for mark, (kind, data, pos) in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.template.base:605 in _include << from genshi.template.loader import TemplateNotFound?

for event in stream:

if event[0] is INCLUDE:

href, cls, fallback = event[1]

for event in stream:

Module genshi.template.base:565 in _flatten << elif kind is EXPR:

result = _eval_expr(data, ctxt, vars) if result is not None:

# First check for a string, otherwise the iterable test

result = _eval_expr(data, ctxt, vars)

Module genshi.template.base:277 in _eval_expr << if vars:

ctxt.push(vars)

retval = expr.evaluate(ctxt) if vars:

ctxt.pop()

retval = expr.evaluate(ctxt)

Module genshi.template.eval:178 in evaluate << traceback_hide = 'before_and_this'

_globals = self._globals(data) return eval(self.code, _globals, {'data': data})

return eval(self.code, _globals, {'data': data})

Module ?:51 in <Expression u"tag_list(c.pkg_dict.get('tags', ))"> << <li py:if="c.pkg_dict.get('tags')" class="sidebar-section">

​https://github.com/okfn/ckanext-pdeu/issues/13

Requires 0.4. 0.5 is a maintenance branch, 0.6 is current.

VDM appears to work correctly (all the tests pass) with 0.5. CKAN does not -- requires more investigation to determine exactly why.

It would actually be nice to be able to run with 0.6 or 0.5, though 0.6 will require some changes to VDM as well

The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach.

We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!)

Some suggestions for achieving this include:

• Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database
• Not setting up and tearing down the database so frequently.

Refactor harvesting code to simplify.

As a User (especially as a Package Owner/Maintainer?) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

Implementation

• Use the existing support for this feature in google analytics or piwiki
• For google analytics see:
  • Integrate to record: ​http://www.google.com/support/analytics/bin/answer.py?answer=55529
  • Accessing analytics data via API (for showing download counts): ​http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html
• Limitations only record downloads by browsers with js turned on.
  • This is OK I think (in any case machine download via e.g. datapkg does not get downloaded)

Old Spec (do it in CKAN)

• Record info of form: resource id (or url?), timestamp
• Do this via javascript capturing of onclick event talking to an api
• API: /api/resource/{id}/download
  • POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
  • GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

Questions

• Do we record ip addresses (to handle de-botting etc)?
• Do we count preview clicks as well?

Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.

Issues

• Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961