{22} Trac tickets (2647 matches)

The right hand side of ​thedatahub.org login screen is a little confusing:

    Not a member?

    Join CKAN to contribute packages under your own name.

     - Register
     - Reset your password
     - Privacy Policy

Suggested improvements to make:

• replace CKAN with thedatahub.org
• in general, edit the copy to be more inviting
• remove "Reset your password" from that bubble
• move the privacy policy link to somewhere less prominent

All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: ​http://ckan.net/package

Exception received on test.ckan.net at 10.12 on 29/7/11

http://test.ckan.net/package?__cache=2759373&page=4
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:153 in search
<<                          }
       
                   query = get.package_search(context,data_dict)
       
                   c.page = h.Page(
>>  query = get.package_search(context,data_dict)
Module ckan.logic.action.get:514 in package_search
<<      for package in query.results:
               result_dict = table_dictize(package, context)
               result_dict = _extend_package_dict(result_dict,context)
       
               results.append(result_dict)
>>  result_dict = _extend_package_dict(result_dict,context)
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: None'

and a similar one:

 http://test.ckan.net/tag/barbecue
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.tag:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__("logging").getLogger("proxy_cache")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
       
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.tag:64 in read
<<          data_dict = {'id':id}
               try:
                   c.tag = get.tag_show(context,data_dict)
               except NotFound:
                   abort(404, _('Tag not found'))
>>  c.tag = get.tag_show(context,data_dict)
Module ckan.logic.action.get:341 in tag_show
<<      extended_packages = []
           for package in tag_dict['packages']:
               extended_packages.append(_extend_package_dict(package,context))
       
           tag_dict['packages'] = extended_packages
>>  extended_packages.append(_extend_package_dict(package,context))
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: CreativeCommonsAttributionCCBY25'

At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases.

Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard).

Options:

• Default this in solr (no need to touch code) but fragile and affects everything ...
• Do it in code and default to score
• Do it in code and have alphabetical (on name or title?) when no criteria otherwise score

Aside: may also wish to support search in query api but that is for later!

Deprecate ckan.net in favour of thedatahub.org

• Announce
• Create thedatahub.org and have it run same data as ckan.net
• Announce closing down of ckan.net with 302
• Implement 302 permanent

Also redirect wiki.ckan.net to wiki.ckan.org

Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers.

In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory.

This extension should be in core now -- functionality is basic and very useful.

Copy across from here: ​http://bitbucket.org/okfn/ckanext-admin

Suggest this no longer even be an extension.

I'm not sure why some timestamps are missing microseconds in the ckan.net data, but plenty seem to. This causes this exception when viewing the package rss feed:

URL: http://ckan.net/package/history/sfk_publishing?format=atom&days=7
In Module ckan.lib.helpers:227 in date_str_to_datetime
WebApp Error: <type 'exceptions.ValueError'>: time data '2008-04-13T20:40:20' does not match format '%Y-%m-%dT%H:%M:%S.%f'

Only seen with ckan 1.4.3b

At the moment this appears to be slow (kindly suggests because missing index in db). Page is /user on a ckan instance.

Would this also speed up user page where we recent changes?

CORS - ​http://www.w3.org/TR/cors/ - support.

This is what you do in Apache. Should do this in lib/base.py or similar.

    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Methods "POST, PUT, GET, OPTIONS"
    Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type"

    # Respond to all OPTIONS requests with 200 OK
    # This could be done in the webapp
    # This is need for pre-flighted requests (POSTs/PUTs)
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} OPTIONS
    RewriteRule ^(.*)$ $1 [R=200,L]

Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend.

For example:

• name and title should have a higher weight than the notes field

After the ckan migrate process, sqlalcehmy reflect does not work and causes an error when reflecting indexes. A bug report needs to be put into sqlalchemy.

Consolidate js. Remove cruft.

• Move stuff to vendor
  • Have all vendor libs local (?)
  • jquery, jquery-ui etc
• Consolidate autocomplete to use jquery-ui
  • package autocomplete (groups)
  • tag autocomplete (package)
  • user autocomplete (authz group create)
    • turns out that authz group edit had autocomplete disabled so refactoring meant enabling ...
• Move forms.css into main
• Remove tag cloud from front page (can be example going forward)
• Remove bookmarklet (since in wiki at: ​http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F)
• Move name slug generation into standard js and clean up

Possible:

1. JS unit tests for the given pages ...
2. Option to use external site to serve js and css (e.g. CDN)

Revisions aren't assigned to packages. Where are they coming from? This seems to have been occurring since the upgrade to 1.4.3b a couple of weeks ago?

Add tables described in ​http://ckan.okfnpad.org/notifications

Changes

• All templates
• [DONE] Routing: /package/ -> /dataset/ (including API)
• [DONE] Tests: self.app.get('/dataset/...') rather than package ...
  • One test to check redirect works?
• Docs

Leave

• Stay with Package in code e.g. stay with 'PackageController?'
  • also leave stuff like pkg = .... (gradual migration)
• Leave extensions

Notes

• Package -> Dataset
• Data Package -> Dataset
• respect capitalization

Sub ticket of #1294

• Initial screen just title, name, description (notes), license (and tags?).
• State that you can add data and more information on next screen. This takes you to Package 'view' where you can then do more stuff.
  • Flash some instructions? e.g. that resources are missing?
• Name slug should by default look uneditable (cf wordpress way to do the slug)

Subticket of #1294.

Can integrate from working system on ckanjs.

Migrate tables for activity streams

This was removed during the auth refactoring, but it's needed in the old controllers

Only happens if the user does not update the password. Most probably introduced during #1229.

The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one.

Could not send reset link: SMTPRecipientsRefused({u'amercadero@…': (550, 'relay not permitted')},)

As I said, adria.mercader@… works fine. The SMTP server used mail.okfn.org

On the CKAN home page there is no suggestion that you can add or register a dataset. The 'Register a dataset' option has been removed at some point! (only visible to users that have registered - why would they do that when the benefits aren't described?)

We need some documentation for the /api/action. Even though it's beta and subject to change it would be good to have a guide.

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

It is very annoying that you have to wrap links in <...> and most people don't bother. Extend h.markdown in code and markdown in js to fix this.

(We should also support 'package' links etc in javascript?)

*ADDITIONAL*

• Note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
• Move the markdown api to /util/markdown

on thedatahub.org

We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in:

• 'with' statement requires from __future__ import with_statement
• import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5.

We get this i18n error for the URL ​http://no.ckan.net/authorizationgroup (which runs 1.4.2 with Norwegian translation). I'm not sure why it occurs. Fixing it is low importance though, because this site doesn't use Authorization Group feature

Usually caused by bots. The user does not appear to be logged in.

WebApp Error: <type 'exceptions.IndexError'>: list index out of range
...
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'

Instead of this:

extras: {
type: ""Primary source""
license_id: ""localauth-withrights""
publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""
}

we should return this:

extras: {
type: "Primary source",
license_id: "localauth-withrights",
publisher_frequency: "2Preño € $ àdrià",
}

I ran into a bug with the size field on resources.

• It would not accept an empty value from form (IMO this clearly equates to null/None)
• This could be fixed via using ignore_empty instead of ignore_missing
• However using this means there was no way to empty the field (e.g. i may just want to set the size field back to null not just change to another value)
• similar issues could arise around other fields (such as last_modified ...)
  • cf cset:645031d07b60

To solve this (cset:58acdcfe6d4e) i created an int_converter temporarily in logic/schema.py (this is almost certainly the wrong place). But I think it raises a bigger issue about the conversion layer and how it works.

If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting ​http://localhost:5000/dataset).

Steps to reproduce:

• Change the solr_url parameter in the ini file to something non-existant
• Visit ​http://localhost:5000

There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (​ticket) we experienced extraordinary memory usage peaks (​ticket). Here are the observed value with Apache default settings:

• eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
• s055, mpm-prefork: base level ~1GB, peaks up to 4GB
• s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (​changeset). This (together with two other tweaks) changed the situation drastically:

• s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to processes=3), and then observing the relation between maximum-requests=25...500 and memory consumption.

On 14/09/11 17:49, David Read wrote:

Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1

Exception caused by bots POSTing 'new group' form, but with bad (old-style) parameters.

<type 'exceptions.ValueError'>: invalid literal for int() with base 10: 'cache'
...
Module ckan.controllers.group:102 in new
<<          if context['save'] and not data:
                   return self._save_new(context)
               
               data = data or {}
>>  return self._save_new(context)
Module ckan.controllers.group:152 in _save_new
<<          try:
                   data_dict = clean_dict(unflatten(
                       tuplize_dict(parse_params(request.params))))
                   context['message'] = data_dict.get('log_message', '')
                   group = create.group_create(data_dict, context)
>>  tuplize_dict(parse_params(request.params))))
Module ckan.logic:57 in tuplize_dict
<<          for num, key in enumerate(key_list):
                   if num % 2 == 1:
                       key_list[num] = int(key)
               tuplized_dict[tuple(key_list)] = value
           return tuplized_dict
>>  key_list[num] = int(key)
ValueError: invalid literal for int() with base 10: 'cache'

Various things in templates can be tidied up after work has finished on #1294:

• remove styling in page in group/index.html (put in css main)
  • ditto for package/search.html
  • ditto for home/index.html
• Forms.css - to merge into style.css; remove explicit imports elsewhere
  • @import at the top of style.css @import url(forms.css)
• Make a note: Success/failure controllers clash CSS with success/failure flash banners

The template for editing a user's profile seems to output the same form twice in one page. Furthermore, the page is made needlessly complex by the preview functionality. The neat inline preview used in package will make this page a lot neater.

• Fix the duplicate form
• Strip the preview functionality from the front-end, replacing it with js editor
• Strip the preview functionality from the back-end and test

If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).

Major UX grip and incredibly simple to do. May also want to have basic autocomplete support within the list (trivial to do as well I believe).

(Talk to RP about this re how we define lists of licenses).

Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install).

Propose:

• ckan.simple_search config option
• If set:
  • query via simple query to database backend
  • do no specialized indexing

Extras

Remove

• TagSearchQuery? from lib/search (just do a search directly)?
• #1360: filter by downloadable and filter by open search options

Possible Extras

(Probably future improvements)

• Re-introduce full-text search indexing where supported in e.g. postgres and use for querying

table: task_status

id
entity_id
entity_type
task_type
key
value
last_updated
state

unique(entity_id, task_type, key)

example keys:
last_task_id

Lots of things rely on the old (i.e. pre-SOLR) search param formats, including ckanclient / DGU. Need a translation layer to convert these params (in api v1&2) to SOLR syntax.

This functionality was broken by the changes in #1277 which went into 1.5a

From Ira: PDF of bugs and suggestions is attached.

This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in #1404.

DONE ?

• [dataset/edit]: Deletions are confirmed with a dialog box
• [dataset/edit]: Long URLs break the edit resources tab. Show name (& 'no name') with a pencil icon.
• [dataset/edit]: Rearrange the delete resources mechanism;
  • Use a little minus icon on the far left, followed by a pencil icon for editing.
  • The 'delete resource' button can become 'hide'
  • Deletions are confirmed with a dialog box
• [dataset/edit]: Rearrange the 'you have unsaved changes' warning;
  • Delete the little green plus sign on the right;
  • Try a flash banner instead (more general purpose & beautiful)
• [dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset.
• [dataset/view]: Subscribe button (& subscribe to history) doesn't work as expected. Modal dialog carrying instructions?
• [dataset/view]: Drop the 'add to a group>>' and 'add tag>>'. Needs inline tag-adder.
• [markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out.
• [dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels.
• [groups/add]: Hide unimportant fields
• [navigation] Add 3D bevel (@see github) to current tab highlight.
• [dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see ​http://groupspaces.com/a/group/
• [groups/view]: Show description and owner. To match dataset section.
• [groups/authz]: Tidy to match admin/authz (pretty buttons & spacing). Add navigation bar.
• [groups]: Entire section to use pretty buttons please.
• [about]: Get the about page working on test.ckan.org. (Works locally and on live).

WONTFIX ?

• [dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold. (wontfix)
• [groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'. (ticket #1403)
• ([groups]: (Investigate) Is it easy to implement a little search-groups feature?) (ticket #1403)
• [*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...) (wontfix: we need to rethink this mechanism; authorization refactor is coming)
• [general]: 'Authorization' refactor to 'Permissions' on nav-bars. (wontfix, further discussion. authorization might be less techy...)
• [search]: (Investigate) Can we quickly & easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets? (wontfix: high cost and maintenance overhead; need compelling user story)

Add a way to share datasets and see mentions of datasets on twitter (and elsewhere)

As a visitor I want to share a link to a dataset I have found. I also want to see how many others have shared this (or mentioned it).

• Sharing platform: simplest option is probably twitter but could generalize to e.g. sharethis system which supports, facebook, twitter etc etc.
  • Will need a config option for relevant API key e.g. ckan.sharethis.apikey
• Location: Put this at top of sidebar on dataset view (a better suggsetion?)
• Should show how many mentions / shares there are. (very important!)

Discussion

• (Tom and Ira and ...): do we really want this, versus e.g. proper follow extension?
• What about bringing conversation back in to CKAN. E.g. show all times this dataset (i.e. its url) was mentioned on twitter. If we do this I think this should be definite +1.
• Also, given our users, I think just doing twitter (identica) may be sufficient (how many people want to share links to datasets on facebook?)

Take the resource archiver (currently part of ckanext-qa) and make it into a standalone extension that works as a celery task.

e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English.

Regression found by updating datacatalogs.

Implement a banner across the top of the screen giving newbies a link to the about page, and making them feel welcome. See stackoverflow for reference?