{22} Trac tickets (2647 matches)

Results (1101 - 1200 of 2647)

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#1199 enhancement pudo pudo closed fixed Implement an email function for users.

We need an email function in CKAN to accept messages sent to users. The basic signature will be:

  • mail_user(user_obj, subject, body, mime_type='text/plain', headers={})

This has a number of use cases:

  • Retrieval of lost passwords
  • E-Mail confirmation

Finally, the mail function should be exposed in the API for sysadmin clients. This way we can have scripts traverse CKAN for 404s, invalid data or missing fields and ping users about that automatically (requires traversal by revision, not package, to get the associated users).

Implementation

Note we have already written code like this (*and* tested it) in isitopen:

1308821986000000 1310555589000000
#1205 defect kindly dread closed fixed db upgrade problem on cz.ckan.net

When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below).

cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan.

In the meantime I've reverted the db to the old version (pre-upgrade).

cz.ckan.net is not working properly until this is fixed.

okfn@eu19:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://<user>:<pass>@<host>/<dbname>
  module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects
2011-06-29 09:07:41,294 INFO  [migrate.versioning.api] 28 -> 29... 
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname
  ret = fn(self, con, *args, **kw)
Traceback (most recent call last):
  File "/usr/bin/paster", line 18, in <module>
    command.run()
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 218, in run
    result = self.command()
  File "/usr/lib/pymodules/python2.6/ckan/lib/cli.py", line 84, in command
    model.repo.upgrade_db()
  File "/usr/lib/pymodules/python2.6/ckan/model/__init__.py", line 163, in upgrade_db
    mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 185, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 364, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py", line 83, in runchange
    change.run(self.engine, step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py", line 140, in run
    script_func(engine)
  File "/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py", line 152, in upgrade
    migrate_engine.execute(q)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1788, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1191, in execute
    params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
    return self.__execute_context(context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1302, in __execute_context
    context.parameters[0], context=context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py", line 299, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "package_group_revision" violates foreign key constraint "package_group_revision_continuity_id_fkey"
DETAIL:  Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table "package_group".
 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'}
1309339700000000 1310568631000000
#1206 defect wwaites dread closed fixed "Content-Type json" header scuppers package POST

Compare these two requests to create a package:

curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'Content-Type: application/json' -H 'X-CKAN-API-KEY: tester'
curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'X-CKAN-API-KEY: tester'

The second one gets the payload through (ckan log):

Retrieving request params: UnicodeMultiDict([('{name:"test"}', u'')])

But the first one causes a ServerError? because the payload (name:"test") doesn't make it to request.POST or request.params:

Retrieving request params: UnicodeMultiDict([])

The only difference is the "ContentType?: application/json" header, which seems a reasonable thing to include. Javascript lib backbone.js (for example) inserts this automatically.

So why does this header cause the payload to not get through to the request object?

1309344348000000 1309450216000000
#1207 enhancement dread dread closed fixed ckanclient.package_entity_get should raise more specific exception

When package does not exist in ckan catalogue, ckanclient.package_entity_get should raise more specific exception, such as CkanNotFoundError? instead of generic CkanApiError?.

1309515582000000 1311325343000000
#1208 enhancement rgrp rgrp ckan-sprint-2011-10-28 closed fixed Webstore: a datastore with web API
  • SQLite based
  • Use Cyclone for async http server on top with auth
  • Designed to not be specific to CKAN or anything else

Current code is here: https://github.com/okfn/webstore.

API Spec

Read

Two basic ways to query::

    GET: /{owner}/{db-name}/?sql=...
    GET: /{owner}/{db-name}/?table=...&attr=value&attr=value&limit=...

Returns::

  {
      u'keys': [u'id', u'name'],
      u'data': [
          [1, u'jones'],
          [u'aaa', u'jones']
          ]
  }

Write

POST to::

/{owner/{database}/{table}

Payload is json data structured as follows::

{

unique_keys: [list of key attributes] data: {dict of values}

}

Authentication and Authorization

Authentication: use basic auth header.

Authorization:

  • Default: all read, owner can write
  • Restricted: owner can read and write, everyone can do nothing

Possible future: config file can specify a python method (TODO: method signature)

Integration with Other Systems

TODO: Specify how to delegate authenatication to user database in some other system.

1309804460000000 1312191646000000
#1209 defect kindly closed fixed sort out rendering of formalchemy package preview

Got broken due to the moderated edits changes #1141.

Need to make sure preview works.

1309817240000000 1310571715000000
#1210 defect dread dread ckan-v1.5-sprint-4 closed fixed POST application/json error handling with newer WebOb

WebOb? from v1.0.7 has some interesting new behaviour with reading request data for different Content-Types:

  • It seems that looking at request.body mangles request.POST. See fix in ckan cset:e83bad9caa1b (problem with WebOb? v1.0.7 but may go back to v1.0.4?)
  • A particular request caused an AssertionError? during looking at request.body, so we need to enclose this in a try/except. I couldn't recreate it in a test unfortunately (problem with WebOb? v1.0.7 but may go back to v1.0.2?)

Example:

Module ckan.controllers.api:206 in create
<<          log.debug('create: %s' % (context))
               try:
                   request_data = self._get_request_data()
               except ValueError, inst:
                   response.status_int = 400
>>  request_data = self._get_request_data()
Module ckan.lib.base:149 in _get_request_data
<<          cls.log.debug('Retrieving request params: %r' % request.params)
               cls.log.debug('Retrieving request POST: %r' % request.POST)
               cls.log.debug('Retrieving request POST body: %r' % request.body)
               if request.POST:
                   try:
>>  cls.log.debug('Retrieving request POST body: %r' % request.body)
Module paste.registry:137 in __getattr__
<<      
           def __getattr__(self, attr):
               return getattr(self._current_obj(), attr)
           
           def __setattr__(self, attr, value):
>>  return getattr(self._current_obj(), attr)
Module webob.request:470 in _body__get
<<          Return the content of the request body.
               """
               self.make_body_seekable() # we need this to have content_length
               r = self.body_file.read(self.content_length)
               self.body_file.seek(0)
>>  self.make_body_seekable() # we need this to have content_length
Module webob.request:697 in make_body_seekable
<<              self.body_file_raw.seek(0)
               else:
                   self.copy_body()
>>  self.copy_body()
Module webob.request:714 in copy_body
<<                  self.body = self.body_file_raw.read(length)
               elif self.is_body_readable:
                   self.body = self.body_file_raw.read()
                   self._copy_body_tempfile()
               else:
>>  self.body = self.body_file_raw.read()
Module webob.request:1190 in read
<<      def read(self, size=-1):
               body = self._get_body()
               if size < 0:
                   v = body[self.position:]
>>  body = self._get_body()
Module webob.request:1207 in _get_body
<<                  self._body = _encode_multipart(self.vars, self.content_type)
                   else:
                       assert 0, ('Bad content type: %r' % self.content_type)
               return self._body
>>  assert 0, ('Bad content type: %r' % self.content_type)
AssertionError: Bad content type: '; charset=utf-8'
1309949451000000 1309974781000000
#1211 enhancement kindly kindly ckan-v1.5 closed fixed [super] Drupal integration

Sub-tasks / tickets:

  • #1335 Action API - access to any logic layer function
  • Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
  • Add package_create_validate & package_update_validate logic functions
  • Allow extensions to provide logic layer functions (IActions)
1309958442000000 1315948703000000
#1215 defect dread closed fixed Edit Resources table - removing all rows prevents adding them again

I'm not sure if this was reported earlier, but I have found a minor bug in the "Add new package" form. As far as I can see this bug exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). This is a problem with adding new resources in "Resources" section. To reproduce the bug do the following:

  1. Go to "Add new package", for instance http://ckan.net/package/new
  2. Scroll down to "Resources" section.
  3. Click on "Remove this row" link (red circle).
  4. Say "Yes" to "Are you sure you wish to remove this row?"
  5. Now there should be no rows. Only the header "URL* Format Description"
  6. Click on "Add row to table"
  7. All you get now is another row with table header "URL* Format Description". You will not be able to add any resources now. Solution is to reload the page.

The simplest solution to this bug will be to hide the "Remove this row" link if there is only one row.

1310133224000000 1310385485000000
#1218 enhancement dread minspamboks@… ckan-sprint-2011-10-28 closed fixed Colour the History tab icon

Change the color of the "History" tab icon to yellowish, like the rest of the icons in the other tabs ("View" and "Edit", "Authorization").

Reasoning

When you view a data package, for instance http://ckan.net/package/thesaurus-w, you will see "View", "Edit", "History" tabs on the top. "History" tab has a black-and-white icon which makes it look like an inactive/disabled tab (since the text is also grayed out when the tab is not selected). This is not a major issue, but it is a little bit confusing for the users. This icon exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net).

The simple solution would be to change the color of the "History" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. "View" and "Edit").

1310375768000000 1310389390000000
#1219 defect timmcnamara closed fixed CSS issues on IE7

As reported on ckan-dev:

items in the footer of CKAN ("Packages", "Groups & Tags", "About", "Language", etc.) are shown vertically instead of horizontally in IE7. This works fine in later browsers like IE8, IE9, FF4, and latest Opera and Chrome.

This seems to exist in all recent CKAN versions up to 1.4.1a.

1310423688000000 1310740534000000
#1221 defect dread dread ckan-sprint-2011-10-28 closed fixed Password reset tidy up
  • No tests.
  • 500 error given when user is not found.
1310475661000000 1310556607000000
#1222 defect dread dread ckan-sprint-2011-10-28 closed fixed Mailer has no tests 1310479136000000 1310556618000000
#1223 enhancement pudo pudo closed fixed Caching of static files

StaticURLParser can have caching - use it

1310573854000000 1310573893000000
#1225 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Tool for mass tag changes

Using ckanapi

1310642161000000 1310642407000000
#1228 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Display OpenID login errors

When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again.

1311086562000000 1311325161000000
#1229 enhancement dread thejimmyg ckan-sprint-2011-10-24 closed fixed Refactor all database code out of the controllers and into the logic layer

Progress can be tracked in this document:

https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdFQzRnVoUHV6ckoyblI5TUdEcnJLVlE&hl=en_US#gid=0

1311093924000000 1319645778000000
#1230 enhancement kindly ckan-sprint-2011-10-28 closed fixed Make sure plugins tables work in tests.

Dgu tests failed as harvesting table are not created at the correct point.

1311100811000000 1311154142000000
#1234 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Setting locale error conditions handled badly

Needs tests for error conditions for bad locale and redirect parameters.

1311179871000000 1311183328000000
#1236 enhancement dread dread ckan-sprint-2011-11-07 closed fixed View a package at a given revision

As a user I want to view a package at a given revision:

  • When I visit /package/osm@yyyy-mm-dd or /package/osm@rrrrrrr I should be shown package at date yyyy-mm-dd or revision rrrrrrr.
  • viewing this page, it should say it is the old revision, just like wikipedia
  • package history page should provide links to these pages
1311269333000000 1330019788000000
#1239 defect rgrp rgrp ckan-sprint-2011-10-28 closed fixed Remove and re-add tag on package not working

See reports on list at:

Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix.

1311441020000000 1312453156000000
#1242 enhancement timmcnamara closed fixed thedatahub.org's login page

The right hand side of thedatahub.org login screen is a little confusing:

    Not a member?

    Join CKAN to contribute packages under your own name.

     - Register
     - Reset your password
     - Privacy Policy

Suggested improvements to make:

  • replace CKAN with thedatahub.org
  • in general, edit the copy to be more inviting
  • remove "Reset your password" from that bubble
  • move the privacy policy link to somewhere less prominent
1311569218000000 1323173292000000
#1245 task rgrp ckan-v1.5 closed fixed Reorganise ckan.org pages

Here is Rufus' plan for ckan.org reorganisation from http://ckan.okfnpad.org/documentation (22/7/11) (ticket made by dread):

/solutions
  /overview <--- current 'about' page (the software)
  /customers <-- say "in addition to our clients..."
  /features <-- needs most work (pictures etc)
/pricing/  <-- simplest thing possible ... talk with Ira (cover pricing and support)
/developers <--- links to http://docs.ckan.org/, getting the software, the wiki, the mailing list /contact
/blog

Tasks:

  • add tagline in header - "the data portal software"
  • make and add glossy PDF
  • change "data hub" to "data portal" passim (if only using one term)
  • add info@…
  • image banner on home page - redo screenshots without menus
1311861708000000 1317422640000000
#1246 defect pudo adrian.pohl@… ckan-sprint-2011-10-28 closed fixed Search results on ckan.net are mistakenly all 'open'

All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: http://ckan.net/package

1311863353000000 1311892816000000
#1247 task rgrp dread ckan-sprint-2011-10-28 closed fixed Put docs on ReadTheDocs

Put docs on ReadTheDocs? instead of eu13.

1311941948000000 1311955185000000
#1248 defect dread dread ckan-sprint-2011-11-07 closed fixed License exception

Exception received on test.ckan.net at 10.12 on 29/7/11

http://test.ckan.net/package?__cache=2759373&page=4
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:153 in search
<<                          }
       
                   query = get.package_search(context,data_dict)
       
                   c.page = h.Page(
>>  query = get.package_search(context,data_dict)
Module ckan.logic.action.get:514 in package_search
<<      for package in query.results:
               result_dict = table_dictize(package, context)
               result_dict = _extend_package_dict(result_dict,context)
       
               results.append(result_dict)
>>  result_dict = _extend_package_dict(result_dict,context)
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: None'

and a similar one:

 http://test.ckan.net/tag/barbecue
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.tag:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__("logging").getLogger("proxy_cache")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
       
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.tag:64 in read
<<          data_dict = {'id':id}
               try:
                   c.tag = get.tag_show(context,data_dict)
               except NotFound:
                   abort(404, _('Tag not found'))
>>  c.tag = get.tag_show(context,data_dict)
Module ckan.logic.action.get:341 in tag_show
<<      extended_packages = []
           for package in tag_dict['packages']:
               extended_packages.append(_extend_package_dict(package,context))
       
           tag_dict['packages'] = extended_packages
>>  extended_packages.append(_extend_package_dict(package,context))
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: CreativeCommonsAttributionCCBY25'
1311954136000000 1319797763000000
#1249 enhancement pudo rgrp ckan-v1.5 closed fixed Exclude script tag from extraction for i18n

Currently have this script section put in for i18n. It shouldn't be.

<script type="text/javascript">
//<![CDATA[
(function($){
    $.fn.ajaxCreateSlug = function(name, url) {
        var title = this;
        var updater = {
            init: function(title, name) {
                // Add a new element where the validity of the package name can be displayed
                this.name_field = name;
                this.title_field = title;
                this.name_field.parent().append('<div id="package_name_valid_msg"></div>');
                this.title_field.blur(this.title_change_handler())
                this.title_field.keyup(this.title_change_handler())
                this.name_field.keyup(this.name_change_handler());
                this.name_field.blur(this.name_blur_handler());
                this.url = url;
            },
....
1311958536000000 1311959356000000
#1250 enhancement pudo rgrp ckan-v1.5 closed fixed Search results should be sorted by score rather than alphabetical

At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases.

Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard).

Options:

  • Default this in solr (no need to touch code) but fragile and affects everything ...
  • Do it in code and default to score
  • Do it in code and have alphabetical (on name or title?) when no criteria otherwise score

Aside: may also wish to support search in query api but that is for later!

1311964752000000 1312370201000000
#1251 enhancement rgrp rgrp ckan-backlog closed fixed Rename wiki.ckan.net to wiki.ckan.org

Inline with general naming policy going forward.

1311965147000000 1317315159000000
#1252 enhancement rgrp rgrp ckan-v1.5 closed fixed Deprecate ckan.net in favour of thedatahub.org

Deprecate ckan.net in favour of thedatahub.org

  • Announce
  • Create thedatahub.org and have it run same data as ckan.net
  • Announce closing down of ckan.net with 302
  • Implement 302 permanent

Also redirect wiki.ckan.net to wiki.ckan.org

1311965339000000 1318160334000000
#1253 enhancement amercader amercader ckan-sprint-2011-10-28 closed fixed Authz refactor

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

1312199491000000 1313401367000000
#1254 task dread dread ckan-sprint-2011-10-28 closed fixed Feature sheet

Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers.

1312211925000000 1313401629000000
#1256 defect kindly kindly ckan-sprint-2011-10-28 closed fixed fix modification extension for moderation

Solr is not working for moderation, pending changes are not being indexed.

1312280026000000 1319812556000000
#1258 enhancement kindly kindly ckan-sprint-2011-10-28 closed fixed Bring purge revision into ckan repo from vdm

In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory.

1312289539000000 1319812452000000
#1263 enhancement dread dread closed fixed Bad name for logic action in API causes Exception

A bad action name: e.g. curl http://test.ckan.net/api/action/user_showaa -d '{}' causes 500 response.

Occurs only on 1.4.3 beta.

1312479020000000 1314029279000000
#1264 enhancement zephod rgrp ckan-sprint-2011-10-10 closed fixed Merge ckanext-admin into core

This extension should be in core now -- functionality is basic and very useful.

Copy across from here: http://bitbucket.org/okfn/ckanext-admin

Suggest this no longer even be an extension.

1312480783000000 1318245716000000
#1265 defect dread dread ckan-sprint-2011-10-28 closed fixed user/edit error conditions cause exceptions

We get exceptions on these occasions:

  • user/edit if not logged in
  • user/edit/bob if bob is not a known user

These are a problem only on 1.4.3b.

Introduced in #1229 - merge of branch feature-1229-db-out-of-controllers

1312565788000000 1312566052000000
#1266 defect dread dread ckan-sprint-2011-10-28 closed fixed Timestamps without microseconds causing exception

I'm not sure why some timestamps are missing microseconds in the ckan.net data, but plenty seem to. This causes this exception when viewing the package rss feed:

URL: http://ckan.net/package/history/sfk_publishing?format=atom&days=7
In Module ckan.lib.helpers:227 in date_str_to_datetime
WebApp Error: <type 'exceptions.ValueError'>: time data '2008-04-13T20:40:20' does not match format '%Y-%m-%dT%H:%M:%S.%f'

Only seen with ckan 1.4.3b

1312794831000000 1312797477000000
#1267 defect dread dread closed fixed Random exception: cannot set the body to a unicode value

Occasionally see this problem when producing an error page. Here's an example

WebApp Error: <type 'exceptions.TypeError'>: You cannot set the body to a unicode value without a charset
URL: http://at.ckan.net/authorizationgroup/new
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:117 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:284 in __call__
<<              if log_debug:
                       log.debug("Calling Response object to return WSGI data")
                   return response(environ, self.start_response)
               
               if log_debug:
>>  return response(environ, self.start_response)
Module webob.exc:248 in __call__
<<              return []
               if not self.body and not self.empty_body:
                   return self.generate_response(environ, start_response)
               return Response.__call__(self, environ, start_response)
>>  return self.generate_response(environ, start_response)
Module webob.exc:239 in generate_response
<<              status=self.status,
                   headerlist=headerlist,
                   content_type=content_type
               )
               return resp(environ, start_response)
>>  content_type=content_type
Module webob.response:94 in __init__
<<                  if charset is None:
                           raise TypeError(
                               "You cannot set the body to a unicode value without a charset")
                       body = body.encode(charset)
                   self._body = body
>>  "You cannot set the body to a unicode value without a charset")
TypeError: You cannot set the body to a unicode value without a charset
1312889084000000 1312897724000000
#1268 enhancement kindly rgrp ckan-sprint-2011-10-10 closed fixed Speed up listing users by revisions by adding db index

At the moment this appears to be slow (kindly suggests because missing index in db). Page is /user on a ckan instance.

Would this also speed up user page where we recent changes?

1312906592000000 1318279592000000
#1270 defect dread dread ckan-sprint-2011-10-28 closed fixed Default site www.ckan.net is a redirect and loses POSTs

The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters:

In [1]: import ckanclient

In [2]: c = ckanclient.CkanClient()

In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api')

In [4]: c.package_search('', search_options={'groups':'openspending'})
Out[4]: 
{u'count': 2102,
 u'results': <generator object _result_generator at 0x10168b1e0>}

In [5]: c2.package_search('', search_options={'groups':'openspending'})
Out[5]: {u'count': 29, u'results': <generator object _result_generator at 0x10168b410>}

Thanks to borior for finding this and raising it.

1312911847000000 1312912188000000
#1271 enhancement rgrp rgrp ckan-sprint-2011-10-28 closed fixed CORS support

CORS - http://www.w3.org/TR/cors/ - support.

This is what you do in Apache. Should do this in lib/base.py or similar.

    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Methods "POST, PUT, GET, OPTIONS"
    Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type"

    # Respond to all OPTIONS requests with 200 OK
    # This could be done in the webapp
    # This is need for pre-flighted requests (POSTs/PUTs)
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} OPTIONS
    RewriteRule ^(.*)$ $1 [R=200,L]
1313241839000000 1313753663000000
#1272 enhancement amercader ckan-sprint-2011-10-10 closed fixed Store spatial extents provided in the package form

Consolidate the extra spatial as the field for providing information about the geographic extent of the package. This will integrate automatically with the spatial search and give developers a single point to develop geospatial features.

1313411787000000 1317381996000000
#1274 enhancement johnglover johnglover ckan-sprint-2011-10-28 closed fixed Testing solr search

Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend.

1313413202000000 1313429049000000
#1275 enhancement johnglover johnglover ckan-sprint-2011-09-26 closed fixed Move solr search extension to ckan core 1313413597000000 1319812967000000
#1276 enhancement johnglover johnglover ckan-sprint-2011-09-12 closed fixed Apply weighting to search fields in solr

For example:

  • name and title should have a higher weight than the notes field
1313414353000000 1315948394000000
#1277 enhancement johnglover johnglover ckan-sprint-2011-09-12 closed fixed Use solr query parser for search instead of ckan query parser

See http://ckan.okfnpad.org/search for details/discussion

1313414446000000 1315948417000000
#1280 defect kindly ckan-backlog closed fixed fix sqlalchemy so that it works with postgres 9.0

After the ckan migrate process, sqlalcehmy reflect does not work and causes an error when reflecting indexes. A bug report needs to be put into sqlalchemy.

1313513090000000 1328786670000000
#1281 defect pudo pudo ckan-sprint-2011-10-28 closed fixed Locale change flash message in wrong language

When you change the locale, it displays a flash message "Language has been set to: English" in the original language, not the one we have just changed to.

1314021656000000 1314021919000000
#1282 enhancement rgrp rgrp ckan-v1.5 closed fixed JS and WUI tidy up

Consolidate js. Remove cruft.

  • Move stuff to vendor
    • Have all vendor libs local (?)
    • jquery, jquery-ui etc
  • Consolidate autocomplete to use jquery-ui
    • package autocomplete (groups)
    • tag autocomplete (package)
    • user autocomplete (authz group create)
      • turns out that authz group edit had autocomplete disabled so refactoring meant enabling ...
  • Move forms.css into main
  • Remove tag cloud from front page (can be example going forward)
  • Remove bookmarklet (since in wiki at: http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F)
  • Move name slug generation into standard js and clean up

Possible:

  1. JS unit tests for the given pages ...
  2. Option to use external site to serve js and css (e.g. CDN)

1314090389000000 1314404540000000
#1283 enhancement dread dread ckan-backlog closed fixed Deleted packages shouldn't be searchable or browsable

A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else.

(But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.)

Same in the Web interface and API.

This ticket ties in with #948 highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can #1076.

1314112481000000 1330083933000000
#1284 defect dread rgrp closed fixed Lots of revisions being created on datahub.org

Revisions aren't assigned to packages. Where are they coming from? This seems to have been occurring since the upgrade to 1.4.3b a couple of weeks ago?

1314113015000000 1315215626000000
#1290 enhancement dread dread ckan-backlog closed fixed Better error when blank database

When installing CKAN, when doing "paster serve development.ini", lots of users encounter the error for every request:

ProgrammingError: (ProgrammingError) relation "user" does not exist

This is because the database tables have not been created - they have forgotten or missed the "paster db init" step.

Can we provide a better error to say that the database is not initialised yet?

Implementation options

  1. At the start of every request we reflect the database tables and check they are there. This is rather expensive!
  1. Uncached requests to the home page start with a cheap database query. If there is an exception then return this error about database setup. I really like Drupal's page for this that has in large letters that the site is currently off-line. Below a line, in small letters, there are developer suggestions on what is wrong and where to look to fix it.
1314264255000000 1314270894000000
#1291 enhancement kindly kindly ckan-sprint-2011-09-12 closed fixed Add activity and activity detail tables.

Add tables described in http://ckan.okfnpad.org/notifications

1314267572000000 1315950394000000
#1292 enhancement dread dread ckan-sprint-2011-11-07 closed fixed Internationalization improvements
  • Review process of releases and updating strings (pot) and translations (po) files
  • Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?)
  • Review long list of strings for improvement
    • Consolidate any?
    • Get rid of i18n for strings in api
    • Try and remove javascript
    • Add in extensions
  • #1374 Fix switching to en from other default language
  • #1417 Fix browser language detection
1314303314000000 1320145732000000
#1293 enhancement zephod rgrp ckan-sprint-2011-09-12 closed fixed Rename Package to Dataset throughout WUI, API

Changes

  • All templates
  • [DONE] Routing: /package/ -> /dataset/ (including API)
  • [DONE] Tests: self.app.get('/dataset/...') rather than package ...
    • One test to check redirect works?
  • Docs

Leave

  • Stay with Package in code e.g. stay with 'PackageController?'
    • also leave stuff like pkg = .... (gradual migration)
  • Leave extensions

Notes

  • Package -> Dataset
  • Data Package -> Dataset
  • respect capitalization
1314405700000000 1317649968000000
#1294 enhancement rgrp rgrp ckan-v1.5 closed fixed [super] Package creation and editing UX improvements

Largely an integration of work from ckanjs

  • #1295 Simplify package create form
  • Improvements to edit form
    • Split edit form into sections
    • Remove preview from edit form (introduce inline preview on description/notes field)
  • #1296 Improved resource adding/editing on dataset/page page
  • #878 File upload integrated into resource creation
  • #1297 In-place editing of notes field for existing packages
1314405811000000 1315948257000000
#1295 enhancement rgrp rgrp ckan-sprint-2011-09-12 closed fixed Simplify package create form

Sub ticket of #1294

  • Initial screen just title, name, description (notes), license (and tags?).
  • State that you can add data and more information on next screen. This takes you to Package 'view' where you can then do more stuff.
    • Flash some instructions? e.g. that resources are missing?
  • Name slug should by default look uneditable (cf wordpress way to do the slug)
1314406014000000 1315948009000000
#1296 enhancement zephod rgrp ckan-sprint-2011-09-12 closed fixed Improved resource adding/editing on dataset/page page

Subticket of #1294.

  • Nice way to edit all resource attributes
  • Nice way to add new resources
    • Including file upload (cf #878).
      • Cf work in ckanjs (this can be a straight integration).
1314406093000000 1315948224000000
#1297 enhancement zephod rgrp ckan-sprint-2011-09-12 closed fixed In-place preview of edited description markdown on edit dataset form

Subticket of #1294.

Can integrate from working system on ckanjs.

1314406183000000 1317649974000000
#1298 enhancement kindly kindly ckan-sprint-2012-01-09 closed fixed Generate activites to be put into activities table.

This should be done from the logic layer or automatically from a session extension.

1314696442000000 1325591582000000
#1299 enhancement seanh kindly ckan-sprint-2012-01-23 closed fixed Activity streams table migration

Migrate tables for activity streams

1314696635000000 1327322739000000
#1300 enhancement rgrp rgrp ckan-sprint-2011-11-07 closed fixed Core changes to base theme

Core changes to theme in order to make it easier to re-theme:

  • page_heading
  • side bar menu must be switchable (left to right) (through config or css)
  • Add optional_footer to complement optional_head
    • And adopt rule for template writers that all extra js must go in optional footer
    • That way we could move optional js to the bottom of the page to improve page load times.
  • Re-add support for body-class ... (seems to be removed by #1108)
  • more divs.
  • better labelling for css (and javascript)
1314862223000000 1319794520000000
#1301 defect amercader amercader ckan-sprint-2011-09-12 closed fixed Give access to extensions from the old formalchemy controllers

This was removed during the auth refactoring, but it's needed in the old controllers

1314882888000000 1314893744000000
#1302 refactor johnglover johnglover ckan-sprint-2011-09-26 closed fixed Move Resource and Tag search code to logic layer

Resource and Tag search is currently still using the old ckan query parser which is unnecessary. Refactor this out and move to the logic layer.

1314886102000000 1317131948000000
#1303 defect amercader amercader ckan-sprint-2011-09-12 closed fixed User can not login again if he has edited his profile

Only happens if the user does not update the password. Most probably introduced during #1229.

1314889472000000 1314978842000000
#1304 enhancement amercader amercader ckan-sprint-2011-09-12 closed fixed Show more descriptive error messages when sending emails

Sending emails is failing in some cases, and the error message just says 'Could not send reset link'.

1314893206000000 1314893749000000
#1305 defect nils.toedtmann amercader ckan-backlog closed fixed SMTP config for thedatahub.org and IATI

The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one.

Could not send reset link: SMTPRecipientsRefused({u'amercadero@…': (550, 'relay not permitted')},)

As I said, adria.mercader@… works fine. The SMTP server used mail.okfn.org

1314956657000000 1315317033000000
#1307 enhancement zephod rgrp ckan-sprint-2011-09-12 closed fixed Implement captcha for signup
  • Major issue with spam signup on thedatahub.org
  • Suggest using recaptcha

Est cost: 1d

1315153860000000 1317649963000000
#1308 defect shevski dread closed fixed 'create package' user case failure

On the CKAN home page there is no suggestion that you can add or register a dataset. The 'Register a dataset' option has been removed at some point! (only visible to users that have registered - why would they do that when the benefits aren't described?)

1315219500000000 1323174312000000
#1310 defect dread dread ckan-sprint-2011-09-12 closed fixed API GET of a revision that has been approved causes exception

e.g. this particular revision on ckan.net: http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c (most others have not been approved) Causes 500 error.

1315234899000000 1315820419000000
#1315 enhancement dread dread ckan-sprint-2011-11-07 closed fixed Document /api/action

We need some documentation for the /api/action. Even though it's beta and subject to change it would be good to have a guide.

1315396722000000 1320173768000000
#1316 defect kindly dread closed fixed /api/action uncaught exception if missing necessary parameter
curl http://ckan.net/api/action/package_show -d '{"name": "uk-quango-data"}'

This gives 500 error due to KeyError? (it looked for the 'id' param), rather than 400.

1315397224000000 1338193724000000
#1318 defect dread dread ckan-sprint-2011-10-28 closed fixed 'Register user' doesn't log you in

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

1315415783000000 1315569362000000
#1319 enhancement rgrp dread ckan-sprint-2011-10-24 closed fixed Require an email address on User registration

Superticket: #1343

This avoids difficulties when they forget their password.

  • (?) Email validation (navl does not have one!)
  • Require email confirmation to be activated (?) (Answer: no, not at present -- see discussion below)
1315415932000000 1324036947000000
#1320 enhancement zephod rgrp ckan-sprint-2011-10-10 closed fixed Extend basic markdown to support normal http / https links

It is very annoying that you have to wrap links in <...> and most people don't bother. Extend h.markdown in code and markdown in js to fix this.

(We should also support 'package' links etc in javascript?)

*ADDITIONAL*

  • Note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
  • Move the markdown api to /util/markdown
1315422000000000 1318164603000000
#1321 defect dread dread ckan-sprint-2011-10-24 closed fixed Flash message deferred

When you reset a password, the flash message saying "check your email" sometimes doesn't display on the next page. You then click on the link in your email and it displays on that page (i.e. too late).

Seen this on thedatahub.org. To do with the proxy caching?

1315474657000000 1319814460000000
#1323 enhancement dread dread ckan-sprint-2011-10-28 closed fixed GET /authorizationgroup causes Exception when logged in

on thedatahub.org

1315475685000000 1315594333000000
#1324 enhancement kindly kindly ckan-sprint-2011-09-12 closed fixed Add extra fields to resource table

add extra fields to resource as outlined in

http://wiki.ckan.net/Domain_Model/Resource

Specifically (all unicode strings except for those noted):

  • name: a name for this resource (could be used in a ckan url)
  • type: the type of the resource. One of: file | api | service | listing
  • mimetype: standard mimetype (e.g. for zipped csv would be application/zip)
  • mimetype_inner: mimetype of innermost object (so for example would be text/csv)
  • size: [integer] size of the resource (content length) in bytes. Usually only relevant for resources of type file.
  • last_modified: [datetime] the date when this resource's data was last modified (NB: not the date when the metadata was modified).
  • cache_url: url for cache of object in ckan storage
  • cache_last_updated: [datetime] timestamp when cached version was created
  • webstore_url:
  • webstore_last_updated: [datetime] timestamp when webstore was last updated
1315578976000000 1315948336000000
#1325 defect dread dread ckan-sprint-2011-09-12 closed fixed python 2.5 incompatibilities

We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in:

  • 'with' statement requires from __future__ import with_statement
  • import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5.
1315580452000000 1315587113000000
#1329 defect dread amercader ckan-sprint-2011-10-28 closed fixed i18n poor in some templates - missing i18n tags

Just a tip for developers regarding translations:

  • When writing templates, mark <p>, <div>, etc elements with the

i18:msg attribute, otherwise the messages don't get extracted properly.

Good: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40

<p i18n:msg="">More about CKAN, and the background to its creation can
   be found in <a
     href="http://m.okfn.org/files/talks/opentech_20080705/">these
     slides</a>.</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

<p class="hints">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href="http://opendatacommons.org/licenses/odbl/1.0/">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the grammar or syntax of the translated language.

1315824288000000 1317404566000000
#1332 defect dread closed fixed i18n IndexError exceptions

We get this i18n error for the URL http://no.ckan.net/authorizationgroup (which runs 1.4.2 with Norwegian translation). I'm not sure why it occurs. Fixing it is low importance though, because this site doesn't use Authorization Group feature

Usually caused by bots. The user does not appear to be logged in.

WebApp Error: <type 'exceptions.IndexError'>: list index out of range
...
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range
1315906429000000 1315999104000000
#1333 enhancement amercader amercader ckan-sprint-2011-10-28 closed fixed Move form converters to ckan core

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

1315909268000000 1316951285000000
#1334 defect amercader amercader ckan-sprint-2011-10-28 closed fixed "AttributeError: 'NoneType' object has no attribute 'name'" exception when logging in

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'
1315909508000000 1319798961000000
#1335 enhancement kindly dread closed fixed Action API - basic

Provide an API to the logic layer at /api/action.

1315910021000000 1315910117000000
#1337 defect amercader amercader ckan-sprint-2011-10-28 closed fixed Group extras are JSON endoded in the REST API responses

Instead of this:

extras: {
type: ""Primary source""
license_id: ""localauth-withrights""
publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""
}

we should return this:

extras: {
type: "Primary source",
license_id: "localauth-withrights",
publisher_frequency: "2Preño € $ àdrià",
}
1315924727000000 1316951259000000
#1338 defect amercader amercader ckan-sprint-2011-10-28 closed fixed Exception when trying to use a custom form schema from an extension

Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'
1315925217000000 1317117738000000
#1339 enhancement kindly rgrp ckan-backlog closed fixed Issues / question re navl and data conversion

I ran into a bug with the size field on resources.

  • It would not accept an empty value from form (IMO this clearly equates to null/None)
  • This could be fixed via using ignore_empty instead of ignore_missing
  • However using this means there was no way to empty the field (e.g. i may just want to set the size field back to null not just change to another value)
  • similar issues could arise around other fields (such as last_modified ...)
    • cf cset:645031d07b60

To solve this (cset:58acdcfe6d4e) i created an int_converter temporarily in logic/schema.py (this is almost certainly the wrong place). But I think it raises a bigger issue about the conversion layer and how it works.

1315947022000000 1316014954000000
#1340 defect dread dread ckan-sprint-2011-10-28 closed fixed Editing non-existent group causes Exception

e.g. http://ckan.net/group/edit/history

Module ckan.controllers.group:135 in edit
<<          try:
                   old_data = get_action('group_show')(context, data_dict)
                   c.grouptitle = old_data.get('title')
                   c.groupname = old_data.get('name')
>>  old_data = get_action('group_show')(context, data_dict)
Module ckan.logic.action.get:347 in group_show
<<      if group is None:
               raise NotFound
       
           check_access('group_show',context, data_dict)
>>  raise NotFound
1315991374000000 1315991534000000
#1342 defect johnglover amercader ckan-sprint-2011-10-10 closed fixed Home controller does not capture SearchError exceptions

If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting http://localhost:5000/dataset).

Steps to reproduce:

1316001227000000 1318256532000000
#1344 enhancement kindly kindly ckan-sprint-2011-10-10 closed fixed datetime error json conversion on search

Json decoding error on search, due to date in resources.

1316020195000000 1317211299000000
#1345 enhancement kindly nils.toedtmann closed fixed Investigate possible memory leak

There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (ticket) we experienced extraordinary memory usage peaks (ticket). Here are the observed value with Apache default settings:

  • eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
  • s055, mpm-prefork: base level ~1GB, peaks up to 4GB
  • s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (changeset). This (together with two other tweaks) changed the situation drastically:

  • s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to processes=3), and then observing the relation between maximum-requests=25...500 and memory consumption.

On 14/09/11 17:49, David Read wrote:

Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1

1316041450000000 1343215489000000
#1346 defect dread dread ckan-sprint-2011-10-28 closed fixed 'get_authorization_group_dict not found' exception

This stems from a feature to allow you to prefill fields in the authorizationgroup creation form.

e.g. http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it.

1316081974000000 1316094632000000
#1347 defect dread dread ckan-sprint-2011-10-28 closed fixed Exception caused by New Group form POST with bad formed parameter

Exception caused by bots POSTing 'new group' form, but with bad (old-style) parameters.

<type 'exceptions.ValueError'>: invalid literal for int() with base 10: 'cache'
...
Module ckan.controllers.group:102 in new
<<          if context['save'] and not data:
                   return self._save_new(context)
               
               data = data or {}
>>  return self._save_new(context)
Module ckan.controllers.group:152 in _save_new
<<          try:
                   data_dict = clean_dict(unflatten(
                       tuplize_dict(parse_params(request.params))))
                   context['message'] = data_dict.get('log_message', '')
                   group = create.group_create(data_dict, context)
>>  tuplize_dict(parse_params(request.params))))
Module ckan.logic:57 in tuplize_dict
<<          for num, key in enumerate(key_list):
                   if num % 2 == 1:
                       key_list[num] = int(key)
               tuplized_dict[tuple(key_list)] = value
           return tuplized_dict
>>  key_list[num] = int(key)
ValueError: invalid literal for int() with base 10: 'cache'
1316094466000000 1316094608000000
#1348 enhancement zephod zephod ckan-sprint-2011-09-26 closed fixed UX polishing (dataset create/view/edit experience)

Tweaking UIs and polishing the work done in #1294.

## Edit Page ##

  • [Done; cset:49e0b059b6f5] Hide upload if storage not enabled
  • [Done; cset:81ff13b42dad] Make 'Add resource' links on edit page clearer and more button-like.
  • [Done; cset:03904fc50cbf] Move edit form tabs to the left
  • [Done; cset:fde26b87d6ce] API placeholder text should not link to a CSV file
  • [Done] We used to autocomplete the format field - restore this functionality
    • Works on live. My local db has an empty 'resource_revision' table; maybe 'paster create-test-data' should insert something here?
  • [Done; cset:1aab499d73db] Summary should contain 'name' (or, if unavailable, use description. Long term: description is to be deprecated)
  • [Done] Remove/move instructions where possible (a lot could just be placeholder text on the elements) (other stuff could only show on hover)
    • NB: reveal_instructions.js already disabled on edit page
    • [RP] have now done this on create -- think we can leave on edit
  • [Done; cset:755b1a622199] We have broken 'error' reporting for resource issues. Either need to move that into error summary or somehow not destroy it when we rewrite table on page load ...
  • [Done; cset:7ad090cb7855] Add slideDown to Resource expanded view
  • [Done; cset:d3595143e506] Need ability to delete resources
    • No thought yet given to uploaded resources. Long term re-think probable.
  • [Done; cset:2430a68a59fc] Empty resource table to resemble view page ("no resources" text)

## General ##

  • [Done; cset:cc0ad47293e5] Highlight the package tab I'm on (view/edit/history)
    • More generally, the presence of the #minornavigation bar is very inconsistent throughout the site.
  • [Done; cset:d5babc3629cf] Remove 'license' filtered search from backend
  • [Done] Move script stuff to footer of page
1316170410000000 1317650043000000
#1349 enhancement zephod zephod ckan-sprint-2011-09-26 closed fixed Template tidying

Various things in templates can be tidied up after work has finished on #1294:

  • remove styling in page in group/index.html (put in css main)
    • ditto for package/search.html
    • ditto for home/index.html
  • Forms.css - to merge into style.css; remove explicit imports elsewhere
    • @import at the top of style.css @import url(forms.css)
  • Make a note: Success/failure controllers clash CSS with success/failure flash banners
1316171025000000 1317650050000000
#1350 enhancement zephod zephod ckan-sprint-2011-09-26 closed fixed i18n of new javascript

New js code running produced in the work following #1294 contains various string literals, and needs proper internationalisation.

See files:

  • application.js
  • templates.js

Furthermore, efforts can be made to tidy up strings as described in #1329.

1316172576000000 1317650056000000
#1351 enhancement zephod zephod ckan-sprint-2011-09-26 closed fixed Tidy edit-user form; strip preview functionality

The template for editing a user's profile seems to output the same form twice in one page. Furthermore, the page is made needlessly complex by the preview functionality. The neat inline preview used in package will make this page a lot neater.

  • Fix the duplicate form
  • Strip the preview functionality from the front-end, replacing it with js editor
  • Strip the preview functionality from the back-end and test
1316534753000000 1317650062000000
#1353 defect nickstenning ckan-v1.5 closed fixed No UI to remove resources

I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org.

1316729765000000 1317075904000000
#1356 enhancement kindly amercader ckan-sprint-2011-10-10 closed fixed Can not recreate a deleted extra

If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).

1317034180000000 1318279617000000
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Note: See TracReports for help on using and creating reports.