CKAN: Ticket Query

#364: Search for 'statistic' returns nothing

dread — Mon, 12 Jul 2010 15:47:00 GMT

On ckan.net there are plenty of packages (and indeed their tags) with the word 'statistic' in them, but no packages turn up when you search for it:

http://ckan.net/package/search?q=statistic&search=Search+Packages+%C2%BB

(Using Postgres full text search)

#395: Set up profiling to analyze performance issues

pudo — Tue, 03 Aug 2010 08:38:59 GMT

At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls.

This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance.

NB: We should also agree on a maximum request latency.

TODO: Read up on all those QoS tickets to avoid overlapping efforts.

#422: Administrator imports meta-data into CKAN

dread — Tue, 10 Aug 2010 09:44:50 GMT

Requirements:

cater naturally for multiple packages with multiple resources
international encoding
large number of packages

#463: package diff exception

dread — Fri, 20 Aug 2010 12:07:17 GMT

Investigate exception:

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               response = self.app(environ, start_response)
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
AUTH_TYPE	'cookie'
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'en-US,en;q=0.8'
HTTP_CACHE_CONTROL	'max-age=0'
HTTP_COOKIE	'__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173'
HTTP_HOST	'ckan.net'
HTTP_REFERER	'http://ckan.net/package/history/dbtune-audioscrobbler'
HTTP_USER_AGENT	'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'92.226.62.168'
REMOTE_PORT	'55102'
REMOTE_USER	'http://pudo.org/'
REMOTE_USER_TOKENS	['']
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0xbb528cc>
beaker.cache	<beaker.cache.CacheManager object at 0xbb528ec>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0xbb5286c>>
beaker.session	{'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: __utma='27730403.1381387877.1281962773.1282248244.1282250173.11' __utmb='27730403.6.10.1282250173' __utmc='27730403' __utmz='27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan' auth_tkt='e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!'>, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0xf702aec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xf6eaf0c>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xf6eaf0c>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0xf6eaa8c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.identity	<repoze.who identity (hidden, dict-like) at 280815340>
repoze.who.logger	<logging.Logger instance at 0xbcdfccc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 196612236>, 'auth_tkt': <AuthTktCookiePlugin 198048140>}
routes.route	<routes.route.Route object at 0xbbc44cc>
routes.url	<routes.util.URLGenerator object at 0xf5e55ec>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0xfe0b0b0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xf5e55ec>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})

#465: Require API key for all API requests.

johnbywater — Fri, 20 Aug 2010 12:55:04 GMT

#467: Admin configures CKAN to expect API key in named HTTP header

johnbywater — Fri, 20 Aug 2010 13:22:42 GMT

#476: Daily database dumps shall be discoverable and downloadable

johnbywater — Fri, 20 Aug 2010 14:10:33 GMT

#477: Discover location of the daily database dumps

johnbywater — Fri, 20 Aug 2010 14:16:28 GMT

#481: Unrestricted total read-only access to catalogue API

johnbywater — Sat, 21 Aug 2010 20:31:37 GMT

#486: Catalogue service shall notify and query SOLR service

johnbywater — Sat, 21 Aug 2010 21:23:10 GMT

#511: Better warnings and errors when using API

dread — Wed, 25 Aug 2010 16:44:37 GMT

#533: The performance of CKAN shall scale

johnbywater — Fri, 27 Aug 2010 12:01:39 GMT

CKAN uses a database. We can put the database on another machine and then clone the CKAN machine. Then there are two machines calling the database and converting SQL results into HTTP responses.

How many machines can you have before the database becomes the bottleneck? At that point, what QoS can be obtained with normal hardware?

Afterwards, can CKAN be made to scale further than this with a little development? Can its Postgres database be mirrored in realtime? Or can we change SQL writes (create/updates) to write to many databases?

Are there any other options for scaling the persistence mechanism?

#544: Backport facet browsing to CKAN 1.2

pudo — Tue, 31 Aug 2010 15:08:12 GMT

This is in IATI, would be nice to have in generic CKAN.

http://bitbucket.org/okfn/iati/changeset/378431974c76

#557: Remove siblings from link_to/from relationship display

pudo — Fri, 03 Sep 2010 19:36:35 GMT

Displaying siblings is not needed for link_to/from.

#564: The system shall support location based search

johnbywater — Thu, 09 Sep 2010 12:03:49 GMT

#648: Enable lock-down of package creation

pudo — Sun, 19 Sep 2010 09:45:19 GMT

copy exisiting tests, modify authz in setUp, adapt and extend tests
- Problem: default_role_actions is read by init_db
- Solution: nuke db after monkey-patching
- role = model.Role('Reader'), del role.actions[...]

self.PRE_MAUTHZ_RULES = copy(mauthz.default_role_actions) mauthz.default_role_actions.remove((Role.READER, Action.CREATE)) #raise Exception(mauthz.default_role_actions) model.Session.remove() model.repo.rebuild_db()

Start from the functional
- Both the api controllers and wui controllers to do
- tests/function/test_authz.py extended to check CREATE in lockedDown mode. fails nicely. http://bitbucket.org/pudo/ckan-authz2

is_authorized(user, Action.Create, model.Package)

-> Doing this will put 'Package' in the context field of the user_object_role table. This will trigger SQLAlchemy to attempt a join towards PackageRole? in all queries. Since for class-level role assignments there never is a PackageRole? join table entry, this will never return any results.

cf branch "classes": http://bitbucket.org/pudo/ckan-authz2/src/6fd0475e0c66
No possibility of including "instance" column in inheritance decision.
Even if this can be cirumvented, it must happen on a per-query level and would require a major re-modelling.

have a ckan install that would not allow visitors to either list packages or list groups
- two paths: create the listing, but for each group/pkg decide that you cannot show this
- lock down the whole page (/package/list)
  - this is class-based, not object-based

is_authorized(user, Action.Package_Create, model.System()) is_authorized(user, Action.Group_Create, model.System())

[Separating Package and Group roles may be useful going forward: PackageEditor?, GroupEditor? etc]

Find a standard way to lock down classes
- possibly add default rows in user_object_role
- introduce lock-down into controllers: group new, package new, REST equivalents,
- confirm tests

#661: Create CKAN site tester

dread — Fri, 24 Sep 2010 17:15:58 GMT

From Rufus:

'blackbox' testing command (read-only only would be fine) -- very
useful for upgrade on sunday
On blackbox thinking of something like:
paster test-site http://hmg.ckan.net {package-name}
Or (maybe better):
paster test-site --site-spec {myconfig.json}
my-config is a json file with info for testing (could overload
--config rather than site-spec):
site-url: http://hmg.ckan.net/
package-name:
package-title:
...etc ...
Then:
Test front-page
Test /revision/
Test /package/{page}  -- check title
Test /package/history
Test /api/rest/package/ ...
Test /api/search/package ...
etc

#745: Web UI shall accept requests for harvesting jobs

johnbywater — Mon, 25 Oct 2010 13:25:48 GMT

#771: Run CLI help command without arguments

johnbywater — Wed, 27 Oct 2010 14:38:38 GMT

Should return something sensible (currently reports an error).

#772: Run CLI harvester command without arguments

johnbywater — Wed, 27 Oct 2010 14:39:33 GMT

Should return help for the harvester (currently crashes).

#773: Fix CLI help command to return CLI help message

johnbywater — Wed, 27 Oct 2010 14:41:15 GMT

#774: Fix CLI harvester command to return harvester help message

johnbywater — Wed, 27 Oct 2010 14:42:02 GMT

#796: AlphaPage for non-latin languages

pudo — Tue, 09 Nov 2010 09:57:42 GMT

The AlphaPage? implementation assumes that all packages start with a latin character. This is not true for Russian, Chinese, Arabian and Persian languages etc. We should either use a unicode solution (i.e. display a full class of characters based on the current locale) or use generic paging instead.

#807: Handle robot traffic better

rgrp — Tue, 16 Nov 2010 09:30:26 GMT

We get a lot of robot traffic. Notice that robots are visiting package edit pages. Suggest:

Add rel=nofollow to various links
Throttle some robots (not all seem to respect robots.txt)

#809: Link to the Wiki pages in package form

pudo — Wed, 17 Nov 2010 11:10:31 GMT

This would allow for a community-driven definition of field values, conventions, etc.

Plus: create wiki.ckan.net

#813: Add link to “Register new package” to the homepage

cygri — Wed, 17 Nov 2010 12:00:53 GMT

A link to the “register new package” page would be handy somewhere on the front page. As a frequent editor, I often go to http://ckan.net in order to create a new package. I don't want to hunt around for the link to the “new package” form.

The link could go into the bar next to “About / Statistics”. Or into the “More information” sidebar. Neither of these truly make sense, but it's a frequently accessed link and that's reason enough to put it somewhere onto the frontpage.

#814: Have an “About CKAN” link on every page

cygri — Wed, 17 Nov 2010 12:10:21 GMT

A link “About CKAN” pointing to http://ckan.net/about should be visible on every page. This is important because people often land on subpages and may have trouble figuring out what CKAN is.

I would put this link into the main navigation bar, on the very right next to “Revision History”. Then I would also remove the “Home” item from the main navigation bar because it is redundant. The CKAN logo is already a link to the homepage. The only other subpage in the “Home” section is “Statistics” and that's already linked from the homepage sidebar.

But anywhere else would be fine as well.

#815: Autocomplete for the search field

cygri — Wed, 17 Nov 2010 12:12:31 GMT

The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page.

#817: Proposed redesign of Resources table on package pages

cygri — Wed, 17 Nov 2010 12:36:24 GMT

There are some problems with the Resources table on package pages:

The column labelled “URL” doesn't contain URLs, but just the word “Download”.
The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links).

Attached is a screenshot for a proposed redesign. The changes are:

Move the Format column to the right of the Description column (description is more important)
Make the description a clickable link
Drop the “URL/Download” column because it's now redundant
Rename “Description” to “Resource” (not that important)

#834: Searching in CKAN

Alexander — Fri, 26 Nov 2010 11:06:04 GMT

Hello.

I've installed stable CKAN v1.1 from PyPI.

I can't find any docs about using CKAN API in order to query packages.

Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible?

Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&open_only=1&downloadable_only=1 and Ckanclient:

result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1})

As result nothing found.

Thanks.

#974: Document site_description

dread — Thu, 10 Feb 2011 12:50:54 GMT

Configuration option 'ckan.site_description' isn't documented

#976: Documentation references knowledgeforge.net

dread — Thu, 10 Feb 2011 14:04:09 GMT

#977: db_upgrade errors

dread — Fri, 11 Feb 2011 10:35:58 GMT

db_upgrade (in ckan/model/init) makes call to validate_authorization_setup but forgets about rest of initialisation.

it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?)

#991: Checkbox defaults to True

dread — Fri, 18 Feb 2011 13:19:35 GMT

Form for new package has CheckboxExtraField? checked, when the value is False. (as used in ckanext-dgu package v3 form)

#766: The system shall support harvesting metadata from ArcGIS

johnbywater — Wed, 27 Oct 2010 11:12:13 GMT

Perhaps 80% of the users of the UKLII may be using ArcGIS.

#786: The system shall support deleting a registered harvest source

johnbywater — Mon, 08 Nov 2010 10:09:42 GMT

#789: The harvest source form shall validate location field by positive identification of source type

johnbywater — Mon, 08 Nov 2010 10:28:26 GMT

#860: full text indexing for semantic.ckan.net

wwaites — Tue, 07 Dec 2010 12:47:09 GMT

...

#862: deploy rdf on catalogue.data.gov.uk

wwaites — Tue, 07 Dec 2010 12:50:37 GMT

handle various dgu extras, configure apache for autonegotiation, etc.

#879: Storage auth API

rgrp — Mon, 20 Dec 2010 18:08:29 GMT

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

#885: reimplement csw module to use owslib

wwaites — Sat, 25 Dec 2010 11:54:59 GMT

#886: move uklii forms api to dgu module

wwaites — Sat, 25 Dec 2010 11:55:45 GMT

doesn't belong in ckan core

#565: The system shall support UKLP code lists

johnbywater — Thu, 09 Sep 2010 12:04:24 GMT

#667: API is slow

dread — Thu, 30 Sep 2010 22:20:28 GMT

See message and script: http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html

1500 requests shouldn't take hours.

#676: The system shall support browsing to correct version of the CKAN documentation for a given service

johnbywater — Tue, 05 Oct 2010 10:55:36 GMT

#679: Improve standard package edit form

dread — Wed, 06 Oct 2010 17:27:30 GMT

Add help text, drawing from:
- ca.ckan.net
- http://wiki.okfn.org/ckan/doc/package
- http://wiki.okfn.org/ckan/doc/faq
Make longer text revealed on click

#711: Packages shall support both UK government and location information with a common set of attributes

johnbywater — Wed, 20 Oct 2010 13:21:06 GMT

Previously, a package type attribute was intended to distinguish between package data that came from government depts. and publishers location information (GEMINI). Now there is an intention use a common set of attributes to support data from either source.

Hence, there currently a v.3 of the government attributes, a document about extracting the attributes from Gemini document using XPath, and a comparison documents mapping between governemtn attributes and Gemini attributes.

That means there are potentially four different list of attributes. I'm not sure how to go about reviewing the lists and converging any differences.

But we do want to end up with a list of names, a description of their purpose, and then perhaps the XPath which would reveal the value from a Gemini doc?

#739: The system shall support retrieving a copy of the harvested metadata for a given package

johnbywater — Fri, 22 Oct 2010 20:35:10 GMT

...obviously, not all packages will have a harvested metadata document.

#757: Create migration script to add harvest source filter attribute to existing tables

johnbywater — Mon, 25 Oct 2010 21:03:04 GMT

#770: Improve the paster harvester help command

johnbywater — Wed, 27 Oct 2010 14:37:00 GMT

Issue 1. Help command without arguments fails:

$paster help --plugin ckan --config development.ini

gives:

EITHER

Command help not recognized

paster help: error: no such option: --plugin

Issue 2: Harvester command with arguments fails.

$ paster harvester Traceback (most recent call last): ...

cmd = self.args[0]

IndexError?: list index out of range

#778: Package edit form - name auto-generated from title

dread — Mon, 01 Nov 2010 16:59:37 GMT

#779: Package edit form - stylesheet in DGU updated

dread — Mon, 01 Nov 2010 17:00:00 GMT

#780: Package edit form - 'preview' button shows preview, with form lower in page

dread — Mon, 01 Nov 2010 17:00:29 GMT

This ticket refers to the form shown in Drupal. This doesn't have a preview function, but we don't have a requirement for it right now, so leaving ticket closed.

#781: Package edit form - 'extra' fields have 'add' and 'delete' buttons

dread — Mon, 01 Nov 2010 17:02:15 GMT

(Just like resource table, but extra fields don't need up/down buttons)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

#795: "Match up" GEMINI2 "metadata point of contact" with registered entities

johnbywater — Mon, 08 Nov 2010 14:53:45 GMT

TM wants to know how this will be "matched up". I don't know whether this always the data provider, or sometimes the publisher.

#884: harvest job delete operation

wwaites — Sat, 25 Dec 2010 11:50:00 GMT

implement paster subcommand to delete harvest jobs

#576: CKAN Requires Old Version of SQLAlchemy

wwaites — Fri, 10 Sep 2010 17:59:33 GMT

Requires 0.4. 0.5 is a maintenance branch, 0.6 is current.

VDM appears to work correctly (all the tests pass) with 0.5. CKAN does not -- requires more investigation to determine exactly why.

It would actually be nice to be able to run with 0.6 or 0.5, though 0.6 will require some changes to VDM as well

#797: Rework core html layout to mirror wordpress twentyten

rgrp — Wed, 10 Nov 2010 15:27:53 GMT

WP twentyten has an excellent core html structure. Furthermore, using that structure makes us compatible with all the WP twentyten compatible themes.

Convert to wp twentyten html
Switch css to be based off twentyten css (and perhaps rework somewhat)

#806: Metadata created and last modified timestamps for packages

rgrp — Mon, 15 Nov 2010 21:01:34 GMT

Packages should provide create and last modified attributes which are included in package dict exposed via api.

These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc)

#828: Update cache support in home controller to use cache config in ini file

rgrp — Thu, 25 Nov 2010 13:52:34 GMT

Currently cache usage in home controller is inconsistent with elsewhere and, in particular, does not respect cache_disabled and is not configurable using path notation (e.g. ckan.controllers.home. ...).

This is especially frustrating in a development environment where is currently running into fact front page is being cached.

Fix this.

#831: Migrate ckan trac to ckan.org

rgrp — Fri, 26 Nov 2010 08:40:20 GMT

Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc.

#832: Convert stats to a plugin

rgrp — Fri, 26 Nov 2010 10:05:15 GMT

Convert stats section to a ckan plugin.

Stats need not be in core and making it a plugin would make it easier to extend and expand.

#992: Use X-forwarded-for whenever it is available to set remote IP address

rgrp — Fri, 18 Feb 2011 19:48:52 GMT

At the moment we only use X-forwarded to determine remote user address when remote-addr header is 127.0.0.1. However if a site is behind a cache remote_addr will always be the IP of the cache.

We should fix this by using x-forwarded-for header whenever available and only use remote-addr when it is not available.

Cost: 10m

#231: Make group versioned

rgrp — Thu, 07 Jan 2010 08:45:23 GMT

Make group object revisioned/versioned.

Involves upgrade script both for group object and group_revision object
No changes required to WUI for present

#402: Archiving worker to back up package resources from a CKAN instance

pudo — Thu, 05 Aug 2010 14:35:12 GMT

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

Naming scheme?
- Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
  - What happens if names change
- File: filename? hash?
Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages.

Extra points for:

Properly checking for source file modification (Last-modified, Etag)
Using PIP VCS Backends for retrieval
OFS/S3 Storage

#647: Authorization Groups for group-based access control

pudo — Sun, 19 Sep 2010 09:43:38 GMT

allow group-level authorization on all system entities

#782: Add and remove users from sysadmin role from the command line

rgrp — Mon, 08 Nov 2010 08:34:47 GMT

Need to be able to create (and remove) sysadmins easily (used to be able to do this via ini file but this functionality was removed a while back in authz system reorganization it seems).

Suggest creating a new paster command as first step.

# user identifier = openid (?)
paster sysadmin add {user identifier}
paster sysadmin remove {user identifier}

Suggest this also be added to main CKAN docs.

This should go into metastable as well as default so that it goes into the v1.2 release (can do this via a transplant from one branch to the other)

#835: Authorization CLI and Documentation

pudo — Mon, 29 Nov 2010 11:11:48 GMT

As a sysadmin I want to take CKAN out of wiki-mode into normal mode. In normal mode visitor and logged in users cannot create or add packages.

Suggest put this as an internal command in authz system and add to cli as paster command.

#903: Proper handling of deleted packages in Solr serps

pudo — Wed, 12 Jan 2011 11:17:34 GMT

Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr.

#940: OpenID login via google for same user generates 2 different openid urls

rgrp — Sun, 30 Jan 2011 15:32:01 GMT

This has been noted twice in the last couple of weeks.

Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See:

To fix this we should set the openid.realm (not sure how atm).

If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache).

Cost: 1h

#946: No default system rights

dread — Mon, 31 Jan 2011 18:52:02 GMT

CKAN currently comes out of the box being uneditable. We'd like to encourage people to have open CKANs and therefore by default init the db with this right:

visitor -> is editor on -> System system

#398: Create Public API documentation for DGU (draft)

johnbywater — Thu, 05 Aug 2010 09:33:48 GMT

#413: Write functional tests for DGU caching

johnbywater — Tue, 10 Aug 2010 09:13:53 GMT

#566: The system shall support harvesting remote metadata

johnbywater — Thu, 09 Sep 2010 13:40:19 GMT

#568: Pull metadata documents from given harvest source entity

johnbywater — Thu, 09 Sep 2010 13:48:51 GMT

#570: Validate metadata document against UKLP schematron

johnbywater — Thu, 09 Sep 2010 13:50:56 GMT

#571: Save metadata document and associate with harvest source entity

johnbywater — Thu, 09 Sep 2010 13:54:55 GMT

#572: Write CKAN package from metadata document

johnbywater — Thu, 09 Sep 2010 13:59:44 GMT

The attributes we need to read in are being defined here: http://okfnpad.org/uklii

The design advice is to prepare, test, and maintain a dictionary of XPaths for each of the attributes.

Work effort is to be directed to the maintainability of the XPath statements. We need to know which are needed for which documents, and to have a way of preventing cruft.

#611: Load local copy of XML schema for metadata document

johnbywater — Sat, 11 Sep 2010 15:23:59 GMT

#612: Check given XML schema validates given metadata document

johnbywater — Sat, 11 Sep 2010 15:25:50 GMT

#617: Check UKLP schematron validates given metadata document

johnbywater — Sat, 11 Sep 2010 15:34:58 GMT

#621: Metadata document links extracted from WAF response

johnbywater — Sat, 11 Sep 2010 15:58:27 GMT

#665: The system shall support withdrawing a harvested dataset or service from publication

johnbywater — Mon, 27 Sep 2010 11:50:50 GMT

Discussion between John and Peter:

Given we can identify a document, does the disappearance of a document from a registered source imply the disappearance of the metadata (such that we delete packages once the documents disappear from the registered source)?

I would expect a more explicit 'delete'. The UKLP Use Case Library describes this as "withdraw a dataset or service from publication" (part of UCD03 Maintain resources).

#675: Draft specification of CSW harvesting behaviour with guide for publishing agents

johnbywater — Mon, 04 Oct 2010 17:54:37 GMT

#689: Updated package for updated metadata document

johnbywater — Mon, 11 Oct 2010 09:25:34 GMT

#692: The system shall support presenting map thumbnail for bounding box.

johnbywater — Mon, 11 Oct 2010 18:50:49 GMT

To do this for the UKLP, the Ordnance Survey would need to add to their map service an operation to return image for a given bounding box, and COI would need to present the image on the search result page.

Otherwise, it would be possible to use Google.

#712: Model government data package with "v0.4" of the attributes

johnbywater — Wed, 20 Oct 2010 13:23:59 GMT

Where said "v0.4" would be the result of converging four lists mentioned in requirement #711.

#713: Review four definitions of DGU package attributes

johnbywater — Wed, 20 Oct 2010 13:25:58 GMT

#752: Change package attribute names used by Gemini harvesting to DGU "v.4"

johnbywater — Mon, 25 Oct 2010 20:40:05 GMT

#753: The system shall support filtered harvest of CSW sources

johnbywater — Mon, 25 Oct 2010 20:58:10 GMT

#755: Add filter attribute to harvest source entity

johnbywater — Mon, 25 Oct 2010 21:01:46 GMT

#756: Add filter field to harvest source form

johnbywater — Mon, 25 Oct 2010 21:02:25 GMT

#758: Change API documentation to indicate harvest source entity has filter attribute

johnbywater — Mon, 25 Oct 2010 21:04:03 GMT

#759: Construct and send filtered CSW GetRecords request

johnbywater — Mon, 25 Oct 2010 21:05:53 GMT

#760: Change "CSW Get Records" request class to accept and used given CSW filter

johnbywater — Mon, 25 Oct 2010 21:09:53 GMT

#768: Obtain example ArcGIS CSW server

johnbywater — Wed, 27 Oct 2010 11:15:34 GMT

#769: Tests for harvesting metadata from ArcGIS CSW API

johnbywater — Wed, 27 Oct 2010 11:16:19 GMT

#783: Delete harvested documents when remote metadata disappears from harvest source

johnbywater — Mon, 08 Nov 2010 09:53:49 GMT

We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer.

#785: Review document from PP regarding CSW and WAF guidance

johnbywater — Mon, 08 Nov 2010 09:58:58 GMT

Scope of CSW guidance document has been broadened to include WAF.

#788: API user deletes harvest source

johnbywater — Mon, 08 Nov 2010 10:13:16 GMT

Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?).

#790: Submit harvest source form with valid CSW location

johnbywater — Mon, 08 Nov 2010 10:34:39 GMT