CKAN: Ticket Query

#338: Reference groups by ID in addition to name, since group names can change

johnbywater — Mon, 07 Jun 2010 08:58:57 GMT

#372: Fix system limits on CKAN for DGU

johnbywater — Fri, 23 Jul 2010 11:49:12 GMT

Set limits in /etc/security/limits.conf so that we can always ssh in at least. Requested by DGU.

#382: Measure quality of service parameters

johnbywater — Wed, 28 Jul 2010 19:56:14 GMT

As a service administrator, I want to measure responsiveness, throughput, and availability of a CKAN service.

#384: Send alert when QoS measurements break expectation

johnbywater — Wed, 28 Jul 2010 20:10:41 GMT

#471: API user sends API key in correctly named header

johnbywater — Fri, 20 Aug 2010 13:48:08 GMT

#472: API user sends API key in incorrectly named header

johnbywater — Fri, 20 Aug 2010 13:48:28 GMT

#473: API user discovers correct header for sending API key

johnbywater — Fri, 20 Aug 2010 13:50:03 GMT

#491: Get form for creating harvest source entity

johnbywater — Sat, 21 Aug 2010 21:43:28 GMT

#492: Submit harvest source create form response to the API

johnbywater — Sat, 21 Aug 2010 21:44:02 GMT

#493: Get harvest source entity

johnbywater — Sat, 21 Aug 2010 21:44:43 GMT

#494: Get form for updating remote metadata entity

johnbywater — Sat, 21 Aug 2010 21:45:06 GMT

#495: Put form for updating remote metadata entity

johnbywater — Sat, 21 Aug 2010 21:45:50 GMT

#545: The system shall support creating packages via the Form API

johnbywater — Wed, 01 Sep 2010 11:08:30 GMT

#546: Get the package create form from the API

johnbywater — Wed, 01 Sep 2010 11:10:14 GMT

#547: Submit package create form response to the API

johnbywater — Wed, 01 Sep 2010 11:11:27 GMT

#567: Post new harvest job for given harvest source

johnbywater — Thu, 09 Sep 2010 13:45:43 GMT

#568: Pull metadata documents from given harvest source entity

johnbywater — Thu, 09 Sep 2010 13:48:51 GMT

#572: Write CKAN package from metadata document

johnbywater — Thu, 09 Sep 2010 13:59:44 GMT

The attributes we need to read in are being defined here: http://okfnpad.org/uklii

The design advice is to prepare, test, and maintain a dictionary of XPaths for each of the attributes.

Work effort is to be directed to the maintainability of the XPath statements. We need to know which are needed for which documents, and to have a way of preventing cruft.

#577: Get remote metadata harvest job errors

johnbywater — Sat, 11 Sep 2010 12:49:41 GMT

#578: Get remote metadata harvest job

johnbywater — Sat, 11 Sep 2010 12:50:07 GMT

#579: Delete remote metadata harvest job

johnbywater — Sat, 11 Sep 2010 12:50:40 GMT

#598: List remote metadata entities for given publisher

johnbywater — Sat, 11 Sep 2010 14:02:10 GMT

#613: Store result of schema validation check on metadata document object

johnbywater — Sat, 11 Sep 2010 15:27:08 GMT

#654: Harvest sources and jobs should return 404 when missing (not 500)

johnbywater — Wed, 22 Sep 2010 15:51:57 GMT

#655: Return status code 404 when harvest source is not found

johnbywater — Thu, 23 Sep 2010 14:33:19 GMT

#656: Return status code 404 when harvesting job is not found

johnbywater — Thu, 23 Sep 2010 14:33:49 GMT

#673: Construct and send CSW GetRecordById request

johnbywater — Mon, 04 Oct 2010 17:51:52 GMT

#674: Extract metadata documents from CSW GetRecordById

johnbywater — Mon, 04 Oct 2010 17:53:00 GMT

#688: Example GeoNetworks service for CSW development

johnbywater — Mon, 11 Oct 2010 08:48:13 GMT

#10: Update package

johnbywater — Mon, 10 Jul 2006 16:50:43 GMT

As a

Authenticated user

I want to

Update the attributes of a package and add/remove releases

So that

The package attributes are changed. Package attributes are:

A title
A description (short or long)
Date added
Project url
package license(s)
download url
A list of releases with their dates and a summary line
a list of owners (referencing authenticated users)

I would also like to be able to add comments

Notes

Only an authenticated user is one of owners of a package may update it
Any authenticated user may add comments (but visitors may not)
can break down into:
- Update a package by changing simple package attributes (value: 8)
- Update a package by adding or removing an owner (value: 5)
- Update a package by adding or removing comments (value: 7)

#34: Page Not Found - returns with 200 rather than 404 as it should

zool — Wed, 08 Nov 2006 18:02:41 GMT

#139: Ensure you cannot add same tag twice to a package

rgrp — Mon, 05 Oct 2009 18:49:12 GMT

Seems some package have ended up with same tag twice: http://www.ckan.net/package/fred2dot0

Check whether this is still possible and if it is fix it ...

Cost: 2h

#146: Problem when an admin user logs in between commencing editing and saving

rgrp — Thu, 08 Oct 2009 14:56:13 GMT

Visitor starts editing a package (or creates a new package)
Visitor logs in before previewing or committing
Visitor is an admin (or sysadmin)
Preview or commit fails

Reason (conjecture): the form we use for package editing is now dependent on whether you have admin privileges (we allow editing of state by admins). When you login the old form is now incorrect and this results in the formalchemy fieldset breaking.

#183: Browse packages by rating

rgrp — Fri, 06 Nov 2009 19:10:06 GMT

At moment order packages by title.

#243: Repetition of number of packages in a group

dread — Tue, 09 Feb 2010 21:17:30 GMT

http://ckan.net/group/publicdomain has:

"There is 16 package in this group. There are 16 packages in this group."

#291: Investigate search options encoding

dread — Thu, 15 Apr 2010 17:14:30 GMT

On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in.

diff -r 813ad8b5de0b ckan/lib/search.py
--- a/ckan/lib/search.py	Mon Mar 01 22:23:36 2010 +0100
+++ b/ckan/lib/search.py	Thu Apr 15 19:11:42 2010 +0200
@@ -63,6 +63,9 @@
     def query(self, options):
         '''For the given search options, returns a query object.'''
         self._options = options
+        # temporary hack!
+        if self._options.q:
+            self._options.q = self._options.q.encode('utf8')
         general_terms, field_specific_terms = self._parse_query_string()
         if not general_terms and \

#442: Importing data

dread — Thu, 19 Aug 2010 14:42:57 GMT

Review what chunks of code we have and how to move forward with requirements.

Requirements:

Bulk parsing is done away from the CKAN server

Possible future requirements:

/importer for importing spreadsheets in WUI
validation-light - 'scoring' a package

Current importers:

/importer controller uses: PackageImporter?, form, PackageSaver?
loader uses: package_importer -> API
API uses: form, PackageSaver?

Modules

package_importer:
- xls/csv --SpreadsheetData?--> table
- table --DataRecords?--> records
- records --PackageImporter?--> pkg_dicts

pkg_dict --form--> fs & validation
fs --PackageSaver?--> package

#463: package diff exception

dread — Fri, 20 Aug 2010 12:07:17 GMT

Investigate exception:

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               response = self.app(environ, start_response)
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
AUTH_TYPE	'cookie'
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'en-US,en;q=0.8'
HTTP_CACHE_CONTROL	'max-age=0'
HTTP_COOKIE	'__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173'
HTTP_HOST	'ckan.net'
HTTP_REFERER	'http://ckan.net/package/history/dbtune-audioscrobbler'
HTTP_USER_AGENT	'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'92.226.62.168'
REMOTE_PORT	'55102'
REMOTE_USER	'http://pudo.org/'
REMOTE_USER_TOKENS	['']
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0xbb528cc>
beaker.cache	<beaker.cache.CacheManager object at 0xbb528ec>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0xbb5286c>>
beaker.session	{'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: __utma='27730403.1381387877.1281962773.1282248244.1282250173.11' __utmb='27730403.6.10.1282250173' __utmc='27730403' __utmz='27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan' auth_tkt='e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!'>, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0xf702aec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xf6eaf0c>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xf6eaf0c>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0xf6eaa8c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.identity	<repoze.who identity (hidden, dict-like) at 280815340>
repoze.who.logger	<logging.Logger instance at 0xbcdfccc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 196612236>, 'auth_tkt': <AuthTktCookiePlugin 198048140>}
routes.route	<routes.route.Route object at 0xbbc44cc>
routes.url	<routes.util.URLGenerator object at 0xf5e55ec>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0xfe0b0b0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xf5e55ec>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})

#511: Better warnings and errors when using API

dread — Wed, 25 Aug 2010 16:44:37 GMT

#731: Geo coverage field losses in Form API

dread — Fri, 22 Oct 2010 09:08:59 GMT

Sometimes editing a package via api on dgu results in some countries being lost in geo coverage.

#771: Run CLI help command without arguments

johnbywater — Wed, 27 Oct 2010 14:38:38 GMT

Should return something sensible (currently reports an error).

#817: Proposed redesign of Resources table on package pages

cygri — Wed, 17 Nov 2010 12:36:24 GMT

There are some problems with the Resources table on package pages:

The column labelled “URL” doesn't contain URLs, but just the word “Download”.
The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links).

Attached is a screenshot for a proposed redesign. The changes are:

Move the Format column to the right of the Description column (description is more important)
Make the description a clickable link
Drop the “URL/Download” column because it's now redundant
Rename “Description” to “Resource” (not that important)

#821: Some resources have disappeared in CKAN

dread — Wed, 17 Nov 2010 17:42:56 GMT

Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end.

Here are the details:

in CKAN's package view (in web interface, API and dump) no resources display
but they WERE created in CKAN, as shown by the revision diffs.
they are in the Drupal front-end

So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal.

Packages affected: anti-social-behaviour-orders-1999-2007 asylum-applications-jan-mar-2009 control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december coroners-statistics-england-and-wales courts-statistics-user-survey-england-and-wales court-statistics-company-insolvency-and-bankruptcy-england-and-wales court-statistics-england-and-wales court-statistics-mortages-and-landlord-possession-england-and-wales crime-in-england-and-wales crime-statistics-local-reoffending-england-and-wales crime-statistics-prison-and-probation-england-and-wales crime-statistics-reoffending-of-adults-england-and-wales crime-statistics-reoffending-of-juvenilles-england-and-wales data_gov_uk-datasets digest-uk-energy-statistics-2008 directgov-central-hottest-pages-monthly directgov-central-internal-search-terms-monthly directgov-section-visits-monthly electricity-consumption-2007 electricity-gas-consumption-2007 energy-consumption-uk-2008 final-energy-consumption-2007 foi-statistics-uk-central-government fuel-poverty-statistics-2007 gas-consumption-2007 gb-reported-bicycling-accidents gb-road-traffic-counts gb-traffic-matrix greenhouse-gas-emissions-2008 high-level-indicators-energy-use-2006 judicial-and-court-statistics-england-and-wales laboratory-tests-and-prices local-authority-carbon-dioxide-emissions-2007 magistrates-courts-statistics-survey-england-and-wales monthly-energy-prices monthly-energy-trends ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig ni_013_migrants_english_language_skills_and_knowledge ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so ni_026_specialist_support_to_victims_of_a_serious_sexual_offence ni_029_gun_crime_rate ni_031_re-offending_rate_of_registered_sex_offenders ni_032_repeat_incidents_of_domestic_violence ni_034_domestic_violence_-_murder ni_036_protection_against_terrorist_attack ni_038_drug_related_class_a_offending_rate ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an ni_109_delivery_of_sure_start_childrens_centres ni_126_early_access_for_women_to_maternity_services ni_127_self_reported_experience_of_social_care_users ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law ni_185_co2_reduction_from_local_authority_operations ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an other-fuels-consumption-2006 police-use-firearms-england-wales-2007-2008 prison-end-of-custody-licence-releases-and-recalls-england-and-wales prison-population-england-and-wales probation-offender-management-caseload-statistics-england-and-wales probation-statistics-quarterly-brief-england-and-wales quality-indicators-energy-data-2007 quarterly-energy-prices quarterly-energy-trends road-transport-energy-consumption-2007 sentencing-statistics-england-and-wales statistics-terrorism-arrests-outcomes-2001-2008 ukba-control-of-immigration-statistics-2008 ukba-control-of-immigration-statistics-2008-supplementary-tables uk-energy-in-brief-2008 uk-energy-sector-indicators-background-2008 uk-energy-sector-indicators-key-supporting-2008 uk-exportcontrollists uk-exportcontrol-sanctions uk-export-control-statistics uk-glossary-exportcontrol uk-ipo-offences weekly-fuel-prices

#834: Searching in CKAN

Alexander — Fri, 26 Nov 2010 11:06:04 GMT

Hello.

I've installed stable CKAN v1.1 from PyPI.

I can't find any docs about using CKAN API in order to query packages.

Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible?

Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&open_only=1&downloadable_only=1 and Ckanclient:

result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1})

As result nothing found.

Thanks.

#905: Unable to search with accented characters in package names

Stiivi — Thu, 13 Jan 2011 17:18:40 GMT

I have a package with name: "Štatistické prehľady agendy cestovných pasov" and when I try to search for "Štatistické" there are no results.

CKAN setup:

psql DB with UTF-8 tables

psql search backend

ckan version: 1.3.1b

Example:

http://verejnedata.sk/package/agenda-cestovnch-pasov

try to search for the first word in the package name.

#923: Search box doesn't work in leaderboard page

dread — Mon, 24 Jan 2011 11:08:48 GMT

Go to: http://ckan.net/stats/leaderboard#content
In the far top-right of the browser, select the search box in 'water'.
Press enter to search. Nothing happens.

Tried in: chrome, firefox

#1133: command line rights manipulation doesn't work

johnlawrenceaspden — Tue, 10 May 2011 19:15:48 GMT

It appears that the command

$ paster rights add russianfan admin warandpeace

has no effect, even though

$ paster rights remove russianfan admin warandpeace

works fine. This may be specific to something I've done, could someone confirm?

If it's the case more generally, then I'm assuming this behaviour is untested? Tests should probably be added.

#1214: API improvements (following javascript use)

dread — Wed, 06 Jul 2011 18:00:22 GMT

Notes on the CKAN API from Aron:

Server returns text/html for errors even when "Accept: application/json" header is set.
DELETE package request requires a Content-Length? Should be 405 Method Not Allowed?
Tag returned as a JSON object when updating but as a string when requesting. Same with "extras" content.
How to remove key from "extras" object? Passing None as described in the docs is invalid JSON and using null doesn't work.
Tags return package names rather than ids when querying GET /tag/{id}
Doesn't return rendered_text property on package update.
Relationships use "object" key rather than id or package_id.

#1226: Genshi stream exception

dread — Mon, 18 Jul 2011 17:07:41 GMT

IndexError? exception in genshi.filters.i18n keeps popping up daily.

e.g.

...
Module ckan.controllers.authorization_group:28 in index
<<              items_per_page=20
               )
               return render('authorization_group/index.html')
           def read(self, id):
>>  return render('authorization_group/index.html')
Module ckan.lib.base:74 in render
<<
           return cached_template(template_name, render_template, cache_key=cache_key,
                                  cache_type=cache_type, cache_expire=cache_expire)
                                  #, ns_options=('method'), method=method)
>>  cache_type=cache_type, cache_expire=cache_expire)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:63 in render_template
<<              stream = item.filter(stream)
               return literal(stream.render(method=method, encoding=None))
           if 'Pragma' in response.headers:
>>  return literal(stream.render(method=method, encoding=None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out=out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:339 in __call__
<<          for filter_ in self.filters:
                   stream = filter_(stream)
               for kind, data, pos in stream:
                   cached = cache_get((kind, data))
                   if cached is not None:
>>  for kind, data, pos in stream:
Module genshi.output:670 in __call__
<<          _gen_prefix = _gen_prefix().next
               for kind, data, pos in stream:
                   output = cache_get((kind, data))
                   if output is not None:
>>  for kind, data, pos in stream:
Module genshi.output:771 in __call__
<<          push_text = textbuf.append
               pop_text = textbuf.pop
               for kind, data, pos in chain(stream, [(None, None, None)]):
                   if kind is TEXT:
>>  for kind, data, pos in chain(stream, [(None, None, None)]):
Module genshi.output:586 in __call__
<<      def __call__(self, stream):
               prev = (None, None, None)
               for ev in stream:
                   if prev[0] is START:
                       if ev[0] is END:
>>  for ev in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module ckanext.googleanalytics.plugin:62 in download_adder
<<              (downloaded %s times)</span>'''
                   count = None
                   for mark, (kind, data, pos) in stream:
                       if mark and kind == START:
                           href = data[1].get('href')
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1175 in __call__
<<          """
               callable_value = hasattr(self.value, '__call__')
               for mark, (kind, data, pos) in stream:
                   if mark is ENTER:
                       if callable_value:
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.markup:378 in _match
<<                      for event in self._match(self._flatten(template, ctxt,
                                                                  **vars),
                                                    ctxt, start=idx + 1, **vars):
                               yield event
>>  ctxt, start=idx + 1, **vars):
Module genshi.template.markup:327 in _match
<<                      break
               for event in stream:
                   # We (currently) only care about start and end events for matching
>>  for event in stream:
Module genshi.template.base:545 in _flatten
<<          while 1:
                   for kind, data, pos in stream:
                       if kind is START and data[1]:
>>  for kind, data, pos in stream:
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range

#1331: Setting a tag twice causes exception

dread — Tue, 13 Sep 2011 09:25:59 GMT

To reproduce:

Create a package with two tags the same: "bulk bulk"
Click 'save'
500 ERROR - 'Server Error'

#1412: application.js crashes when viewing a dataset with no resources

seanh — Mon, 24 Oct 2011 10:03:45 GMT

Create a new dataset, don't add any resources to the dataset yet, view the dataset in ckan, if you have firebug enabled you'll see the error "resource is undefined" at line 699 of application.js.

#1467: CKAN dumps dgu miss certain publisher information

thejimmyg — Tue, 15 Nov 2011 16:54:02 GMT

Pawel knows about this so David Read, Pawel and I need to find time to discuss it.

#1615: CKAN Should work behind a proxy server

thejimmyg — Wed, 04 Jan 2012 14:37:21 GMT

This would allow deployment via Nginx or Apache using proxy to Paster, uWSGI. At the moment CKAN isn't aware of the proxy's IP address so when you perform an action which does a redirect (such as adding a package), CKAN redirects you to the *internal IP* not the external *proxy IP*.

We would like this work to facilitate testing within VMs as part of our new build infrastructure.

It would also be nice if CKAN worked when mounted at a path other than /. That could be dealt with in another ticket because it isn't a problem at the moment.

#1646: Resource navigator options display spuriously

dread — Fri, 13 Jan 2012 11:05:00 GMT

When viewing a dataset, the "Resources" navigation button contained the Resource titles on the Resource navigator button, instead of in a drop-down mouse-hover menu.

http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f

Chrome/Linux?

#2374: tag search paging

dread — Fri, 04 May 2012 17:53:45 GMT

Currently in the logic function tag_search you can specify limit and offset, but no count is returned. Therefore pagination is not possible for tag results. This is desired though.

#2400: Password reset fails

amercader — Wed, 16 May 2012 17:16:54 GMT

One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed

"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

#2435: ckanext-qa doesn't work with CKAN 1.7+

seanh — Fri, 25 May 2012 16:27:08 GMT

#2488: Group update via API fails validation with organization and publisher profile enabled

johnglover — Mon, 04 Jun 2012 10:39:34 GMT

Reported on IRC by 'floapps'.

when posting a group update with {packages: [package-1, package-2, etc.]}, getting ValidationError?: {'Packages': {'name': Missing value?}}, although package objects did contain a name field
same API call worked with organization and publisher profile disabled
also worked if added a form_to_db_schema_options function which used default_update_group_schema for api updates instead of default_group_schema
tried on both 1.7 and 1.7.1b

#2800: tooltip on popular datasets with number of views

shevski — Mon, 06 Aug 2012 09:05:25 GMT

be good to do this in phase 3

#2817: Item Type Reverts on Invalid URL

danieljohnlewis — Thu, 09 Aug 2012 09:21:16 GMT

Problem: When creating a related item, if you put in a invalid URL it loses the choice of Item Type (e.g. Visualisation, Application) and reverts to the first time (API). Expected: It remembers which one was chosen

#2819: Related Item Summary of Descriptions Markdown Issue

danieljohnlewis — Thu, 09 Aug 2012 09:25:21 GMT

Problem: When creating a related item it mentions that you can use markdown in the description, however markdown doesn't render in apps page nor in related items page. Expected: Make shortened descriptions (on apps page & related items page) render markdown

#26: A registered person creates their own tags for a package

johnbywater — Mon, 10 Jul 2006 17:09:11 GMT

#75: Record and display package "usage" information

rgrp — Fri, 17 Jul 2009 11:06:25 GMT

Number of package page visits on ckan (can we get this straight from google analytics)
Number of times url or download url is used - now ticket:937 (Record download stats for resoures)

How do we do this?

Google analytics will miss a lot of this usage (and how do we get that data out anyway)
Could use javascript but again misses usage.
One option is to redirect link but that is kind of nasty (but may be only option ...)

#87: Multiple download links

rgrp — Mon, 27 Jul 2009 11:15:02 GMT

Multiple download links, including links to mirrors and multiple formats/versions

#97: Do not create a distribution on a path is something already exists there

rgrp — Tue, 11 Aug 2009 09:39:17 GMT

(2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option).

Cost: 1h

#106: Regularly convert CKAN data to RDF and put on Talis CC

rgrp — Fri, 28 Aug 2009 10:14:34 GMT

Sister to ticket:90 (Link to RDF version of CKAN data on Talis Connected Commons).

Talis have already kindly done an initial conversion. We should repeat this process regularly and re-upload the data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Attached is the ruby script used by Talis for conversion

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

#125: Edit Generic Package Attributes in WUI

rgrp — Wed, 23 Sep 2009 12:41:52 GMT

Split out from ticket:43

#137: User has editable home page

dread — Mon, 05 Oct 2009 11:21:43 GMT

Generic text box for markdown about the user 'About'

Model's user table reflects these:

'about' attribute

#144: Most popular packages listed on homepage

dread — Thu, 08 Oct 2009 13:59:51 GMT

Based on number of views.

Related to ticket:143.

#147: Parser and loader for esw.org data

dread — Tue, 13 Oct 2009 13:31:35 GMT

#151: User object should have a created attribute

rgrp — Thu, 15 Oct 2009 06:54:54 GMT

User object should have a "created" attribute initialized to current datetime.

Require a db migration but o/w very simple.

Cost: 1.5h

#153: Group's packages listed alphabetically

dread — Thu, 15 Oct 2009 15:45:15 GMT

This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there.

Suggested by Jonathan Gray

#155: Adding multiple packages to a group

dread — Thu, 15 Oct 2009 15:49:39 GMT

Ability to add multiple packages to a group in one go (e.g. with 'add' link which makes drop down menu appear - so can add one after another - then submit simultaneously)

Use a bit of javascript to add more dropdowns.

Suggested by Jonathan Gray

#168: Show admins for a group in group view

dread — Fri, 23 Oct 2009 09:51:21 GMT

#169: Package derivations

dread — Fri, 23 Oct 2009 13:35:27 GMT

A 'Derived' relationship can be applied from one package to another.

e.g. sussex-demography is derived from census-2001

'Derived' relationship is:

directional
many:many
stateful

'derived' table columns:

id (primary key)
source_package (foreign key)
result_package (foreign key)
description (markdown text)

Further tickets:

WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y
WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate
REST if - expose reading and writing this property

#176: Package dependencies

dread — Mon, 02 Nov 2009 11:53:32 GMT

(Related to ticket:169 - Package derivations)

A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.)

e.g. london-traffic-visualisation depends on road-map

'Dependency' relationship is:

directional
many:many
stateful

'dependency' table columns:

id (primary key)
dependent (foreign key)
dependency (foreign key)

Further tickets:

WUI - package view - have list of dependencies (do not need to list packages which depend on this one)
WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package')
REST api - expose reading and writing 'depends on' property.

Issues

How do we deal with dependency at a particular version?

#180: Tag cloud as way to view CKAN tags

jwyg — Fri, 06 Nov 2009 19:04:14 GMT

Create big tag cloud with all CKAN tags - perhaps weighting with size and colour...

#186: Automated upload to archive.org s3

rgrp — Mon, 09 Nov 2009 21:50:30 GMT

(Follows on from ticket:107). We want to provide facility for users to automatically upload material.

Create an upload page
Store the material on archive.org using s3 - http://www.archive.org/help/abouts3.txt
Main issue is how we cache large datasets ...

#188: Improve package listing views

rgrp — Tue, 10 Nov 2009 16:20:31 GMT

Propose change to tabular-like format showing these attributes (perhaps should be configurable?)

Openness status
Title (not sure name is needed)
Tags

Cost: 4h

#228: Deal with duplicate packages

rgrp — Tue, 29 Dec 2009 11:22:43 GMT

This needs to be thought out ...

#245: Support for composite primary keys

rgrp — Thu, 11 Feb 2010 10:03:50 GMT

Problem here is that foreign key then becomes "complicated" (composite).

Could also deprecate continuity_id field in favour of the basic foreign key on ie

#246: Support for primary key not named id

rgrp — Thu, 11 Feb 2010 10:07:42 GMT

At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys)

#268: Select groups in Package edit form

dread — Mon, 08 Mar 2010 17:21:36 GMT

#294: Add/remove extra fields in Package edit form

dread — Tue, 20 Apr 2010 09:43:11 GMT

Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources.

#296: Commit CKAN revisions to changeset system

johnbywater — Mon, 26 Apr 2010 10:58:41 GMT

#297: Update CKAN repository from changeset system

johnbywater — Mon, 26 Apr 2010 10:59:16 GMT

#298: Pull changesets from remote CKAN instance

johnbywater — Mon, 26 Apr 2010 10:59:51 GMT

#299: Merge diverging lines of changesets

johnbywater — Mon, 26 Apr 2010 11:01:38 GMT

#306: datapkg build command

rgrp — Wed, 28 Apr 2010 17:03:32 GMT

Need to be able to build a distribution. Need:

new 'build' command
specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution.

#308: Autocomplete package names & tags in package search

rgrp — Wed, 05 May 2010 09:09:09 GMT

Extracted from ticket:216.

Dubious of its merit.

#321: Delegate authentication to Drupal

johnbywater — Mon, 24 May 2010 12:47:14 GMT

When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model.

The Drupal front-end shall have:

Login page - fixed location, can authenticate users, on successful authentication sets auth cookie and redirects to HTTP_REFERER.

Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions.

Access denied page - fixed location, static resource, gently

indicates what has happened, and how to ask for permission.

The CKAN slave edit page shall:

Try to detect a Drupal session key (passed as cookie or as request param).

Redirect to Drupal login page if no session key.

Check authorisation if session key is found.

Redirect to access denied page if session key not authorised.

Present the Package edit page.

Reject unauthenticated or unauthorised edit submissions.

Snag invalid edit submissions from authenticated and authorised users.

Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end.

#358: Resources in REST API

dread — Fri, 25 Jun 2010 16:28:02 GMT

(spun out of ticket:336)

Resource added to model API at:

api/rest/resource

Example model request

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028

returns:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors",
   "format": "CSV",
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "",
   "position": 2
 }]

Authorization

Have it generic (ie. not per resource) and use an action/role on system
Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
Introduce Resource in authorization system (requires migration)

Mixed model

Create / Edit:
if resource associated to package:
    check_permissions(package, update)
else:
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)

#394: Fix munin on DGU (?)

johnbywater — Fri, 30 Jul 2010 10:22:31 GMT

#395: Set up profiling to analyze performance issues

pudo — Tue, 03 Aug 2010 08:38:59 GMT

At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls.

This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance.

NB: We should also agree on a maximum request latency.

TODO: Read up on all those QoS tickets to avoid overlapping efforts.

#402: Archiving worker to back up package resources from a CKAN instance

pudo — Thu, 05 Aug 2010 14:35:12 GMT

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

Naming scheme?
- Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
  - What happens if names change
- File: filename? hash?
Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages.

Extra points for:

Properly checking for source file modification (Last-modified, Etag)
Using PIP VCS Backends for retrieval
OFS/S3 Storage

#440: Write and pass comprehensive performance tests

dread — Thu, 19 Aug 2010 14:08:52 GMT

Run latest ckan on eu0. Automate some queries and searches. Check load and database connections / processes.

#441: CKAN read-only state

dread — Thu, 19 Aug 2010 14:15:14 GMT

When performing maintenance on CKAN it may be necessary to make CKAN obviously read-only, telling the users and restricting access to 'edit' pages.

Examples of use:

Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done.
A CKAN is used just for distributing metadata and so is always read-only. Updates may still arrive through direct db manipulation, e.g.:
- another (but writable) CKAN instance is connected to the same db
- restoring database dumps from another CKAN db
Should a security be breached, all editing could be stopped

#444: Discuss package relationships ideas with JF

dread — Fri, 20 Aug 2010 10:13:58 GMT

Create test data on visible ckan
Discuss with JF

#456: Daily dump

dread — Fri, 20 Aug 2010 10:25:17 GMT

#464: Request dgu db server access

dread — Fri, 20 Aug 2010 12:08:24 GMT