id type owner reporter milestone status resolution summary description PosixTime ModifiedTime 10 enhancement johnbywater johnbywater milestone15 closed worksforme Update package "== As a == Authenticated user == I want to == Update the attributes of a package and add/remove releases == So that == The package attributes are changed. Package attributes are: * A title * A description (short or long) * Date added * Project url * package license(s) * download url * A list of releases with their dates and a summary line * a list of owners (referencing authenticated users) I would also like to be able to add comments == Notes == * Only an authenticated user is one of owners of a package may update it * Any authenticated user may add comments (but visitors may not) * can break down into: * Update a package by changing simple package attributes (value: 8) * Update a package by adding or removing an owner (value: 5) * Update a package by adding or removing comments (value: 7) " 1152550243000000 1183648150000000 34 defect somebody zool closed worksforme Page Not Found - returns with 200 rather than 404 as it should 1163008961000000 1253781550000000 139 defect rgrp rgrp v0.11 closed worksforme Ensure you cannot add same tag twice to a package "Seems some package have ended up with same tag twice: http://www.ckan.net/package/fred2dot0 Check whether this is still possible and if it is fix it ... Cost: 2h" 1254768552000000 1255188974000000 146 defect pudo rgrp closed worksforme Problem when an admin user logs in between commencing editing and saving " 1. Visitor starts editing a package (or creates a new package) 2. Visitor logs in before previewing or committing 3. Visitor is an admin (or sysadmin) 4. Preview or commit fails Reason (conjecture): the form we use for package editing is now dependent on whether you have admin privileges (we allow editing of state by admins). When you login the old form is now incorrect and this results in the formalchemy fieldset breaking." 1255013773000000 1291829862000000 183 enhancement rgrp rgrp closed worksforme Browse packages by rating At moment order packages by title. 1257534606000000 1290604779000000 243 defect rgrp dread closed worksforme Repetition of number of packages in a group "http://ckan.net/group/publicdomain has: ""There is 16 package in this group. There are 16 packages in this group.""" 1265750250000000 1266837796000000 291 defect rgrp dread closed worksforme Investigate search options encoding "On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in. {{{ diff -r 813ad8b5de0b ckan/lib/search.py --- a/ckan/lib/search.py Mon Mar 01 22:23:36 2010 +0100 +++ b/ckan/lib/search.py Thu Apr 15 19:11:42 2010 +0200 @@ -63,6 +63,9 @@ def query(self, options): '''For the given search options, returns a query object.''' self._options = options + # temporary hack! + if self._options.q: + self._options.q = self._options.q.encode('utf8') general_terms, field_specific_terms = self._parse_query_string() if not general_terms and \ }}} " 1271351670000000 1273254895000000 442 requirement dread dread ckan-v1.3 closed worksforme Importing data "Review what chunks of code we have and how to move forward with requirements. Requirements: * Bulk parsing is done away from the CKAN server Possible future requirements: * /importer for importing spreadsheets in WUI * validation-light - 'scoring' a package Current importers: * /importer controller uses: PackageImporter, form, PackageSaver * loader uses: package_importer -> API * API uses: form, PackageSaver Modules * package_importer: * xls/csv --SpreadsheetData--> table * table --DataRecords--> records * records --PackageImporter--> pkg_dicts * pkg_dict --form--> fs & validation * fs --PackageSaver--> package" 1282228977000000 1294661028000000 463 defect dread ckan-v1.3 closed worksforme package diff exception "Investigate exception: {{{ URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:73 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.revision:119 in diff << c.revision_to = model.Session.query(model.Revision).get( request.params.getone('diff')) diff = pkg.diff(c.revision_to, c.revision_from) c.diff = diff.items() c.diff.sort() >> diff = pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:340 in diff << display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] >> display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'en-US,en;q=0.8' HTTP_CACHE_CONTROL 'max-age=0' HTTP_COOKIE '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173' HTTP_HOST 'ckan.net' HTTP_REFERER 'http://ckan.net/package/history/dbtune-audioscrobbler' HTTP_USER_AGENT 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' REMOTE_ADDR '92.226.62.168' REMOTE_PORT '55102' REMOTE_USER 'http://pudo.org/' REMOTE_USER_TOKENS [''] REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '
Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80
\n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173') paste.parsed_querystring ([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'} repoze.who.identity repoze.who.logger repoze.who.plugins {'openid': , 'auth_tkt': } routes.route routes.url webob._parsed_query_vars (GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgiorg.routing_args (, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}) }}}" 1282306037000000 1294916148000000 511 requirement dread ckan-v1.3 closed worksforme Better warnings and errors when using API 1282754677000000 1297075354000000 731 bug dread dread ckan-v1.3 closed worksforme Geo coverage field losses in Form API Sometimes editing a package via api on dgu results in some countries being lost in geo coverage. 1287738539000000 1288038226000000 771 story johnbywater ckan-v1.3 closed worksforme Run CLI help command without arguments Should return something sensible (currently reports an error). 1288190318000000 1294413256000000 817 requirement cygri ckan-v1.3 closed worksforme Proposed redesign of Resources table on package pages "There are some problems with the Resources table on package pages: 1. The column labelled “URL” doesn't contain URLs, but just the word “Download”. 2. The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links). Attached is a screenshot for a proposed redesign. The changes are: 1. Move the Format column to the right of the Description column (description is more important) 2. Make the description a clickable link 3. Drop the “URL/Download” column because it's now redundant 4. Rename “Description” to “Resource” (not that important)" 1289997384000000 1297073724000000 821 defect dread ckan-v1.4-sprint-2 closed worksforme Some resources have disappeared in CKAN "Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end. Here are the details: * in CKAN's package view (in web interface, API and dump) no resources display * but they WERE created in CKAN, as shown by the revision diffs. * they are in the Drupal front-end So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal. Packages affected: anti-social-behaviour-orders-1999-2007 asylum-applications-jan-mar-2009 control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december coroners-statistics-england-and-wales courts-statistics-user-survey-england-and-wales court-statistics-company-insolvency-and-bankruptcy-england-and-wales court-statistics-england-and-wales court-statistics-mortages-and-landlord-possession-england-and-wales crime-in-england-and-wales crime-statistics-local-reoffending-england-and-wales crime-statistics-prison-and-probation-england-and-wales crime-statistics-reoffending-of-adults-england-and-wales crime-statistics-reoffending-of-juvenilles-england-and-wales data_gov_uk-datasets digest-uk-energy-statistics-2008 directgov-central-hottest-pages-monthly directgov-central-internal-search-terms-monthly directgov-section-visits-monthly electricity-consumption-2007 electricity-gas-consumption-2007 energy-consumption-uk-2008 final-energy-consumption-2007 foi-statistics-uk-central-government fuel-poverty-statistics-2007 gas-consumption-2007 gb-reported-bicycling-accidents gb-road-traffic-counts gb-traffic-matrix greenhouse-gas-emissions-2008 high-level-indicators-energy-use-2006 judicial-and-court-statistics-england-and-wales laboratory-tests-and-prices local-authority-carbon-dioxide-emissions-2007 magistrates-courts-statistics-survey-england-and-wales monthly-energy-prices monthly-energy-trends ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig ni_013_migrants_english_language_skills_and_knowledge ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so ni_026_specialist_support_to_victims_of_a_serious_sexual_offence ni_029_gun_crime_rate ni_031_re-offending_rate_of_registered_sex_offenders ni_032_repeat_incidents_of_domestic_violence ni_034_domestic_violence_-_murder ni_036_protection_against_terrorist_attack ni_038_drug_related_class_a_offending_rate ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an ni_109_delivery_of_sure_start_childrens_centres ni_126_early_access_for_women_to_maternity_services ni_127_self_reported_experience_of_social_care_users ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law ni_185_co2_reduction_from_local_authority_operations ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an other-fuels-consumption-2006 police-use-firearms-england-wales-2007-2008 prison-end-of-custody-licence-releases-and-recalls-england-and-wales prison-population-england-and-wales probation-offender-management-caseload-statistics-england-and-wales probation-statistics-quarterly-brief-england-and-wales quality-indicators-energy-data-2007 quarterly-energy-prices quarterly-energy-trends road-transport-energy-consumption-2007 sentencing-statistics-england-and-wales statistics-terrorism-arrests-outcomes-2001-2008 ukba-control-of-immigration-statistics-2008 ukba-control-of-immigration-statistics-2008-supplementary-tables uk-energy-in-brief-2008 uk-energy-sector-indicators-background-2008 uk-energy-sector-indicators-key-supporting-2008 uk-exportcontrollists uk-exportcontrol-sanctions uk-export-control-statistics uk-glossary-exportcontrol uk-ipo-offences weekly-fuel-prices" 1290015776000000 1298486642000000 834 task Alexander ckan-v1.3 closed worksforme Searching in CKAN "Hello. I've installed stable CKAN v1.1 from PyPI. I can't find any docs about using CKAN API in order to query packages. Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible? Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&open_only=1&downloadable_only=1 and Ckanclient: result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1}) As result nothing found. Thanks." 1290769564000000 1291633657000000 905 defect thejimmyg Stiivi ckan-sprint-2012-02-06 closed worksforme Unable to search with accented characters in package names "I have a package with name: ""Štatistické prehľady agendy cestovných pasov"" and when I try to search for ""Štatistické"" there are no results. CKAN setup: - psql DB with UTF-8 tables - psql search backend - ckan version: 1.3.1b Example: http://verejnedata.sk/package/agenda-cestovnch-pasov try to search for the first word in the package name. " 1294939120000000 1328638536000000 923 defect rgrp dread closed worksforme Search box doesn't work in leaderboard page "1. Go to: http://ckan.net/stats/leaderboard#content 2. In the far top-right of the browser, select the search box in 'water'. 3. Press enter to search. Nothing happens. Tried in: chrome, firefox" 1295867328000000 1340632144000000 1133 defect johnlawrenceaspden closed worksforme command line rights manipulation doesn't work "It appears that the command $ paster rights add russianfan admin warandpeace has no effect, even though $ paster rights remove russianfan admin warandpeace works fine. This may be specific to something I've done, could someone confirm? If it's the case more generally, then I'm assuming this behaviour is untested? Tests should probably be added." 1305054948000000 1324057072000000 1214 defect dread dread closed worksforme API improvements (following javascript use) "Notes on the CKAN API from Aron: * Server returns text/html for errors even when ""Accept: application/json"" header is set. * DELETE package request requires a Content-Length? Should be 405 Method Not Allowed? * Tag returned as a JSON object when updating but as a string when requesting. Same with ""extras"" content. * How to remove key from ""extras"" object? Passing None as described in the docs is invalid JSON and using null doesn't work. * Tags return package names rather than ids when querying GET /tag/{id} * Doesn't return rendered_text property on package update. * Relationships use ""object"" key rather than id or package_id." 1309975222000000 1314029628000000 1226 defect dread closed worksforme Genshi stream exception "IndexError exception in genshi.filters.i18n keeps popping up daily. e.g. {{{ ... Module ckan.controllers.authorization_group:28 in index << items_per_page=20 ) return render('authorization_group/index.html') def read(self, id): >> return render('authorization_group/index.html') Module ckan.lib.base:74 in render << return cached_template(template_name, render_template, cache_key=cache_key, cache_type=cache_type, cache_expire=cache_expire) #, ns_options=('method'), method=method) >> cache_type=cache_type, cache_expire=cache_expire) Module pylons.templating:249 in cached_template << return content else: return render_func() >> return render_func() Module ckan.lib.base:63 in render_template << stream = item.filter(stream) return literal(stream.render(method=method, encoding=None)) if 'Pragma' in response.headers: >> return literal(stream.render(method=method, encoding=None)) Module genshi.core:183 in render << method = self.serializer or 'xml' generator = self.serialize(method=method, **kwargs) return encode(generator, method=method, encoding=encoding, out=out) def select(self, path, namespaces=None, variables=None): >> return encode(generator, method=method, encoding=encoding, out=out) Module genshi.output:57 in encode << _encode = lambda string: string if out is None: return _encode(''.join(list(iterator))) for chunk in iterator: out.write(_encode(chunk)) >> return _encode(''.join(list(iterator))) Module genshi.output:339 in __call__ << for filter_ in self.filters: stream = filter_(stream) for kind, data, pos in stream: cached = cache_get((kind, data)) if cached is not None: >> for kind, data, pos in stream: Module genshi.output:670 in __call__ << _gen_prefix = _gen_prefix().next for kind, data, pos in stream: output = cache_get((kind, data)) if output is not None: >> for kind, data, pos in stream: Module genshi.output:771 in __call__ << push_text = textbuf.append pop_text = textbuf.pop for kind, data, pos in chain(stream, [(None, None, None)]): if kind is TEXT: >> for kind, data, pos in chain(stream, [(None, None, None)]): Module genshi.output:586 in __call__ << def __call__(self, stream): prev = (None, None, None) for ev in stream: if prev[0] is START: if ev[0] is END: >> for ev in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module ckanext.googleanalytics.plugin:62 in download_adder << (downloaded %s times)''' count = None for mark, (kind, data, pos) in stream: if mark and kind == START: href = data[1].get('href') >> for mark, (kind, data, pos) in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1175 in __call__ << """""" callable_value = hasattr(self.value, '__call__') for mark, (kind, data, pos) in stream: if mark is ENTER: if callable_value: >> for mark, (kind, data, pos) in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.template.base:618 in _include << tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: >> for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:618 in _include << tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: >> for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:618 in _include << tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: >> for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:605 in _include << from genshi.template.loader import TemplateNotFound for event in stream: if event[0] is INCLUDE: href, cls, fallback = event[1] >> for event in stream: Module genshi.template.markup:378 in _match << for event in self._match(self._flatten(template, ctxt, **vars), ctxt, start=idx + 1, **vars): yield event >> ctxt, start=idx + 1, **vars): Module genshi.template.markup:327 in _match << break for event in stream: # We (currently) only care about start and end events for matching >> for event in stream: Module genshi.template.base:545 in _flatten << while 1: for kind, data, pos in stream: if kind is START and data[1]: >> for kind, data, pos in stream: Module genshi.filters.i18n:177 in _generate << msgbuf.append(*previous) previous = None for event in msgbuf.translate(gettext(msgbuf.format())): yield event if previous: >> for event in msgbuf.translate(gettext(msgbuf.format())): Module genshi.filters.i18n:1029 in translate << ) parts = parse_msg(string) parts_counter = {} for order, string in parts: >> parts = parse_msg(string) Module genshi.filters.i18n:1143 in parse_msg << if string: parts.append((stack[-1], string)) return parts >> parts.append((stack[-1], string)) IndexError: list index out of range }}}" 1311008861000000 1314029434000000 1331 defect dread dread closed worksforme Setting a tag twice causes exception "To reproduce: 1. Create a package with two tags the same: ""bulk bulk"" 2. Click 'save' 3. 500 ERROR - 'Server Error'" 1315905959000000 1325355631000000 1412 defect rgrp seanh ckan-sprint-2012-02-06 closed worksforme application.js crashes when viewing a dataset with no resources "Create a new dataset, don't add any resources to the dataset yet, view the dataset in ckan, if you have firebug enabled you'll see the error ""resource is undefined"" at line 699 of application.js." 1319450625000000 1328226385000000 1467 defect thejimmyg thejimmyg ckan-sprint-2012-01-09 closed worksforme CKAN dumps dgu miss certain publisher information Pawel knows about this so David Read, Pawel and I need to find time to discuss it. 1321376042000000 1326120319000000 1615 enhancement thejimmyg ckan-v1.6 closed worksforme CKAN Should work behind a proxy server "This would allow deployment via Nginx or Apache using proxy to Paster, uWSGI. At the moment CKAN isn't aware of the proxy's IP address so when you perform an action which does a redirect (such as adding a package), CKAN redirects you to the *internal IP* not the external *proxy IP*. We would like this work to facilitate testing within VMs as part of our new build infrastructure. It would also be nice if CKAN worked when mounted at a path other than /. That could be dealt with in another ticket because it isn't a problem at the moment." 1325687841000000 1328888870000000 1646 defect zephod dread ckan-sprint-2012-01-23 closed worksforme Resource navigator options display spuriously "When viewing a dataset, the ""Resources"" navigation button contained the Resource titles on the Resource navigator button, instead of in a drop-down mouse-hover menu. http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f Chrome/Linux" 1326452700000000 1327407044000000 2374 enhancement ross dread ckan-v1.8 closed worksforme tag search paging Currently in the logic function tag_search you can specify limit and offset, but no count is returned. Therefore pagination is not possible for tag results. This is desired though. 1336154025000000 1340287976000000 2400 defect amercader ckan-sprint-2012-05-29 closed worksforme Password reset fails "One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed {{{ ""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"" }}} " 1337188614000000 1338205947000000 2435 defect seanh seanh ckan-v1.8 closed worksforme ckanext-qa doesn't work with CKAN 1.7+ 1337963228000000 1340810257000000 2488 defect johnglover johnglover ckan-sprint-2012-06-25 closed worksforme Group update via API fails validation with organization and publisher profile enabled "Reported on IRC by 'floapps'. * when posting a group update with {packages: [package-1, package-2, etc.]}, getting ValidationError: {'Packages': {'name': ['Missing value']}}, although package objects did contain a name field * same API call worked with organization and publisher profile disabled * also worked if added a form_to_db_schema_options function which used default_update_group_schema for api updates instead of default_group_schema * tried on both 1.7 and 1.7.1b" 1338806374000000 1339412538000000 2800 enhancement aron.carroll shevski demo phase 5 closed worksforme tooltip on popular datasets with number of views be good to do this in phase 3 1344243925000000 1344544461000000 2817 defect danieljohnlewis demo phase 5 closed worksforme Item Type Reverts on Invalid URL "Problem: When creating a related item, if you put in a invalid URL it loses the choice of Item Type (e.g. Visualisation, Application) and reverts to the first time (API). Expected: It remembers which one was chosen" 1344504076000000 1344544901000000 2819 defect danieljohnlewis demo phase 5 closed worksforme Related Item Summary of Descriptions Markdown Issue "Problem: When creating a related item it mentions that you can use markdown in the description, however markdown doesn't render in apps page nor in related items page. Expected: Make shortened descriptions (on apps page & related items page) render markdown " 1344504321000000 1344545479000000 26 enhancement somebody johnbywater closed duplicate A registered person creates their own tags for a package 1152551351000000 1152555283000000 75 enhancement dread rgrp closed duplicate "Record and display package ""usage"" information" " * Number of package page visits on ckan (can we get this straight from google analytics) * Number of times url or download url is used - now ticket:937 (Record download stats for resoures) How do we do this? * Google analytics will miss a lot of this usage (and how do we get that data out anyway) * Could use javascript but again misses usage. * One option is to redirect link but that is kind of nasty (but may be only option ...) " 1247828785000000 1296341223000000 87 enhancement rgrp rgrp closed duplicate Multiple download links Multiple download links, including links to mirrors and multiple formats/versions 1248693302000000 1258470719000000 97 enhancement rgrp rgrp closed duplicate Do not create a distribution on a path is something already exists there "(2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option). Cost: 1h" 1249983557000000 1318181317000000 106 enhancement dread rgrp closed duplicate Regularly convert CKAN data to RDF and put on Talis CC "Sister to ticket:90 (Link to RDF version of CKAN data on Talis Connected Commons). Talis have already kindly done an initial conversion. We should repeat this process regularly and re-upload the data to Talis CC. In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required. Attached is the ruby script used by Talis for conversion Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)" 1251454474000000 1256140649000000 125 enhancement dread rgrp v1.0 closed duplicate Edit Generic Package Attributes in WUI Split out from ticket:43 1253709712000000 1258377621000000 137 enhancement rgrp dread closed duplicate User has editable home page " * Generic text box for markdown about the user 'About' Model's user table reflects these: * 'about' attribute " 1254741703000000 1254741830000000 144 enhancement rgrp dread v0.11 closed duplicate Most popular packages listed on homepage "Based on number of views. Related to ticket:143." 1255010391000000 1265284457000000 147 enhancement dread dread v0.11 closed duplicate Parser and loader for esw.org data 1255440695000000 1255515162000000 151 enhancement dread rgrp v0.11 closed duplicate User object should have a created attribute "User object should have a ""created"" attribute initialized to current datetime. Require a db migration but o/w very simple. Cost: 1.5h" 1255589694000000 1257414545000000 153 enhancement dread dread v0.11 closed duplicate Group's packages listed alphabetically "This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there. Suggested by Jonathan Gray" 1255621515000000 1258971895000000 155 enhancement dread dread v1.0 closed duplicate Adding multiple packages to a group "Ability to add multiple packages to a group in one go (e.g. with 'add' link which makes drop down menu appear - so can add one after another - then submit simultaneously) Use a bit of javascript to add more dropdowns. Suggested by Jonathan Gray " 1255621779000000 1271760041000000 168 enhancement rgrp dread closed duplicate Show admins for a group in group view 1256291481000000 1257414795000000 169 enhancement dread dread closed duplicate Package derivations "A 'Derived' relationship can be applied from one package to another. e.g. sussex-demography is derived from census-2001 'Derived' relationship is: * directional * many:many * stateful 'derived' table columns: * id (primary key) * source_package (foreign key) * result_package (foreign key) * description (markdown text) Further tickets: * WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y * WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate * REST if - expose reading and writing this property" 1256304927000000 1266928708000000 176 enhancement dread dread closed duplicate Package dependencies "(Related to ticket:169 - Package derivations) A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.) e.g. london-traffic-visualisation depends on road-map 'Dependency' relationship is: * directional * many:many * stateful 'dependency' table columns: * id (primary key) * dependent (foreign key) * dependency (foreign key) Further tickets: * WUI - package view - have list of dependencies (do not need to list packages which depend on this one) * WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package') * REST api - expose reading and writing 'depends on' property. === Issues === * How do we deal with dependency at a particular version?" 1257162812000000 1266928721000000 180 enhancement rgrp jwyg v0.11 closed duplicate Tag cloud as way to view CKAN tags Create big tag cloud with all CKAN tags - perhaps weighting with size and colour... 1257534254000000 1265284374000000 186 enhancement rgrp rgrp closed duplicate Automated upload to archive.org s3 "(Follows on from ticket:107). We want to provide facility for users to automatically upload material. * Create an upload page * Store the material on archive.org using s3 - http://www.archive.org/help/abouts3.txt * Main issue is how we cache large datasets ..." 1257803430000000 1296341182000000 188 enhancement rgrp rgrp v0.11 closed duplicate Improve package listing views "Propose change to tabular-like format showing these attributes (perhaps should be configurable?) * Openness status * Title (not sure name is needed) * Tags Cost: 4h" 1257870031000000 1265294090000000 228 enhancement rgrp rgrp closed duplicate Deal with duplicate packages This needs to be thought out ... 1262085763000000 1290596875000000 245 enhancement rgrp rgrp closed duplicate Support for composite primary keys "Problem here is that foreign key then becomes ""complicated"" (composite). * Could also deprecate continuity_id field in favour of the basic foreign key on ie" 1265882630000000 1297066620000000 246 enhancement rgrp rgrp closed duplicate Support for primary key not named id At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys) 1265882862000000 1297066757000000 268 defect rgrp dread closed duplicate Select groups in Package edit form 1268068896000000 1285070682000000 294 enhancement thejimmyg dread closed duplicate Add/remove extra fields in Package edit form Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources. 1271756591000000 1291830960000000 296 enhancement johnbywater johnbywater closed duplicate Commit CKAN revisions to changeset system 1272279521000000 1294407032000000 297 enhancement johnbywater johnbywater closed duplicate Update CKAN repository from changeset system 1272279556000000 1294407051000000 298 enhancement johnbywater johnbywater closed duplicate Pull changesets from remote CKAN instance 1272279591000000 1294407080000000 299 enhancement johnbywater johnbywater closed duplicate Merge diverging lines of changesets 1272279698000000 1294407099000000 306 enhancement rgrp rgrp closed duplicate datapkg build command "Need to be able to build a distribution. Need: * new 'build' command * specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution. " 1272474212000000 1318181194000000 308 enhancement rgrp rgrp closed duplicate Autocomplete package names & tags in package search "Extracted from ticket:216. Dubious of its merit." 1273050549000000 1275302577000000 321 enhancement thejimmyg johnbywater closed duplicate Delegate authentication to Drupal "When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model. The Drupal front-end shall have: 1. Login page - fixed location, can authenticate users, on successful authentication sets auth cookie and redirects to HTTP_REFERER. 2. Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions. 3. Access denied page - fixed location, static resource, gently indicates what has happened, and how to ask for permission. The CKAN slave edit page shall: 1. Try to detect a Drupal session key (passed as cookie or as request param). 2. Redirect to Drupal login page if no session key. 3. Check authorisation if session key is found. 4. Redirect to access denied page if session key not authorised. 5. Present the Package edit page. 6. Reject unauthenticated or unauthorised edit submissions. 7. Snag invalid edit submissions from authenticated and authorised users. 8. Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end. " 1274705234000000 1291831399000000 358 enhancement rgrp dread ckan-v1.5 closed duplicate Resources in REST API "(spun out of ticket:336) Resource added to model API at: api/rest/resource === Example model request === GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028 returns: {{{ [{""id"": ""a3dd8f64-9078-4f04-845c-e3f047125028"", ""package_id"": ""b8a325c8-af2a-43f3-8245-9db7d73dfbfe"", ""URL"": ""http://scraperwiki.com/lincolnshire-councillors"", ""format"": ""CSV"", ""Description"": ""Scrape of www.lincs.gov/councillors.pdf by ScraperWiki."", ""hash"": """", ""position"": 2 }] }}} == Authorization == 1. Have it generic (ie. not per resource) and use an action/role on system 2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages) 3. Introduce Resource in authorization system (requires migration) Mixed model {{{ Create / Edit: if resource associated to package: check_permissions(package, update) else: check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System) }}}" 1277483282000000 1310128782000000 394 task johnbywater johnbywater closed duplicate Fix munin on DGU (?) 1280485351000000 1294407189000000 395 task pudo ckan-v1.3 closed duplicate Set up profiling to analyze performance issues "At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls. This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance. NB: We should also agree on a maximum request latency. TODO: Read up on all those QoS tickets to avoid overlapping efforts. " 1280824739000000 1294417538000000 402 task pudo pudo ckan-v1.3 closed duplicate Archiving worker to back up package resources from a CKAN instance "Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system. * Naming scheme? * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? * What happens if names change * File: filename? hash? * Store hash back on ckan instance? The caching worker will consumer update notifications and fetch packages. Extra points for: * Properly checking for source file modification (Last-modified, Etag) * Using PIP VCS Backends for retrieval * OFS/S3 Storage " 1281018912000000 1296467635000000 440 task dread dread closed duplicate Write and pass comprehensive performance tests "Run latest ckan on eu0. Automate some queries and searches. Check load and database connections / processes." 1282226932000000 1294417436000000 441 requirement dread dread ckan-v1.3 closed duplicate CKAN read-only state "When performing maintenance on CKAN it may be necessary to make CKAN obviously read-only, telling the users and restricting access to 'edit' pages. Examples of use: * Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done. * A CKAN is used just for distributing metadata and so is always read-only. Updates may still arrive through direct db manipulation, e.g.: * another (but writable) CKAN instance is connected to the same db * restoring database dumps from another CKAN db * Should a security be breached, all editing could be stopped " 1282227314000000 1292586309000000 444 task dread closed duplicate Discuss package relationships ideas with JF "* Create test data on visible ckan * Discuss with JF" 1282299238000000 1294414008000000 456 story johnbywater dread ckan-v1.2 closed duplicate Daily dump 1282299917000000 1282665858000000 464 task rgrp dread closed duplicate Request dgu db server access 1282306104000000 1282325194000000 467 story johnbywater ckan-v1.3 closed duplicate Admin configures CKAN to expect API key in named HTTP header 1282310562000000 1294411681000000 477 story johnbywater ckan-v1.3 closed duplicate Discover location of the daily database dumps 1282313788000000 1294411761000000 485 story johnbywater closed duplicate Performance beats QoS criteria 1282425219000000 1294411946000000 486 requirement johnbywater ckan-v1.3 closed duplicate Catalogue service shall notify and query SOLR service 1282425790000000 1291639321000000 487 story johnbywater closed duplicate Notify SOLR service of model events 1282425910000000 1291639404000000 497 story johnbywater johnbywater closed duplicate "Respond to CSW ""GetRecords"" request" 1282427334000000 1294407718000000 500 defect dread ckan-v1.2 closed duplicate Exception from diff "Investigate exception occured occasionally in last couple of days on ckan.net: {{{ WebApp Error: : 'NoneType' object has no attribute 'key' URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:73 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.revision:119 in diff << c.revision_to = model.Session.query(model.Revision).get( request.params.getone('diff')) diff = pkg.diff(c.revision_to, c.revision_from) c.diff = diff.items() c.diff.sort() >> diff = pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:340 in diff << display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] >> display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'zh-cn,zh-tw' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Baiduspider+(+http://www.baidu.com/search/spider.htm)' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' REMOTE_ADDR '123.125.66.32' REMOTE_PORT '63767' REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '
Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80
\n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (, '') paste.parsed_querystring ([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'} repoze.who.logger repoze.who.plugins {'openid': , 'auth_tkt': } routes.route routes.url webob._parsed_query_vars (GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgiorg.routing_args (, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}) }}}" 1282553033000000 1287747652000000 501 requirement pudo ckan-v1.2 closed duplicate Read-only maintenance mode CKAN should have a read-only maintenance mode with a nice little banner on all pages, appropriate REST messages etc. Bonus points if this is triggered via an environment variable and thus can be triggered by the surrounding apache. 1282554617000000 1282724566000000 512 story dread closed duplicate User creates package via API with incorrect core fields specified 1282754750000000 1294917121000000 514 defect dread ckan-v1.2 closed duplicate Inconsistent use of 'location' header in API "When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all. I've removed the info about the header in the docs in the meantime." 1282757357000000 1282757391000000 535 defect dread ckan-v1.2 closed duplicate genshi error when logged into sl.ckan.net Genshi exception when rendering the page whilst logged in to sl.ckan.net. 1283165774000000 1283167040000000 537 task wwaites wwaites closed duplicate Caching and Performance improvement "There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests. Introducing caching has to be done carefully and should be done in a ''standards compliant manner''. == General strategy == * Where possible, cache output within the pylons app (beaker). * Facilitate external caching in an end-user's web browser or a caching proxy * Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes). * When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data. == Tasks == These tasks should be broken into sub-tickets: * caching of parts of templates that are expensive to render (package list, tag list, group list) * caching of entire output using beaker particularly for API read operations. * need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches. * cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: http://www.mnot.net/cache_docs/#CACHE-CONTROL * Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well * Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner == Future Work == * Investigate ckanclient library maintaining a local cache as a web browser would * Investigate using a CDN like Google Storage or Amazon for serving cached data." 1283184362000000 1311178929000000 543 task wwaites rgrp closed duplicate Investigate partial page caching and edge-side includes "Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page. Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish. Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat." 1283244784000000 1311178918000000 544 requirement pudo ckan-v1.3 closed duplicate Backport facet browsing to CKAN 1.2 "This is in IATI, would be nice to have in generic CKAN. * http://bitbucket.org/okfn/iati/changeset/378431974c76" 1283267292000000 1291638966000000 561 defect pudo ckan-v1.2 closed duplicate Deleted packages are returned in the API Anja is reporting this, severe bug, I think. 1283775578000000 1283775711000000 563 requirement thejimmyg johnbywater closed duplicate Support a minimal CSW server interface or export to GeoNetwork 1284033576000000 1296592472000000 570 story johnbywater johnbywater ckan-v1.3 closed duplicate Validate metadata document against UKLP schematron 1284040256000000 1294407974000000 573 story johnbywater closed duplicate Add metadata entity to harvesting queue 1284045353000000 1284220987000000 574 story johnbywater closed duplicate Create UKLII package with attributes from remote metadata record 1284045805000000 1284222410000000 580 story johnbywater closed duplicate Write (create or update) CKAN package for metadata document 1284210730000000 1284223068000000 612 task johnbywater johnbywater ckan-v1.3 closed duplicate Check given XML schema validates given metadata document 1284218750000000 1294408188000000 617 task johnbywater johnbywater ckan-v1.3 closed duplicate Check UKLP schematron validates given metadata document 1284219298000000 1294408164000000 665 requirement johnbywater johnbywater ckan-v1.3 closed duplicate The system shall support withdrawing a harvested dataset or service from publication "Discussion between John and Peter: > Given we can identify a document, does the disappearance of a document > from a registered source imply the disappearance of the metadata (such > that we delete packages once the documents disappear from the registered > source)? > I would expect a more explicit 'delete'. The UKLP Use Case Library describes this as ""withdraw a dataset or service from publication"" (part of UCD03 Maintain resources). " 1285588250000000 1297268097000000 691 requirement thejimmyg johnbywater ckan-backlog closed duplicate Package Relationships 1286822735000000 1295610145000000 702 requirement johnbywater johnbywater ckan-v1.2 closed duplicate The system shall support changing package groups when editing a package 1287403778000000 1287403850000000 730 task rgrp closed duplicate Back up package data from all CKAN packages to storage.ckan.net "Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net. * Naming scheme? * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? * What happens if names change * File: filename? hash? * Store hash back on ckan instance?" 1287737109000000 1291139609000000 740 requirement thejimmyg johnbywater closed duplicate Get copy of harvested metadata for a given package 1287779799000000 1296592889000000 748 story johnbywater closed duplicate Link new sample package to previous sample package of continuous series 1288013849000000 1294412976000000 749 story johnbywater closed duplicate Fold up continuous series in search results behind newest sample package 1288014002000000 1294412986000000 750 enhancement thejimmyg johnbywater closed duplicate Get CSW records modified since given time 1288014402000000 1296592940000000 751 story johnbywater closed duplicate Get harvested document for a given package 1288014518000000 1288014616000000 752 task johnbywater johnbywater ckan-v1.3 closed duplicate "Change package attribute names used by Gemini harvesting to DGU ""v.4""" 1288039205000000 1294408472000000 755 task johnbywater johnbywater ckan-v1.3 closed duplicate Add filter attribute to harvest source entity 1288040506000000 1294408632000000 756 task johnbywater johnbywater ckan-v1.3 closed duplicate Add filter field to harvest source form 1288040545000000 1294408642000000 757 task thejimmyg johnbywater ckan-v1.3 closed duplicate Create migration script to add harvest source filter attribute to existing tables 1288040584000000 1296593448000000 758 task johnbywater johnbywater ckan-v1.3 closed duplicate Change API documentation to indicate harvest source entity has filter attribute 1288040643000000 1294409053000000 759 story johnbywater johnbywater ckan-v1.3 closed duplicate Construct and send filtered CSW GetRecords request 1288040753000000 1294408652000000 760 task johnbywater johnbywater ckan-v1.3 closed duplicate "Change ""CSW Get Records"" request class to accept and used given CSW filter" 1288040993000000 1294409111000000 764 enhancement thejimmyg dread ckan-v1.5 closed duplicate Read-only CKAN Web UI "Whilse using CKAN web interface, you are not tempted to edit stuff: * You know at all times this CKAN is read-only * All editing facilities are still seen but greyed-out with an indication why it is." 1288091619000000 1310133334000000 766 requirement wwaites johnbywater ckan-v1.3 closed duplicate The system shall support harvesting metadata from ArcGIS "Perhaps 80% of the users of the UKLII may be using ArcGIS." 1288177933000000 1296593519000000 768 task johnbywater johnbywater ckan-v1.3 closed duplicate Obtain example ArcGIS CSW server 1288178134000000 1294409308000000 769 task johnbywater johnbywater ckan-v1.3 closed duplicate Tests for harvesting metadata from ArcGIS CSW API 1288178179000000 1294409318000000 772 story johnbywater ckan-v1.3 closed duplicate Run CLI harvester command without arguments Should return help for the harvester (currently crashes). 1288190373000000 1294413313000000 773 task johnbywater ckan-v1.3 closed duplicate Fix CLI help command to return CLI help message 1288190475000000 1294413400000000 774 task johnbywater ckan-v1.3 closed duplicate Fix CLI harvester command to return harvester help message 1288190522000000 1294413412000000 776 enhancement rgrp dread closed duplicate Avoid generating vdm warnings "We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided? {{{ 2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object 2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object 2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object 2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object 2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object 2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object }}}" 1288372797000000 1297066840000000 781 task thejimmyg dread ckan-v1.3 closed duplicate Package edit form - 'extra' fields have 'add' and 'delete' buttons "(Just like resource table, but extra fields don't need up/down buttons) Nice to have: a blank field is added when you tab from the last filled-in field in the table." 1288630935000000 1294415081000000 783 story johnbywater johnbywater ckan-v1.3 closed duplicate Delete harvested documents when remote metadata disappears from harvest source We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer. 1289210029000000 1294409399000000 786 requirement wwaites johnbywater ckan-v1.3 closed duplicate The system shall support deleting a registered harvest source 1289210982000000 1294410004000000 788 story johnbywater johnbywater ckan-v1.3 closed duplicate API user deletes harvest source Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?). 1289211196000000 1294410007000000 789 requirement wwaites johnbywater ckan-v1.3 closed duplicate The harvest source form shall validate location field by positive identification of source type 1289212106000000 1296593257000000 790 story johnbywater johnbywater ckan-v1.3 closed duplicate Submit harvest source form with valid CSW location 1289212479000000 1294409709000000 791 story johnbywater johnbywater ckan-v1.3 closed duplicate Submit harvest source form with invalid CSW location 1289212751000000 1294409723000000 795 story thejimmyg johnbywater ckan-v1.3 closed duplicate """Match up"" GEMINI2 ""metadata point of contact"" with registered entities" "TM wants to know how this will be ""matched up"". I don't know whether this always the data provider, or sometimes the publisher." 1289228025000000 1296593361000000 799 requirement johnbywater johnbywater closed duplicate WAF harvesting shall be optimised in respect of reharvesting only records that have changed 1289483264000000 1294232675000000 800 story johnbywater johnbywater ckan-v1.3 closed duplicate Get WAF records modified since given time 1289483343000000 1294245610000000 801 enhancement wwaites johnbywater ckan-v1.4-sprint-3 closed duplicate Get/Set last harvested time for a given harvest source Implies that last harvested time must be stored on source by harvesting job (see #802). 1289483546000000 1300196714000000 802 story johnbywater johnbywater closed duplicate Save last harvested time on source 1289484226000000 1294233294000000 815 requirement cygri ckan-v1.3 closed duplicate Autocomplete for the search field The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page. 1289995951000000 1294410951000000 841 enhancement kindly dread ckan-v1.4-sprint-4 closed duplicate Caching docs (as a whole) "Documentation article on caching / improving performance. (To complement configuration docs.) * Different sorts of cache - beaker style, etags, package_dict in search results(?) * How each one affects performance * How to turn them on/off and configure them * Is it possible to bypass each of them in the browser or with wget/curl?" 1291308879000000 1300364333000000 849 defect pudo closed duplicate Import extra CSS from main template The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import. 1291714923000000 1291715179000000 886 defect wwaites wwaites ckan-v1.3 closed duplicate move uklii forms api to dgu module doesn't belong in ckan core 1293278145000000 1294916538000000 896 story thejimmyg closed duplicate Distributed Data and Syncing Between CKAN Instances "Based on the ideas in: * DistributingChanges * SyncingInstances Merging the tickets #296, #297, #298, #299 into this one. Related ideas include: * Using dcat to exchange data * Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance This should be considered a wishlist item for the timebeing. " 1294406999000000 1300217375000000 897 defect dread dread closed duplicate """New package"" highlighted in Navigator when searching" "On ckan.net 1.3.1b, click on ""Search"" in the Navigator. Now both ""Search"" and ""New Package"" are highlighted." 1294648815000000 1294914333000000 916 enhancement rgrp pudo vdm-0.8 closed duplicate Port new vdm to Mongodb "We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG. " 1295277011000000 1297066902000000 922 enhancement kindly kindly closed duplicate Make Resources first class entity. "Why? * Will make it easier to give resources their own form, independent of packages. * Makes uploading, harvesting and previewing data cleaner as its separate from the packages. * Gives the option of sharing a resource between packages. * Leads the way to supporting storing richer semantic information against them." 1295713858000000 1320664187000000 947 enhancement rgrp closed duplicate Move the assignment of roles to a package on package creation to an extension "At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded. This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833). An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own)." 1296557478000000 1314031310000000 958 enhancement kindly ckan-v1.5 closed duplicate ui for new kind field Ui for related new field in resource (#957) 1296822896000000 1320664462000000 971 enhancement sebbacon sebbacon closed duplicate Support middleware as a plugin I need to completely munge the rendered data coming from the app. There's not currently a plugin hook for this, where I get access to the entire environment. The most generic form of plugin would be WSGI Middleware. I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system. 1297277473000000 1299245064000000 985 enhancement pudo pudo pdeu-1 closed duplicate Harvest other data catalogues " * data.london.gov.uk (1d, ping DERI) * catalogues from Paris and Vienna (1d) Others? * digitaliser.dk (1d) * data-publica.com (2d) * data.suomi.fi (2d) * dati.piedmonte.it (1d) * portalu.de (2d)" 1297679799000000 1306408134000000 987 defect pudo pudo closed duplicate Common harvesting framework "We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include: * CSW/WFS for INSPIRE/UKLII (yields CKAN packages) * Catalogue scraping for LOD2 experiments (yields RDF graphs) * Atom/DCat for LOD2 production (yields RDF graphs) * OAI-PMH for http://datadryad.org/ and other dspace (yields CKAN packages) We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios. In general, such a system would have the following stages: * Source selection: find what to download/scrape/harvest/parse * Index retrieval (i.e. package index) * Item retrieval (i.e. package entity) * (Optional: Serialization) * Normalisation * Loading/Merging into CKAN Exisiting harvesters are at: * CSW: https://bitbucket.org/okfn/ckanext-csw/src/ * Scraper+CKAN: https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/" 1297684756000000 1311177705000000 997 defect kindly kindly closed duplicate change create on cli to upgrade When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. 1298287981000000 1298288665000000 1005 defect dread dread closed duplicate Editing in ckan missing publishers Package edit form at hmg.ckan.net has no publishers any more. 1298561348000000 1300100085000000 1024 enhancement pudo pudo ckan-v1.4-sprint-3 closed duplicate Lock down all controller actions in CKAN When a user visits the site, he/she may only see a login box. 1299493428000000 1299668648000000 1034 defect dread closed duplicate Flash message cached " 1. Login (shows ckan home page with flash message ""Welcome back xyz"") 2. Click away, to ""Add package"" say 3. Click back to ckan ""Home"". It shows ""Welcome back xyz"" again - it shouldn't, I assume? " 1299845308000000 1320174353000000 1064 defect amercader closed duplicate Remove Workers from ckanext-queue "The current implementation of Workers in ckanext-queue is broken. Basically the various consume / callback functions expect three arguments (routing_key, operation, payload) when they are in fact receiving only two of them (message_data, message). This is fairly easy to fix, but the question is if Workers add an extra complexity to use the messaging library directly." 1301417891000000 1323169787000000 1094 enhancement thejimmyg thejimmyg ckan-v1.5 closed duplicate [super] Refactor the Auth System "Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787. Two man reasons for the changes are: * We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer. * Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able. The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package). 1. The first thing the is_authorized method is a hook to a plugin which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!) Reason: authorization can be completely delegated to another system (or partially) 2. is_authorized method currently takes (username, action, object) but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model. 3. Rename these two classes to better reflect what they are * AuthorizationGroup -> UserGroup * Group -> PackageGroup 4. Rename the Editor role to PriveledgeUser since Editors sometimes can't edit. Although this sounds a bit radical we already have auth extensions. == Read-only CKAN Web UI == (Additional requirement from #764) Whilse using CKAN web interface, you are not tempted to edit stuff: * You know at all times this CKAN is read-only * All editing facilities are still seen but greyed-out with an indication why it is." 1303117973000000 1311173649000000 1102 defect johnlawrenceaspden closed duplicate searching broken in development setup "With the default test data created by paster db clean paster db init paster create-test-data going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy But none of those words ""Wonderful"", etc produce search hits. In fact as far as I can tell, nothing produces any search hits. That isn't true on ckan.net, where searching seems to work. " 1303491912000000 1303744552000000 1103 defect johnlawrenceaspden closed duplicate searching broken in development setup "With the default test data created by paster db clean paster db init paster create-test-data going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy But none of those words ""Wonderful"", etc produce search hits. In fact as far as I can tell, nothing produces any search hits. That isn't true on ckan.net, where searching seems to work. " 1303494538000000 1303744575000000 1161 requirement pudo amercader pdeu-1 closed duplicate Build a simplified theme for PDEU "We need to offer a strongly simplified version, read-only of CKAN under publicdata.eu, with a focus on its role as search engine instead of a data catalogue. This ticket relates to work on the PDEU theme only!" 1306407835000000 1306408026000000 1200 enhancement pudo closed duplicate Port PDEU theme to CKAN.net "PDEU has a newer layout which could be adapted to also be used on ckan.net. For this, things should be cleaned up, copied to the ckanext-ckan.net repo as needed and re-colored the core CKAN color scheme. Maybe we could have this coincide with a CKAN rename? " 1308824017000000 1311180218000000 1279 enhancement ross amercader ckan-v1.8 closed duplicate Refactor or deprecate Authorization Groups "Authorization Groups are going to be deprecated in 1.8 It is still unclear how Authorization Groups fit in the new authz model, so they have been left out of the logic and auth refactoring (Tickets #1229 and #1253)" 1313415367000000 1338212028000000 1309 enhancement zephod zephod ckan-sprint-2011-09-12 closed duplicate Registering new user requires immediate login "Registering a new user directs you to that user's profile page but does not log you in. There is an empty set of ""Recent changes"" and no ability to modify the page. Users must follow the login link in the top-right corner to continue. Registering a new user should immediately log you in." 1315225510000000 1315586335000000 1313 enhancement zephod dread closed duplicate Tweaks to captcha configuration "Awesome to have the captcha implementation and running on thedatahub.org! Here are some further suggestions concerning the config options, before this goes into a release: * Document ""ckan.recaptcha.publickey"" and ""ckan.recaptcha.privatekey"" in doc/configuration.rst * Which one config option defines whether recaptcha is enabled? There is disagreement between the template and the lib. Maybe better to have a third option for this. When setting the syntax for this, I can see us wanting to use captcha for other operations, like package edit, package create, group create etc. so perhaps we'd want to be able to turn each one on/off separately, using the name of the logic action. e.g. {{{ckan.recaptcha.enable = user_create, group_create}}} * We got this exception last night from the template: {{{ Module ?:37 in <<