id type owner reporter milestone status resolution summary description PosixTime ModifiedTime 1259 enhancement johnglover pudo ckan-backlog new """Add a row"" for Extras on Package form" "The default package form offers 4 empty extras fields. Like the resource section, it should have an ""add more"" button to add another row. " 1312302693000000 1312907056000000 2989 enhancement seanh ckan 2.0 new """Add dataset to organization"" should auto-select the organization" "'Add dataset to organization' button, when you get through to the third stage of the new dataset form the organization you came from is not selected. I don't see any option to choose the group when adding or updating a dataset, but if I add a dataset via the ""Add dataset to group"" button on a group's page, then the dataset seems to get added to that group. (And I can also add/remove existing datasets by editing the group.) I wonder if organizations should work the same way, instead of having an Organization drop-down when creating or updating a dataset. The add dataset page needs to somehow indicate that you're adding a dataset to a certain group or organization though, doesn't currently." 1350299052000000 1350299052000000 1486 defect seanh seanh closed fixed """allow_partial_update"" option does not work with resource lists of packages" "1. Call ckan.logic.action.update:package_update() 2. Pass in a context dict with ""allow_partial_update"":True 3. Pass in a partial data dict containing only the package's ID and those fields that you want to update (e.g. title). => All of the package's resource will have their statuses changed to ""deleted"". The ""allow_partial_update"" option should apply to the resource lists of packages, but it does not." 1322061318000000 1323172027000000 2971 enhancement seanh ckan 2.0 new """Are ytou sure you want to delete this member?"" should say which member" when deleting members from groups and orgs 1350296840000000 1350296840000000 1334 defect amercader amercader ckan-sprint-2011-10-28 closed fixed """AttributeError: 'NoneType' object has no attribute 'name'"" exception when logging in" "On same cases this exception is thrown when logging in: {{{ File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in user_dict = get_action('user_show')(context,data_dict) File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show revision_dict = revision_show(context,{'id':revision.id}) File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show ref_package_by=ref_package_by) File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict for pkg in revision.packages] AttributeError: 'NoneType' object has no attribute 'name' }}} " 1315909508000000 1319798961000000 1206 defect wwaites dread closed fixed """Content-Type json"" header scuppers package POST" "Compare these two requests to create a package: {{{ curl http://test.ckan.net/api/rest/package -d '{name:""test""}' -H 'Content-Type: application/json' -H 'X-CKAN-API-KEY: tester' curl http://test.ckan.net/api/rest/package -d '{name:""test""}' -H 'X-CKAN-API-KEY: tester' }}} The second one gets the payload through (ckan log): {{{ Retrieving request params: UnicodeMultiDict([('{name:""test""}', u'')]) }}} But the first one causes a ServerError because the payload (name:""test"") doesn't make it to request.POST or request.params: {{{ Retrieving request params: UnicodeMultiDict([]) }}} The only difference is the ""ContentType: application/json"" header, which seems a reasonable thing to include. Javascript lib backbone.js (for example) inserts this automatically. So why does this header cause the payload to not get through to the request object?" 1309344348000000 1309450216000000 935 defect pudo pudo closed fixed """groups"" field bug in IATI" "An internal server error was reported when trying to create a new package (not when editing): WebApp Error: : (IntegrityError) null value in column ""name"" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'} The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the ""groups"" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server. Possible causes include: * Stale pyc file * Threading issue * Dependency issue As the problem is gone for now, it cannot be further investigated atm. " 1296302915000000 1306778682000000 1395 defect seanh closed fixed """Import Error: cannot import name UnicodeMultiDict"" when installing ckan from source" "At the ''paster db init'' command when installing ckan from source I get the error ''""Import Error: cannot import name UnicodeMultiDict""'' (happens with both ckan 1.4.2 and today's latest bleeding edge code, on Ubuntu 10.04.3). UnicodeMultiDict has been removed in a recent version of python-webob, and the ''pip install ... lucid_missing.txt'' causes a too-new version of python-webob to be installed into ckan's virtualenv (the new webob gets installed as a dependency of formalchemy). I manually did ''pip uninstall webob'' and then ran ''paster db init'' again and it worked." 1318520183000000 1320857823000000 2993 defect seanh ckan 2.0 new """logged_in"" and ""visitor"" show in user list at /users" 1350466922000000 1350484826000000 795 story thejimmyg johnbywater ckan-v1.3 closed duplicate """Match up"" GEMINI2 ""metadata point of contact"" with registered entities" "TM wants to know how this will be ""matched up"". I don't know whether this always the data provider, or sometimes the publisher." 1289228025000000 1296593361000000 897 defect dread dread closed duplicate """New package"" highlighted in Navigator when searching" "On ckan.net 1.3.1b, click on ""Search"" in the Navigator. Now both ""Search"" and ""New Package"" are highlighted." 1294648815000000 1294914333000000 369 enhancement shudson@… ckan-backlog new """Package Listing Key"" should appear on Tag results" "Currently there's a nice legend titled ""Package Listing Key"" that appears in right side of ""Browse Packages"" results. The same key should show on other search results like when searching for a tag." 1279821634000000 1339774666000000 1735 enhancement zephod dread ckan-sprint-2012-02-20 closed wontfix """Unsaved changes"" warning briefly when you click ""Save""" "Super ticket #1744 If you edit a dataset, simply add a tag and click on ""Save"" then you get the message ""You have unsaved changed. Make sure to click 'Save Changes' below before leaving this page."" inserted whilst the submission takes place. (This is because the click moves focus away from the tag field). This (reasonably common) edge-case could usefully be squashed. Also note typo: ""unsaved changed""" 1328009420000000 1328609804000000 1194 defect dread closed fixed """Welcome back"" message for newly registered user" "1. Register a new user (/user/register) 2. It redirects you to user page and displays ""Welcome back"" even though you are a new user." 1308563392000000 1317053688000000 2887 defect seanh ckan-v1.8.1 new """Welcome to CKAN!"" on front page is untranslatable" "ckan/templates/home/index.html contains:

Welcome to ${g.site_title}!

which means that just ""Welcome to"" ends up in the ckan.po files for translation. This makes the phrase untranslatable in languages that have a different word ordering than English where the site title needs to be somewhere other than at the end of the sentence." 1345627079000000 1345627233000000 1626 enhancement dread dread ckan-sprint-2012-01-23 closed fixed 'About CKAN' page update thedatahub.org/about contains info that is very general to the CKAN and really quite technical. The text should be changed to be both specific to thedatahub.org and provide the context in a non-technical way. It should be easy to customise the About page to be appropriate for say new-york.ckan.net - a bit of info about who runs it, plus the general stuff about CKAN powering it and it was written by OKF to further open data. 1326205236000000 1326215877000000 2407 enhancement amercader markw ckan-v1.9 new 'Access denied' message is unhelpful "This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied': http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html The problem was he didn't have access to the group. A message like 'You don't have access to the group ""LODcloud""' would have been clearer." 1337612500000000 1340633440000000 2692 defect aron.carroll shevski demo phase 2 closed fixed 'add your own data' link in no search results wrong "the ""add your own data"" link in ""Try another search term, browse the datasets below or add your own data."" here http://s031.okserver.org:2375/dataset?q=stuff doesn't take you to add a dataset, just back to the search/datasets page" 1342540803000000 1342617328000000 2966 enhancement seanh ckan 2.0 new 'Add' button text is wrong when editing organization members e.g. if I just changed a member's capacity I am not adding anything 'save' is better maybe 1350296149000000 1350296149000000 1710 enhancement dread dread ckan-sprint-2012-02-06 closed fixed 'Announce' email list for thedatahub "We want to make email announcements to users of thedatahub.org. The most simple way to do this is to get a dump of the email addresses and subscribe/invite them to a mailman list. Different ways to do it: * 'paster dump-users-csv' creates a CSV file. Apache serves up the directory using password protection. Is this secure enough? * CKAN admin pages provides a button to produce the CSV and download it. A config option may be necessary to enable this (disabled by default). Something more advanced in the future could be: * CKAN tells mailman about new subscribers using the mailman web interface. Could be brittle. * Mailman doesn't have an Internet API, but it does have a python interface. So CKAN could tell mailman about new subscribers using a python proxy. There may be one out there, or maybe easy to do. But annoying to have to maintain another process running on machine. * CKAN sends out the emails itself. A sysadmin is given a form. Users have an opt in/out option in their account. But receiving emails is a sensitive thing so it would need to be right first time, and there are several risks to this approach. " 1327579259000000 1327583922000000 1042 task dread dread closed fixed 'Ckanext' split-up "It's not good to have ckanext doing lots of different things with different dependencies. Split it off into: * ckanext-importlib etc. And then deprecate the ckanext repo itself." 1300293907000000 1300969865000000 1308 defect shevski dread closed fixed 'create package' user case failure On the CKAN home page there is no suggestion that you can add or register a dataset. The 'Register a dataset' option has been removed at some point! (only visible to users that have registered - why would they do that when the benefits aren't described?) 1315219500000000 1323174312000000 983 defect dread dread ckan-v1.3 closed fixed 'db upgrade' creates system priviledges (should leave priviledges alone) 1297518265000000 1297773407000000 1346 defect dread dread ckan-sprint-2011-10-28 closed fixed 'get_authorization_group_dict not found' exception "This stems from a feature to allow you to prefill fields in the authorizationgroup creation form. e.g. http://ckan.net/authorizationgroup/new?name=science It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it." 1316081974000000 1316094632000000 824 enhancement dread dread closed fixed 'Getdata' moved out of CKAN "Data importers should use API instead of object model directly: * saves load on server * much better error checking and authentication * eating our own dogfood (tests the API well)" 1290595507000000 1290595559000000 2265 enhancement dread ckan-future new 'More Like This' for a dataset "When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar. SOLR has a feature for finding documents similar to a particular document, called 'More Like This'. We would like this for DGU." 1332865220000000 1339771350000000 1440 defect dread dread ckan-sprint-2011-11-07 closed fixed 'paster db load' doesn't update search index "{{{paster db load ckan.pgdump}}} will load in the database but not update the search index, no we're using SOLR. = implementation = We can't tack on 'rebuild' to this command, because it needs a 'db upgrade' first. I'm tempted to make 'db load' do the upgrade and rebuild all in one command." 1320232415000000 1320235084000000 1318 defect dread dread ckan-sprint-2011-10-28 closed fixed 'Register user' doesn't log you in "When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to ""log in"" and ""register"". Also the ""create package"" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected." 1315415783000000 1315569362000000 1827 enhancement dread ckan-backlog new 'Register' link should be hidden if you not allowed to register "{{{ I have just deny visitors the create-user permission: sudo -u ckanstd paster --plugin=ckan roles deny reader create-user -c /etc/ckan/std/std.ini sudo -u ckanstd paster --plugin=ckan roles deny anon_editor create-user -c /etc/ckan/std/std.ini and after restarting, the register link is *not* hidden, but now when you access the register page, it shows you this message ""Unauthorized to create a user"" (when not logged in). But anyway that is an improvement. }}}" 1329924939000000 1339773730000000 1493 defect dread dread ckan-sprint-2011-12-05 closed fixed 'search-index rebuild/clear' doesn't work if no ckan.site_id "You can't delete things from the SOLR search index if the ckan.site_id and ckan.site_url are blank. Should assert that one of these are set up." 1322484422000000 1324474360000000 542 task wwaites rgrp closed fixed 'Show and Tell' page customization using cookie + js "To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: http://www.ibm.com/developerworks/web/library/wa-rails2/ In the first place we just need to apply this to: * You are logged in as * Package/group edit/authz tabs" 1283244432000000 1283278942000000 2847 defect toby shevski demo phase 3 closed fixed 'show inactive sources' checkbox displaying incorrectly "On harvest page: http://s031.okserver.org:2375/harvest the checkbox appears above the text and not clear what it refers to/ Beter to relocate to the left (as on publicdata.eu) see screenshots attached. " 1344864322000000 1345023769000000 1586 enhancement toby dread ckan-v1.7 closed duplicate 'Sort by' field for package search "We default to sort packages search results by ""search rank"", but we can change it to be ""name"", ""title"", ""metadata_modified"" date, ""metadata_created"" date. It works in the API now, so let's have (say) a drop-down in the Web UI for this. Follows on from #191." 1324479588000000 1337079419000000 2607 defect seanh ckan-backlog assigned 'Upload a file' appears on resource form when storage not enabled "if the user tries to upload a file they will get ""Failed to get credentials for storage upload. Upload cannot proceed"" Maybe add a test for it this time, this bug has appeared and reappeared before" 1340803808000000 1346663383000000 101 enhancement rgrp rgrp closed fixed (Improved) Download support " 1. Given a url download from that url to disk 2. Given a package use download url to download to disk Once on disk should uncompress (if necessary). == Details == * Should download either to tmp directory (defaulted in config) or a specified path on disk * In download would like to support as many types of target urls as possible: 1. Single files (usually compressed filesets but could be individual files) 2. Index pages listing files to download 3. Revision control repositories 4. ... This will obviously be a significant amount of work. Suggest: 1. Start off just supporting the first option. 2. Have something like a plugin system to handle new target types 3. Reuse existing work (e.g. easy_install and pip handle standard vcs such as svn, hg, git etc) Cost: 1d " 1249988866000000 1267648607000000 1098 task dread dread ckan-v1.4-sprint-6 closed fixed --ckan-migration tests not initialised correctly Only tests with failing --ckan-migration fail, due to authz not being initialised. 1303377336000000 1303406017000000 2411 enhancement toby markw ckan-future new .ini file is a mess The .ini file needs cleaning up in assorted ways which Toby will expand upon. 1337686900000000 1338210872000000 2684 enhancement aron.carroll toby demo phase 2 closed fixed .js leaving forms warning "I think it would be nice to have all form pages monitor their forms - excluding search ones - and if a field has changed then warn the user that they have not saved the form data but are leaving the page. This would help with things like changing resources on the add dataset resource form etc." 1342510477000000 1342539637000000 1316 defect kindly dread closed fixed /api/action uncaught exception if missing necessary parameter "{{{ curl http://ckan.net/api/action/package_show -d '{""name"": ""uk-quango-data""}' }}} This gives 500 error due to KeyError (it looked for the 'id' param), rather than 400." 1315397224000000 1338193724000000 2868 enhancement ross ross closed fixed 1.8 SOLR error "From: Date: Wed, Aug 15, 2012 at 8:34 AM Subject: WebApp Error: : SOLR returned an error running query: {'sort': 'metadata_modified desc', 'fq': u'groups:""welcome"" capacity:""public"" +site_id:""www.ckan.net"" +state:active', 'facet.mincount': 1, 'rows': 21, 'facet.limit': '50', 'facet': 'true', 'q': '*:*', 'start': -20, 'wt': 'json', 'fl': 'id data_dict'} Error: ""'start' parameter cannot be negative"" To: ckan-sysadmin@okfn.org, kindly@gmail.com URL: http://thedatahub.org/feeds/group/welcome.atom?page=0 Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:155 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:131 in __call__ << r'\1', oldpath) response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:239 in __call__ << try: res = WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> res = WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.feed:180 in group << data_dict['fq'] = 'groups:""%s""' % id item_count, results = _package_search(data_dict) navigation_urls = self._navigation_urls(params, >> item_count, results = _package_search(data_dict) Module ckan.controllers.feed:57 in _package_search << # package_search action modifies the data_dict, so keep our copy intact. query = get_action('package_search')(context, data_dict.copy()) return query['count'], query['results'] >> query = get_action('package_search')(context, data_dict.copy()) Module ckan.logic.action.get:1130 in package_search << query = search.query_for(model.Package) query.run(data_dict) for package in query.results: >> query.run(data_dict) Module ckan.lib.search.query:350 in run << except SolrException, e: raise SearchError('SOLR returned an error running query: %r Error: %r' % (query, e.reason)) try: data = json.loads(solr_response) >> (query, e.reason)) SearchError: SOLR returned an error running query: {'sort': 'metadata_modified desc', 'fq': u'groups:""welcome"" capacity:""public"" +site_id:""www.ckan.net"" +state:active', 'facet.mincount': 1, 'rows': 21, 'facet.limit': '50', 'facet': 'true', 'q': '*:*', 'start': -20, 'wt': 'json', 'fl': 'id data_dict'} Error: ""'start' parameter cannot be negative"" CGI Variables CKAN_CURRENT_URL '/feeds/group/welcome.atom?page%3D0' CKAN_LANG 'ja' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' PATH_INFO '/feeds/group/welcome.atom' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/ja/feeds/group/welcome.atom' QUERY_STRING 'page=0' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '33139' REQUEST_METHOD 'GET' REQUEST_URI '/ja/feeds/group/welcome.atom?page=0' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80
\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1345016040.884951, '_creation_time': 1345016040.884951} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script '' mod_wsgi.input_chunked '0' mod_wsgi.listener_host '' mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (, '') paste.parsed_dict_querystring (MultiDict([('page', '0')]), 'page=0') paste.parsed_querystring ([('page', '0')], 'page=0') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'group', 'controller': u'feed', 'id': u'welcome'} repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([('page', '0')]), 'page=0') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgi.version (1, 1) wsgiorg.routing_args (, {'action': u'group', 'controller': u'feed', 'id': u'welcome'}) " 1345114696000000 1345124503000000 2870 enhancement seanh ross new 1.8 tag_list not defined "From: Date: Thu, Aug 16, 2012 at 3:20 AM Subject: WebApp Error: : ""tag_list"" not defined To: ckan-sysadmin@okfn.org, kindly@gmail.com URL: http://ckan.net/dataset/uk-postboxes Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:155 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:131 in __call__ << r'\1', oldpath) response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:239 in __call__ << try: res = WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> res = WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.package:322 in read << template = template[:template.index('.') + 1] + format return render(template, loader_class=loader) def comments(self, id): >> return render(template, loader_class=loader) Module ckan.lib.base:153 in render << try: return cached_template(template_name, render_template, loader_class=loader_class) except ckan.exceptions.CkanUrlException, e: raise >> loader_class=loader_class) Module pylons.templating:249 in cached_template << return content else: return render_func() >> return render_func() Module ckan.lib.base:102 in render_template << if loader_class == NewTextTemplate: return literal(stream.render(method=""text"", encoding=None)) return literal(stream.render(method=method, encoding=None, >> return literal(stream.render(method=""text"", encoding =None)) Module genshi.core:183 in render << method = self.serializer or 'xml' generator = self.serialize(method=method, **kwargs) return encode(generator, method=method, encoding=encoding, out=out) def select(self, path, namespaces=None, variables=None): >> return encode(generator, method=method, encoding=encoding, out =out) Module genshi.output:57 in encode << _encode = lambda string: string if out is None: return _encode(''.join(list(iterator))) for chunk in iterator: out.write(_encode(chunk)) >> return _encode(''.join(list(iterator))) Module genshi.output:569 in __call__ << def __call__(self, stream): strip_markup = self.strip_markup for event in stream: if event[0] is TEXT: data = event[1] >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module ckanext.googleanalytics.plugin:93 in download_adder << [downloaded %s times]''' count = None for mark, (kind, data, pos) in stream: if mark and kind == START: href = data[1].get('href') >> for mark, (kind, data, pos) in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1175 in __call__ << """""" callable_value = hasattr(self.value, '__call__') for mark, (kind, data, pos) in stream: if mark is ENTER: if callable_value: >> for mark, (kind, data, pos) in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.template.base:605 in _include << from genshi.template.loader import TemplateNotFound for event in stream: if event[0] is INCLUDE: href, cls, fallback = event[1] >> for event in stream: Module genshi.template.base:565 in _flatten << elif kind is EXPR: result = _eval_expr(data, ctxt, vars) if result is not None: # First check for a string, otherwise the iterable test >> result = _eval_expr(data, ctxt, vars) Module genshi.template.base:277 in _eval_expr << if vars: ctxt.push(vars) retval = expr.evaluate(ctxt) if vars: ctxt.pop() >> retval = expr.evaluate(ctxt) Module genshi.template.eval:178 in evaluate << __traceback_hide__ = 'before_and_this' _globals = self._globals(data) return eval(self.code, _globals, {'__data__': data}) >> return eval(self.code, _globals, {'__data__': data}) Module ?:51 in <<
  • Tags

    ${tag_list(c.pkg_dict.get('tags', ''))}
  • >> ${tag_list(c.pkg_dict.get('tags', ''))} Module genshi.template.eval:309 in lookup_name << val = BUILTINS.get(name, val) if val is UNDEFINED: val = cls.undefined(name) return val >> val = cls.undefined(name) Module genshi.template.eval:410 in undefined << """"""Raise an ``UndefinedError`` immediately."""""" __traceback_hide__ = True raise UndefinedError(key, owner=owner) >> raise UndefinedError(key, owner=owner) UndefinedError: ""tag_list"" not defined CGI Variables CKAN_CURRENT_URL '/dataset/uk-postboxes' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CKAN_PAGE_CACHABLE True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,text/plain,text/xml' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'vi,en-us;q=0.7,en;q=0.3' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'coccoc/1.0 ()' PATH_INFO '/dataset/uk-postboxes' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/dataset/uk-postboxes' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '47065' REQUEST_METHOD 'GET' REQUEST_URI '/dataset/uk-postboxes' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
    Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1345083636.079552, '_creation_time': 1345083636.079552} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script '' mod_wsgi.input_chunked '0' mod_wsgi.listener_host '' mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (, '') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'} repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgi.version (1, 1) wsgiorg.routing_args (, {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'}) " 1345114732000000 1345114732000000 2866 enhancement ross ross closed fixed 1.8 template error " URL: http://thedatahub.org/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8 Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:155 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:131 in __call__ << r'\1', oldpath) response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:239 in __call__ << try: res = WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> res = WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.user:409 in perform_reset << c.reset_key = request.params.get('key') if not mailer.verify_reset_link(user_obj, c.reset_key): h.flash_error(_('Invalid reset key. Please try again.')) abort(403) >> if not mailer.verify_reset_link(user_obj, c.reset_key): Module ckan.lib.mailer:100 in verify_reset_link << if not user.reset_key or len(user.reset_key) < 5: return False return key.strip() == user.reset_key >> return key.strip() == user.reset_key AttributeError: 'NoneType' object has no attribute 'strip' CGI Variables CKAN_CURRENT_URL '/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%253Db4c2d03fa8' CKAN_LANG 'it' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' PATH_INFO '/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/it/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d' QUERY_STRING 'key%3Db4c2d03fa8' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '54634' REQUEST_METHOD 'GET' REQUEST_URI '/it/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
    Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1345066159.520708, '_creation_time': 1345066159.520708} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script '' mod_wsgi.input_chunked '0' mod_wsgi.listener_host '' mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (, '') paste.parsed_dict_querystring (MultiDict([('key=b4c2d03fa8', '')]), 'key%3Db4c2d03fa8') paste.parsed_querystring ([('key=b4c2d03fa8', '')], 'key%3Db4c2d03fa8') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'perform_reset', 'controller': u'user', 'id': u'3086e91c-fe09-4a98-92e1-19de67a9ac9d'} repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([('key=b4c2d03fa8', '')]), 'key%3Db4c2d03fa8') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgi.version (1, 1) wsgiorg.routing_args (, {'action': u'perform_reset', 'controller': u'user', 'id': u'3086e91c-fe09-4a98-92e1-19de67a9ac9d'}) " 1345114641000000 1346670324000000 2865 enhancement ross ross closed fixed 1.8 unicode error " URL: http://thedatahub.org/el/tag/jutatt%C3%A1sok Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:155 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:131 in __call__ << r'\1', oldpath) response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:239 in __call__ << try: res = WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> res = WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.template:30 in view << """""" try: return render(url) except TemplateNotFound: if url.endswith('.html'): >> return render(url) Module ckan.lib.base:153 in render << try: return cached_template(template_name, render_template, loader_class=loader_class) except ckan.exceptions.CkanUrlException, e: raise >> loader_class=loader_class) Module pylons.templating:249 in cached_template << return content else: return render_func() >> return render_func() Module ckan.lib.base:95 in render_template << template = globs['app_globals'].genshi_loader.load(template_name, cls=loader_class) stream = template.generate(**globs) >> cls=loader_class) Module genshi.template.loader:223 in load << loadfunc = directory(loadfunc) try: filepath, filename, fileobj, uptodate = loadfunc(filename) except IOError: continue >> filepath, filename, fileobj, uptodate = loadfunc(filename) Module genshi.template.loader:286 in _load_from_directory << def _load_from_directory(filename): filepath = os.path.join(path, filename) fileobj = open(filepath, 'U') mtime = os.path.getmtime(filepath) def _uptodate(): >> fileobj = open(filepath, 'U') UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 73: ordinal not in range(128) CGI Variables CKAN_CURRENT_URL '/el/tag/jutatt%C3%A1sok' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CKAN_PAGE_CACHABLE True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' PATH_INFO '/el/tag/jutatt\xc3\xa1sok' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/el/tag/jutatt\xc3\xa1sok' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '49520' REQUEST_METHOD 'GET' REQUEST_URI '/el/tag/jutatt%c3%a1sok' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
    Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1345113298.778472, '_creation_time': 1345113298.778472} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script '' mod_wsgi.input_chunked '0' mod_wsgi.listener_host '' mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (, '') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'url': u'el/tag/jutatt\xe1sok', 'action': u'view', 'controller': u'template'} repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgi.version (1, 1) wsgiorg.routing_args (, {'url': u'el/tag/jutatt\xe1sok', 'action': u'view', 'controller': u'template'}) " 1345114613000000 1346670331000000 2867 enhancement ross new 1.8 url_for error "From: Date: Thu, Aug 16, 2012 at 10:30 AM Subject: WebApp Error: : url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6 To: ckan-sysadmin@okfn.org, kindly@gmail.com URL: http://ckan.net/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:155 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:131 in __call__ << r'\1', oldpath) response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:239 in __call__ << try: res = WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> res = WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.storage:190 in file << return fapp(request.environ, self.start_response) else: h.redirect_to(file_url) >> h.redirect_to(file_url) Module ckan.lib.helpers:57 in redirect_to << if are_there_flash_messages(): kw['__no_cache__'] = True return _redirect_to(url_for(*args, **kw)) def url(*args, **kw): >> return _redirect_to(url_for(*args, **kw)) Module ckan.lib.helpers:79 in url_for << # fix ver to include the slash kw['ver'] = '/%s' % ver my_url = _routes_default_url_for(*args, **kw) kw['__ckan_no_root'] = no_root return _add_i18n_to_url(my_url, locale=locale, **kw) >> my_url = _routes_default_url_for(*args, **kw) Module routes.util:265 in url_for << if not isinstance(url, str) and url is not None: raise GenerationException(""url_for can only return a string, got "" ""unicode instead: %s"" % url) if url is None: raise GenerationException( >> ""unicode instead: %s"" % url) GenerationException: url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6 CGI Variables CKAN_CURRENT_URL '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'utf-8;q=0.7,iso-8859-1;q=0.2,*;q=0.1' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)' PATH_INFO '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '37236' REQUEST_METHOD 'GET' REQUEST_URI '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
    Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1345109433.273328, '_creation_time': 1345109433.273328} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script '' mod_wsgi.input_chunked '0' mod_wsgi.listener_host '' mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (, '') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'} repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgi.version (1, 1) wsgiorg.routing_args (, {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'}) " 1345114669000000 1345128279000000 1772 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed fixed 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST] 1328539902000000 1329733962000000 1773 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD] 1328540130000000 1329733534000000 1774 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed 22. Mixed Licenses MUST PHASE 2 1328540153000000 1329733550000000 1770 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed wontfix 3. Themes/Taxonomy as well as tags WONT PHASE 2 1328539834000000 1329733935000000 1771 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed 32. Browse by Tags PHASE 1 MUST 1328539862000000 1329733648000000 1376 enhancement kindly kindly ckan-sprint-2011-10-10 closed fixed 404 raised when approving package Regression found by updating datacatalogs. 1318088589000000 1318279651000000 1093 defect dread dread ckan-v1.4-sprint-6 closed fixed 500 errors on GET to api/rest/licenses CKAN gets its license list from a license service, which can be a local file, but is often the http://licenses.opendefinition.org/2.0/ckan_original server. This server is currently flakey, but I think we only request the list on start up. The problem is we query it much more often than required. It is queried for every request to api/rest/licenses, and we are returning lots of 500 errors when the license server is timing out. 1302862261000000 1302865470000000 368 defect anonymous closed wontfix 500 Server error when creating package " I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response: ""publish: New Zealand Coastline (new-zealand-coastline)"" opening connection to nz.ckan.net... opened <- ""POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n"" <- ""{\""name\"":\""new-zealand-coastline\"",\""title\"":\""New Zealand Coastline\""}"" -> ""HTTP/1.1 500 Internal Server Error\r\n"" -> ""Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n"" -> ""Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n"" -> ""Vary: Accept-Encoding\r\n"" -> ""Content-Type: text/html; charset=iso-8859-1\r\n"" -> ""Cache-Control: proxy-revalidate\r\n"" -> ""Content-Length: 640\r\n"" -> ""Connection: close\r\n"" -> ""\r\n"" reading 640 bytes... -> ""\n\n500 Internal Server Error\n\n

    Internal Server Error

    \n

    The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.

    \n

    Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.

    \n

    More information about this error may be available\nin the server error log.

    \n
    \n
    Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80
    \n\n"" read 640 bytes Conn close The server should not respond with a 500 error if the package was created. " 1279622462000000 1291831811000000 2601 defect ross seanh closed fixed 500 Server Error when editing datasets with organizations plugins enabled "Add the organizations and organizations_dataset plugins to ini file, run ckan, login, create an organization, create a dataset and put in organization, edit the dataset and set organization to none, try to edit the dataset again (you must be sysadmin to do this) and get a 500: {{{ Error - : [] has no member named ""get"" URL: http://127.0.0.1:5000/dataset/edit/fooset File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 162 in __call__ app_iter = self.application(environ, sr_checker) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__ return self.wrap_app(environ, session_start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 221 in __call__ res = WSGIController.__call__(self, environ, start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__ response = self._dispatch_call() File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call return func(**args) File '/home/seanh/Projects/ckan/ckan/ckan/controllers/package.py', line 499 in edit extra_vars=vars) File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 148 in render return cached_template(template_name, render_template, loader_class=loader_class) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template return render_func() File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 101 in render_template return literal(stream.render(method=method, encoding=None, strip_whitespace=True)) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/core.py', line 183 in render return encode(generator, method=method, encoding=encoding, out=out) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 57 in encode return _encode(''.join(list(iterator))) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 339 in __call__ for kind, data, pos in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 670 in __call__ for kind, data, pos in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 771 in __call__ for kind, data, pos in chain(stream, [(None, None, None)]): File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 586 in __call__ for ev in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/core.py', line 288 in _ensure for event in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 605 in _include for event in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/markup.py', line 327 in _match for event in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 545 in _flatten for kind, data, pos in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/directives.py', line 169 in _generate attrs = _eval_expr(self.expr, ctxt, vars) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 277 in _eval_expr retval = expr.evaluate(ctxt) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/eval.py', line 178 in evaluate return eval(self.code, _globals, {'__data__': data}) File '/home/seanh/Projects/ckan/ckan/ckanext/organizations/templates/organization_package_form.html', line 110 in File '/home/seanh/Projects/ckan/ckan/ckan/config/environment.py', line 248 in genshi_lookup_attr val = cls.undefined(key, owner=obj) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/eval.py', line 410 in undefined raise UndefinedError(key, owner=owner) UndefinedError: [] has no member named ""get"" }}}" 1340722754000000 1341291379000000 1769 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed fixed 6. Publisher Registration Improvements MUST PHASE 1 1328539789000000 1329737597000000 2420 enhancement toby toby closed fixed @depreciated decorator "create a depreciated decorator to wrap functions we intend to remove logs a warning maybe check that 'depreciated' is included in the docstring too" 1337858144000000 1337950422000000 1055 defect dread dread ckan-v1.4-sprint-4 closed fixed @search_related tests not running "Tests marked decorated ""@search_related"" should only be run against postgresql, but in fact they don't get run at all. " 1300985228000000 1300992395000000 448 story thejimmyg dread closed invalid [Assist with] editing packages through Drupal 1282299478000000 1294417061000000 1599 enhancement rgrp rgrp ckan-sprint-2012-01-09 closed fixed [contrib] Simple embeddable dataset count widget (esp for group count) "Simple embeddable widget for use on 3rd party sites showing dataset counts for a given search query. Have a specific version just for groups. * Simple group count widget in JS for embedding in wordpress and elsewhere * Requested by several people (e.g Guo Xu from Econ working group). Already have something like this in CKAN JS for doing an embeddable search box. * All you need to do is do a dataset query over the API e.g. http://thedatahub.org/api/search/dataset?groups=economics and then embed in some html! Estimate: 30m (for someone who knows their jquery). " 1325246358000000 1325555201000000 1523 enhancement icmurray icmurray closed invalid [super] "Form refactor. New create/edit package form wizard for DGU. See attached spec." 1323172661000000 1323172904000000 1515 enhancement seanh seanh ckan-v1.7 closed fixed [super] Activity Streams "We want to add ""activity streams"" (chronological lists of activities) to CKAN. Activity streams standard: http://activitystrea.ms/ Super branch where the feature was initially developed (this has now been merged into master): https://github.com/okfn/ckan/tree/feature-1515-activity-streams Etherpad: http://ckan.okfnpad.org/notifications (most relevant parts inlined here) Friedrich's mockup: http://datahub.pudo.org/pudo == Tasks == ~~Strikethrough~~ means the task is done. * ~~New domain objects Activity and ActivityStream [2d] #1298~~ * Migrate existing data into activity streams model [3.5d] #1510 * ~~Generate activities when objects are saved [4d] #1298~~ * ~~Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] #1511~~ * ~~API call for getting a user's public activity stream as JSON [0.5d] #1495~~ * ~~API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] #1494~~ * ~~Render a user's activity stream on her user page [3d] #1496~~ (needs review/page design work) * ~~Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.~~ * ~~Logic functions for getting activity streams for packages, groups, and users.~~ * ~~API calls for getting activity streams of packages, groups, and users~~ as JSON. * ~~Rendering activity streams for packages and groups etc. into their pages.~~ (done but disabled pending page design work) * ~~API calls for inserting events into activity streams.~~ (Still need to handle activity details in the API, ticketed). * Conform to activitystrea.ms standard * Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page. * Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams * Email notifications * Extension point for extensions to register themselves as renderers for different types of activity * Deploy to UAT [0.5d] * Docs and announce [1d]" 1323105048000000 1338204149000000 833 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-6 closed fixed [super] Administrative dashboard extension "Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview. Possible features: * Purge revisions (or sets of revisions) and purge objects #1076 * Set roles for users #1075 * Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages) * WONTFIX * Overview of activity * WONTFIX - already have revision log Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/. https://bitbucket.org/okfn/ckanext-admin === Tickets === * ticket:1031 - user autocomplete api in ckan === Notes === Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group): {{{ # first remove permissions from roles # this is hacky but have to do it because we hardcode assignment of # role permissions on package on package create (see model/authz.py) paster roles deny editor edit paster roles deny editor create-authorization-group paster roles deny editor create-group paster roles deny editor create-package paster roles deny reader create-package # make superuser group # create authz group administrators / Administrators (if not exists) paster rights make agroup:administrators admin system }}}" 1290765921000000 1303236364000000 1792 enhancement toby kindly ckan-sprint-2012-03-05 closed fixed [super] api to use same validation and logic as wui forms. The api currently uses the default validation schemas regardless of the form you specified on the front end. There should be a way to post through the api using the same validation rules. 1329146487000000 1333037710000000 1240 enhancement kindly rgrp ckan-backlog assigned [super] API v4 "(Just creating this ticket as somewhere to keep notes) * Decide on REST api versus action API * Do we want to support both? * Tidying * Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...) * extras merged into normal field list in package * Get rid of /rest/ so just have api/v1/package * Get rid of separation of search api from 'rest' api * Propose that GET on REST index is search e.g. /package/?q=... * This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned) * Resource read/write in API (separate from package) * Does this need authorization work? * user/account API - read/write * Remove autocomplete -- can just use search * Do not worry about backwards compat as should only be used in our js (if others using it too bad!) " 1311525660000000 1325473312000000 954 enhancement kindly rgrp ckan-v1.5 closed fixed [super] API version 3 "Child tickets: * #1107 Move package autocomplete from package controller and move to API * #1087 version and contact info api call Move to a format that has a separate responseHeader and response. === A standard package response === {{{ { responseHeader: { status: 0, } response: {package-dict} } }}} On error: {{{ { responseHeader: { status: {err-code}, error: 'message' } response: none } }}} === A search query === Based directly on solr. {{{ { responseHeader: { status: 0, } response: { numFound: 5, start: 0 docs: [ ] } } }}} == Issues == This is a breaking change for clients == References == * http://wiki.apache.org/solr/SolJSON * http://www.flickr.com/services/api/response.json.html " 1296811899000000 1320142744000000 1065 enhancement zephod johnlawrenceaspden ckan-v1.6 closed fixed [super] Change Authorization System "Child tickets * #1198 Publisher hierarchy * #1050 Authz lib improvement and refactor of ckan/lib/authztool.py * #1004 Group creation instructions missing * #1099 Strange interactions between two browsers while playing with authz groups * #1115 can have two authzgroups with the same name * #1133 command line rights manipulation doesn't work * #1138 minor navigations behave inconsistently Old ticket description: 1. Change name of AuthzGroup to UserGroup to reflect what it is for 2. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones. * Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system. * Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages) 3. Change UserGroups so that they can have a hierarchical structure, == More info on Hierarchy change == e.g. UserGroup NHS contains the User nhsysadmin, as well as the UserGroups SURREY and BERKS, which themselves contain users. One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not. Each Group has permissions over various objects. A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively. Algorithm: possible(user, action, package): if user has permission for action on package or any of have that permission or any of his groups group-children (but not user-children), and so on recursively have the permission. " 1301508331000000 1324550041000000 1096 defect rufuspollock pudo ckan-future new [super] CKAN Hosted "Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas: * introduce model.Site and c.site * site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI) * Subdomain detector to activate sites. * use site in Authorizer instead of System, have a NullSite for global things * allow cross-site search * packages are in a list of sites, m:n rather than 1:n * list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally. " 1303235062000000 1339774484000000 1593 enhancement icmurray amercader ckan-sprint-2012-04-16 closed fixed [super] Create feeds from search results "Recent changes (see #191 and #1498) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy. Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest. Apart from the general feed with changes on the whole instance, other different ""pre-set"" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags: {{{ http:///feed/group/{id}.{format:rss|atom} http:///feed/tag/{id}.{format:rss|atom} }}} These could be shown on the group/tag page, both on the section, so browsers can autodiscover them, and on the UI, with a message like ""Subscribe to this group/tag"". Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on {{{ http:///feed/custom.{format:rss|atom}?res_format=api%2Fsparql http:///feed/custom.{format:rss|atom}?q=tags:animal* }}} On this case a message ""Subscribe to these search results"" would appear. Note that a lot of work on this front needs to be done for the IATI registry project (see #1590) Other tickets related: * #1592: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now. " 1324489115000000 1334566668000000 1190 enhancement rgrp rgrp ckan-v1.5 closed fixed [super] CREP 0004 Data API and Data Processing System "For some time (e.g. 1y+!) we have known that we want to integrate some kind of datastore / data processing system with CKAN. We've had a CREP in progress on this for some months (may copy that here at some point): http://wiki.ckan.org/CEP0004 We can distinguish 3 modules that are needed: 1. ""Webstore"": A datastore with dataapi - #1208 Suggestion is this would be sqlite based with a simple sql based API. http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query} 2. Automated conversion of suitable resources into datastore upon resource creation so that e.g. they are accessible via the API. #1398 3. A data processing system which utilizes this datastore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) - #1432 == More info == * Overview diagram: https://docs.google.com/drawings/d/1XK7dcpFXNlMzVFgLPYPZUXOsFRPzgCEvS-w7FcvydEQ/edit?hl=en_GB" 1308227611000000 1323270522000000 1602 enhancement rgrp rgrp ckan-v1.6 closed fixed [super] Data Previewer / Viewer v2 "We already have first pass of Data Preview that was done as part of #1357. This is about v2 of this. Two main components: * A new version of the separate Data Explorer library (now called Recline Data Explorer) - #1446 '''Est: 10-15d''' * Why: * Cleaner, backbone based solution. * Provides editing support if wanted. * Collaboration with Max Ogden * Integrate this into CKAN (into resource view page) - see #1630 '''Est: 1d''' * Improvements to Data Viewer / Previewer system in CKAN itself - '''Est: 2-3d''' * ~~(Update Data Preview-er (View-er) to support images ...)~~ * ~~(WONTFIX here - (?) Update Data Preview-er to support geodata - #1151)~~ " 1325267544000000 1328175719000000 2283 enhancement rgrp ckan-v1.8 closed wontfix [super] Data Previewer / Viewer v3 " * Basic geo map * Facets * Filtering * Pivot table support (?) * Export data (?) * Save data query and share * More and better graphing * Embedding! - /resource/../embed User stories and analysis: http://ckan.okfnpad.org/feature-2283-data-viewer-v3" 1334138341000000 1340623843000000 1432 enhancement rgrp ckan-backlog new [super] Data processing system for CKAN and Webstore "Super ticket: #1190 A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)" 1320142747000000 1339774041000000 1607 enhancement icmurray dread ckan-v1.7 closed fixed [super] Data.gov.uk Maintenance Refactor and UKLP Development Support "A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU. Design doc: https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB The design doc is the authoritative source of tasks. This ticket acts as a synopsis. Current sprint (2012-02-20): * 7a. ~~Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1~~ * 6. ~~Publisher Registration Improvements MUST PHASE 1~~ * 3. ~~Themes/Taxonomy as well as tags WONT PHASE 2~~ * 32. ~~Browse by Tags PHASE 1 MUST~~ * 12. ~~Edit/Delete for Harvested Datasets MUST URGENT PHASE 1~~ * 13. ~~Provider labelling MUST PHASE 1 [UKLP #14 MUST]~~ * 17. ~~Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]~~ * 22. ~~Mixed Licenses MUST PHASE 2~~ Plus, brought forward from last sprint: * ~~Publisher hierarchy support~~ * ~~Improved dataset creation tools (form) - #1525~~ * ~~Edit/delete for harvested datasets~~ Overview of tasks: see the design doc." 1325503348000000 1337159969000000 1327 enhancement rgrp ckan-v1.6 closed duplicate [super] Dataset Archiving "Split out of #852. Automated archiving of datasets (related to QA). ''' Automated archiving using worker process ''' * #890 - Timed actions in ckanext-queue * #891 - Resource download worker daemon * #892 - Make stored data available in WUI" 1315821490000000 1320662446000000 852 enhancement rgrp wwaites ckan-v1.5 closed fixed [super] Dataset upload and archiving "3 (4) use cases: (With and without access to auth keys.) 1. upload form on ckan (by default local system has auth key for storage) 2. datapkg upload (client) (by default without key) (plumbing done) 3. background job archiving downloads (by queue client) Most important are: 1. Upload form on ckan.net (with access to keys) 2. API for getting credentials for upload from ckan.net api (without access to keys). Aside: Once 2 was done we could deprecate storage.ckan.net === Sub-tickets === ''' Required infrastructure ''' * #826 - Resource 'extra' fields * #358 - Resources in REST API * #235 - Resource format normalization and detection ''' User upload via Web UI ''' * #877 - File upload in WUI * #878 - Integrate file upload with workflow around package resources ''' User upload via CLI/datapkg ''' * #853 - plumbing for credential-less rest upload * #879 - Storage auth API Also related: * #405 - Retrieval options for package resources === More Info === * Upload/storage discussion: http://ckan.okfnpad.org/storage * For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/ " 1291722993000000 1315821628000000 1525 enhancement icmurray icmurray ckan-v1.7 closed fixed [super] DGU package form refactor "Form refactor. New create/edit package form wizard for DGU. Etherpad: http://ckan.okfnpad.org/dgu-package-form Tasks: - #1526 : Simple one-page form. - #1527 : Tab-ify the form using JS. No validation between steps. - #1541 : Setup server for the DGU form-refactor. - #1537 : Package create form wizard - #1538 : ~~Store partially-filled in package form data as a draft.~~ - #1539 : ~~Package forms auto-save~~ How the rest of the work is carried out is up for discussion, as it depends on the approach taken. Whether to continue with validation using AJAX queries, or whether to break the form down into multi pages. [https://docs.google.com/spreadsheet/ccc?key=0Av2RfDDpj67zdGpSdFlKVjNrQmFXV3lHbGM3ZFlDdnc&hl=en_US#gid=1 Google doc outlining design(s)] " 1323172839000000 1337159836000000 1211 enhancement kindly kindly ckan-v1.5 closed fixed [super] Drupal integration "Sub-tasks / tickets: * #1335 Action API - access to any logic layer function * Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict. * Add package_create_validate & package_update_validate logic functions * Allow extensions to provide logic layer functions (IActions) " 1309958442000000 1315948703000000 1508 enhancement ross rgrp ckan-v1.6 closed fixed [super] Finalize and deploy some of our major (new) features as of Dec 2011 "This ticket is about finalizing, then deploying (to http://thedatahub.org/), and finally announcing some of recent major new features such as: * Deploy wordpresser extension for thedatahub (so we can start blogging) - #1533 - WONTFIX * ~~(Deploy new Resource and Dataset pages (invite comments))~~ * Get thedatahub geo-ified i.e. ckanext-spatial - #1601 (?) * Any additional dev that needs to be done? (e.g. geo in search ...?) * ~~(Deploy / re-enable ckanext-googleanalytics on thedatahub - #1451 (more stuff #1566))~~ * Rename the extension to ckanext-analytics? * Custom forms (and use of 'dataset types') - Ticket (??) * ~~(Get QA deployed and running - #1582)~~ It would be best if this was done by someone *unfamiliar* with the extensions so feedback could be provided on quality of install docs etc etc." 1323088817000000 1328000871000000 1566 enhancement David Raznik jilly mathews ckan-v1.7 closed fixed [super] Finalize Google analytics extension " * ~~(Deploy it)~~ * Integrate with stats plugin (now that is in core) - #1101 * Blog about it #1581 * (Other tickets re upgrading for v1.5.1 - done but where are tickets?) " 1324293168000000 1338203338000000 1477 enhancement kindly kindly ckan-v1.6 closed fixed [Super] Group refactor "This is the implementation of the final draft in http://wiki.ckan.org/Groups_Refactor Steps to complete this will follow (somewhat) in the following order. Change package_group table to membership. (#1478) 3d * model migration. (done) * simplify revisioning (done) * Remove any sqlalchemy releations from groups. (done) * change logic functions (done) Single type per group (#1531) * migration add type to group table. (done) 1d Add capacity table (#1522) * model migration. (done) * add to logic functions. Allow other objects to be added to groups. (#1531) * change logic functions (done) Adapt thedatahub.org to use new model. * move auth groups over to groups/members * change user_object roles to use groups? keep user object roles for time being. * make create update package/group put the correct items in the group/members/capacity tables. * work out how ui works. --not doing the below as part of this super ticker now #1669. Create model for dgu. * adapt auth functions to use auth. * adapt package_create/update group_create/update to fill in the correct capacties/members. Adapt Iati, and potentially other custom instances. Sub-tickets: * #1279 Deprecate Authorization Groups * #1198 Publisher hierarchy " 1321959320000000 1328016209000000 1521 enhancement zephod shevski ckan-sprint-2012-01-23 closed fixed [super] Groups UI and UX improvements in v1.6 from Cygri "Various UX and UI improvements to groups. Etherpad: http://ckan.okfnpad.org/feature-1521-groups-ux == Improved groups == From Richard Cyganiak. This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as http://ie.ckan.net/ could be turned into a simple group on the [http://thedatahub.org Data Hub] without losing their separate identity. * ~~'''Group homepages:''' Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.~~ * '''Subdomains for groups:''' Enable subdomains such as {{{ie.thedatahub.org}}} as aliases for {{{thedatahub.org/group/ireland}}} * ~~'''Search within the group:''' Have a big search box that searches just within the group on the group page, with a look like on http://thedatahub.org/dataset~~ * ~~Have an '''Add a dataset to this group''' link on the group page, perhaps in the sidebar.~~ * ~~'''Navigation within the group:''' Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from http://thedatahub.org/dataset~~ * ~~'''Group documentation:''' Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.~~ * ~~The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.~~ * Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines ''About'', ''Frequently Asked Questions'' and ''How to get involved'', we'd then get three links in the sidebar that point to {{{datahub.org/group/ireland/about}}}, {{{datahub.org/group/ireland/frequently-asked-questions}}}, {{{datahub.org/group/ireland/how-to-get-involved}}} * RP: WONTFIX this is being considered for future (generally useful) but cost/benefit here was low particularly as it wasn't clear what optimal implementation was. == Additional == * ~~Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)~~ * Better group index page (sort by number of dataset, allow searching ...) - #1403 * Add search to groups: http://thedatahub.org/group * Way to filter by activity, number of datasets * #1597 '''Tag search, filtered by group''' * RP: not worth it in terms of benefit/cost IMO == Steps == * ~~""Add Dataset"" button on action bar~~ * ~~Make it a dropdown? -> New Dataset / Existing Dataset~~ * ~~Redirect Add Existing -> Edit~~ * ~~Check permissions~~ * ~~Expandable ""about this group"" box~~ * ~~Implement simplest search-this-group box~~ * ~~turn the list of datasets into a dynamic list of search results~~ * ~~pagination~~ * ~~Matching set of facets in the sidebar (matches 'real' search page)~~ * ~~Deleteable, displayed facets~~ == Nice-To-Have == * Get rid of the old group_package_show action (confirm with team) * Tabbed ""about this group"" box (extract headers?) * ~~Scrollable group members widget in the sidebar~~" 1323169961000000 1330675930000000 1787 enhancement dread ckan-future new [super] Improve RESTful API " * Lists of entities should be full URLs, rather than just the names * Discoverability - /api/v3/rest should list the entity types that can be listed This could be v3 of the RESTful interface." 1328702082000000 1328702082000000 1233 enhancement thejimmyg ckan-backlog new [super] Improve wiki-style functionality for history "At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas: * #191 Searching by modification date * #193 Searching by time-related field * #301 Package discussion pages * #1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages * #1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page * Other improvements as per my word doc." 1311179392000000 1315948668000000 927 enhancement rgrp rgrp ckan-v1.4-sprint-3 closed fixed [super] Improvements to CKAN documentation "List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along. == Major == * ~~(Make blog.ckan.org into ckan.org and move ckan.org to trac.ckan.org/)~~ * Document caching system - ticket:841 * Howto guide for writing an extension - ''started on this at http://wiki.ckan.net/Extending_CKAN'' == Miscellaneous == * Contributors / credits page: ticket:928 * ~~(Data proxy documentation: move it to packages.python.org/dataproxy)~~ * ~~(List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))~~ * Document license configuration and license system - ticket:973 * Refactor trac frontpage moving general feature overview to ckan.org * Create proper roadmap information on http://trac.ckan.org/ * Add link to wiki.ckan.net to ckan.org " 1295869492000000 1300105638000000 1232 enhancement thejimmyg ckan-backlog new [super] Interface improvements "Child tickets: * #1194 ""Welcome back"" message for newly registered user * #1202 Links to datapkg utility don't lead to info about it * #925 Change the search box icon to remove the down arrow * #923 Search box doesn't work in leaderboard page in stats extension * #1034 Flash message cached * #737 Markdown syntax summary page * #811 Extra field editing form layout breaks when there are long field names " 1311178296000000 1315948536000000 1571 enhancement rgrp jilly mathews ckan-backlog assigned [super] Issues Extension """Proper"" issues extension. * Etherpad: http://ckan.okfnpad.org/issues * Code: https://github.com/okfn/ckanext-issues Overview * This would be an extension and improvement of existing todo extension - #842 (https://github.com/okfn/ckanext-issues) * Current extension does work but lacks polish * Should we polish or do significant improvements? * Functionality - see etherpad == Steps == * ~~(Rename todo(s) -> issue(s) in existing extension)~~" 1324294225000000 1330863183000000 1142 enhancement annapowellsmith rgrp ckan-v1.5 closed fixed [super] Major Overhaul and Extension of CKAN Documentation "Child tickets: * #1041 Start Using the CKAN Wiki for Tutorial-style documentation * #1192 Convert CKAN Sphinx docs into admin/reference manual Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927 * CKAN 1-page overview (for enterprise and for data hackers) * Administrator's Guide (including install) * Extensions Guide * Separate CKAN.net info from Software Documentation (?) Also now a wiki page with more detail: http://wiki.ckan.net/Documentation_Plans == Minor Items == " 1305727452000000 1313775665000000 2355 enhancement toby toby closed fixed [super] make extensions cleaner "* Do an analysis of why upgrades are problematic * What can we do better * Decide what to implement" 1335885051000000 1338212946000000 1231 requirement kindly thejimmyg ckan-backlog closed wontfix [super] Management Information Reporting "Child tickets: * #1101 Integrate stats and googlanalytics into site nav We have a spreadsheet from UKLP of statistics we'd like to generate" 1311173919000000 1325474447000000 1141 CREP johnglover ckan-backlog closed fixed [super] Moderated Edits User Interface "'''Proposer''': John Glover[[BR]] '''Seconder''': James Gardner == Abstract == We are trying to achieve these goals: * To get people involved with making edits to CKAN metadata. * To have an ownership model as to who can moderate and validate these changes * To not put too huge a burden on these owners. This feature allows anyone to edit a package and create a new revision, but requires an owner/moderator to approve a revision before it is are made ""official"". There have been a lot of discussions around the revisioning system side of this ticket (CREP 0002) and I think these are now largely resolved. We now want to discuss the user interface. == The Problem == We require the following functionality: * Allow a group of changes to be stored as a new revision. * Allow a linear stack of ""community"" revisions. * Provide a way for the editor and moderator to compare previous revisions to the current one. * When a moderator approves a change it creates a new revision flagged ""moderated"" (this is analogous to a merge commit) * Provide a way for the editor and moderator comment on revisions if necessary. Extra features: * Need a way to summarise the changes (as part of the preview perhaps) * Sysadmin needs to purge a revision completely == Specification == === UI/UX === UI Mockup: [[Image(ModeratedEdits2.png)]] Revisions: * Revisions are per package rather than per field. * Internally CKAN has separate revisions for resources, extras and package metadata. From a user's point of view this could be confusing to expose, so everything that they see on a package form when they hit save is a single revision. On the Edit page: * We have a panel on the right, listing all the revisions with the current moderated one selected. Moderated revisions are highligted in some way (red and bold?). * The values displayed in the form are by default populated from the latest revision (whether community or moderated) * Under each field is a ""shadow"", showing the value of the field in the revision selected in the panel, if it is different from the value in the field. By default the shadow values are populated from the latest moderated revision which is the one selected in the revision panel by default too. * When you change the value of a field, a shadow may appear or disappear accordingly. If they disappear a box saying that they are the same replaces it * If you want to edit values from a previous revision, you first select that revision to get the shadows populated. There is a button named ""Replace fields with values from this revision"" under the revision list. You click this, a warning pops up and then you say ""Yes"". You then select the moderated revision again. * We also allow package comments the same way as the todo extension works at the moment. Additionally, we need to be able to differentiate between what the moderator wrote and what a community member wrote, and so we may need to make a small change to the todo extension to facilitate this. * In addition to package comments, each revision will have a revision log (analogous to a commit message). === Technical Details === * This CREP will result in a new CKAN extension. * It depends heavily on the new revisioning system (CREP0002), some of the details of which are yet to be finalised. * This CREP therefore requires working closely with David Raznick to come up with an API that the UI AJAX calls can use. * We will then use suitable test data to mimic these API calls until CREP0002 is ready. == Why do it this way == This hopefully provides a clear and consistent mechanism allowing both a community member to make new revisions and a moderator to view and approve revisions, with largely the same UI/UX. == Implementation plan == === Deliverables === A new CKAN extension, consisting of: * Code: Python, HTML, CSS, Javascript * Unit tests * Localization * Documentation === Participants === John Glover to do it. === Progress === John has implemented the bulk of this UI. Just some things to tidy up before it is complete: * Genshi stream filters to be updated with CKAN 1.5 / 1.5.1 templates * history_ajax / read_ajax to be replaced with calls to Action API (or Util REST API) I've split these two off into a new ticket #1604. === Related Progress === The Todo extension is written and available at: [https://bitbucket.org/johnglover/ckanext-todo]. In the section 'The Problem', under extra features, we mention a need for the sysadmin to be able to purge a revision already. This is already done. === See also === #1129 Backend work " 1305721003000000 1325352507000000 1736 enhancement kindly kindly ckan-v1.7 closed fixed [Super] Multilingual Metadata "Translate all areas of metatdata to selected language. This includes finding datasets in all languages. Multilingal Metadata: Tasks: * Add translation table. 2d * Api to add translation 3d * Add extention point to dataset view. 3d * Add extension point to search index to add translations. 2d. * Add all eurovoc data to translations. 1d. * Make multilingual solr schema 2d #1715 " 1328012697000000 1338205016000000 1605 enhancement kindly rgrp ckan-v1.7 closed fixed [super] Multilingual support in CKAN "Multi-language: * dataset and resource metadata (and other objects such as groups?) * Options for implementation in http://ckan.okfnpad.org/multilingual * Field values in taxonomy (e.g. country names - Eurovoc) * #1665 Research into Eurovoc * Display Taxonomy in different languages * Field values not in taxonomy (e.g. title & description) * Use extra fields e.g. _i18n_title_fr = Le data. * (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.) * EC extension (templates, form) * Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions? " 1325473367000000 1338193641000000 1294 enhancement rgrp rgrp ckan-v1.5 closed fixed [super] Package creation and editing UX improvements "Largely an integration of work from ckanjs * #1295 Simplify package create form * Improvements to edit form * Split edit form into sections * Remove preview from edit form (introduce inline preview on description/notes field) * #1296 Improved resource adding/editing on dataset/page page * #878 File upload integrated into resource creation * #1297 In-place editing of notes field for existing packages " 1314405811000000 1315948257000000 1669 enhancement ross ross ckan-v1.7 closed fixed [super] Publisher profile "Implement the necessary backend changes to build profile for publishers using the group refactor. See http://ckan.okfnpad.org/publisherprofiles Subtickets are at: * Publisher profile analysis #1670 [2d] * Publisher profile sysadmin authz #1671 [1d] * Publisher profile user authz #1672 [1d] * Test auth logic hooks for publisher profile #1674 [3d] * Publisher admin authz #1675 [3d] * Publisher editor authz #1676 [2d] * Add approval status field to groups #1673 [1d]" 1326802473000000 1338204071000000 1594 enhancement johnglover johnglover ckan-v1.7 closed wontfix [super] QA Improvements "Now that QA has been updated for Celery and CKAN 1.5.1, there are a number of improvements that we should probably implement. * Integrate QA into frontend - 1d - #1583 * QA report improvements - 2.5d - #1584 * QA - Check license as part of 5 star rating - #1587 * QA - Give SPARQL endpoints a 4 star rating - #1588 * QA - Give 5 star rating to datasets with link metadata - #1589 Main pad: http://ckan.okfnpad.org/qa Some QA discussion (relating to Celery in particular) at http://ckan.okfnpad.org/queue" 1324489620000000 1338203215000000 1578 enhancement rgrp ckan-backlog new [super] Re-enable and refactor ratings 1324322443000000 1325473015000000 2226 enhancement rgrp rgrp ckan-sprint-2012-03-19 closed fixed [super] Refactor and improve documentation (v1.7) "Documentation is key to the success of projects! = Move material into main docs from wiki = * http://wiki.ckan.org/Writing_asynchronous_tasks * http://wiki.ckan.org/Domain_Model * http://wiki.ckan.org/Deployment = Create a User Guide = Basic step-by-step on getting started with CKAN (See start on set of slides here e.g. http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/) * Publishing data * Authorization and workflows * Publisher Profile (Workflow) = Break into sections = Suggest something like: {{{ # General * Intro * Conceptual Overview * What CKAN helps you do (http://ckan.org/) * FAQ ? (or point to the wiki) # For Administrators * Installation and Setup * Configuration options * Customization - Theming etc * Authorization and Workflows * Storage # For Users (Publishers, Data Wranglers, etc etc) * Walkthrough of publishing a dataset * Storage * API (see Developers section) # CKAN Developers * Domain Model * API * Extensions * i18n CKAN Developers * Buildbot }}}" 1331481965000000 1332475494000000 1094 enhancement thejimmyg thejimmyg ckan-v1.5 closed duplicate [super] Refactor the Auth System "Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787. Two man reasons for the changes are: * We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer. * Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able. The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package). 1. The first thing the is_authorized method is a hook to a plugin which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!) Reason: authorization can be completely delegated to another system (or partially) 2. is_authorized method currently takes (username, action, object) but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model. 3. Rename these two classes to better reflect what they are * AuthorizationGroup -> UserGroup * Group -> PackageGroup 4. Rename the Editor role to PriveledgeUser since Editors sometimes can't edit. Although this sounds a bit radical we already have auth extensions. == Read-only CKAN Web UI == (Additional requirement from #764) Whilse using CKAN web interface, you are not tempted to edit stuff: * You know at all times this CKAN is read-only * All editing facilities are still seen but greyed-out with an indication why it is." 1303117973000000 1311173649000000 961 enhancement kindly rgrp ckan-v1.5 closed fixed [super] Refactoring of forms, validation and model synchronization "This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN. ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html == The Issue == From #926: The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise. From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html: ... !FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing. At least 3 functions involved [in this area]: 1. Generating (or just filling) a form template with 'form data' (and errors) 2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization' 3. Converting form data to model data (and validating) (inverse of previous step) == Related Tickets == * #926 - Pick a simpler form framework * #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict * #1079 Refactor API to use new logic layer and dictization * #1078 Refactor WUI controllers and forms to use logic layer * cf existing dumper and importer code * This will fix #662 * [not ticketed yet] - validation layer (should work on serialized objects?) * #662 - Can't put entity that is returned by posting to package register (Defect) * #972 - Merge 'extras' into main package dict rather than having separate key * #1035 - Form impressions are given IDs * #810 - Move ""add packages"" field up in group form (easier to do this once forms are done)" 1297069849000000 1310126100000000 2204 enhancement ross rgrp ckan-sprint-2012-04-30 closed fixed [super] Related (Stuff) Extension "This is a reworking of the existing apps extension. Initial proposal at http://wiki.ckan.org/Proposals#Apps_in_CKAN and http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN == Naming == What do we call this extension? * related * RP best IMO but perhaps too close to separate relationships concept? * relatedstuff * links * references == Proposal == ""Related Stuff"": Apps as in an application (website/service/tool) that uses this dataset (as in Apps and Ideas extension) * This case seems a general example of wanting to link some external thing to a dataset (e.g. this policy paper refs this dataset, this application uses this dataset, this visualization is built on this dataset cf http://lists.okfn.org/pipermail/ckan-discuss/2011-August/001607.html) == Implementation == New table named Related with following structure {{{ || id (int) || type || title || description (markdown) || image_url || owner_id || url || created (timestamp) }}} * type = Idea | App | API | Visualization | Post | Paper | News Article * Suggest we make this a ckan.ini config option (comma separated ...?) * Do we want the possibility of different templates for different types of Related objects? * image: ?? Depends where we store images. Simplest option would be to change to image_url and leave it to users to have already uploaded an image somewhere. If not we need to support image uploading and storage. See #1692 (add image attribute to datasets and groups) for more discussion, once implemented the URL here can be an internal url. * owner_id = user_id or creating user (see authorization below) Related2Dataset (note that related_id, dataset_id tuple should be unique). This allows for m2m connections. If a given related item is only with one dataset this could be simplified. May contain status so dataset owner can turn this on/off. {{{ || id || dataset_id || related_id || status }}} status should be used to allow for a dataset owner (for dataset_id) to de-activate the relationship between the dataset and the related. === Url === * /dataset/{dataset-name}/related/{related-item-id}/{related-item-title-stringified} * If a reference item could exist in its own right (and perhaps refer to multiple datasets then it should get its own url at e.g. /related/{id} * /dataset/{dataset-name}/related/add => Modal dialog on related tab so we can use API to create them. /dataset/{id}/related <- list * use image_url for small icon etc, title description (shortened?) * Click through to full related item (optional) * dataset owner is shown show / hide button ... (or on /dataset/{id}/related/{id} ) * related owner sees an edit button / icon (pops up modal) /dataset/{id}/related/{id} (optional) /dataset/{id} will have a Related tab (with bubble with count). * Drop down with Add Related -> Pop-up modal and save via API (Not used: /related/add with dataset prefilled ... ) /related/{id}/edit === Authorization === Addition of related item be considered orthogonal to datasets (and hence with separate authorization i.e. i can add the info that site X uses dataset Y without needing permission to edit dataset Y). Thus any logged in user could add a Related item. We set the owner of the related item to creating user and going forward only that user or a sysadmin can update or delete. NB: we could have a system where datasets owners have to approve related items before they show up next to their dataset. This would add substantial complexity so I propose we leave out of phase 1. === Tasks and estimates {7.5d} === [x] Model + Migration for Related table. {0.75d} [x] Controller for Related (or relevant sections in Package controller). {0.75d} [x] Routing setup. {0.25d} [x] Schema for related. {0.5d} [x] Logic layer actions. {1.0d} [x] Auth (default + publisher). {0.5d} [x] Templates + Dataset changes (new tab etc). {1.0d} [x] JS Application for adding Related objects in a modal. {1.0d} [x] Testing. {0.75d} [x] Dataset owner disabling of Related (via M2M table). {0.5d} - Updated to allow author of related to delete as well [x] Documentation. {0.25d} " 1330862992000000 1337594723000000 1397 enhancement kindly rgrp ckan-v1.7 closed fixed [super] Resource archiving "We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...) Etherpad: http://ckan.okfnpad.org/queue (most relevant parts inlined here) == Preliminaries == * Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions) === Configuration setup for daemons === Pass config through to workers i.e site_url, user, api_key. Need to make site user account. #1408 celeryd config: All providers of tasks will add an item to the following entry point: {{{ [ckan.tasks] name = ckanext.{name}.tasks:.... }}} celeryconfig.py {{{ from pkg_resources import iter_entry_points for entry in iter_entry_points: celeryimports.appen(....) CELERY_IMPORTS = celeryimports }}} == Work Items == * Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383 * Generate archiving request on resource url change [0.25d][0.25d] - #1399 * Make site user account. * Make entry point system for celery config * Archiver daemon #891 1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d] 2. Rewrite archiver to use external storage. (decide how!)[3d][~2d] * Write to resource and task status table.[1d][0.75d] * [Required?] Make archived data available in WUI - #892 * Documentation - #1400 " 1318529593000000 1338202920000000 1032 enhancement rgrp rgrp ckan-v1.6 closed fixed [super] Resources in WUI "Add resources into Web User Interface. * Locate at: /dataset/{dataset}/resource/{id} * CRUD * Authorization Done: * ~~#945 - Richer resources - Resource Groups, new fields, improved UI~~ * ~~#1445 - Resource View page in WUI~~ * ~~#1450 - Dataset view pages to match~~ Moved to superticket #1506: * #978 - Edit Resource Extras in Web UI " 1299782021000000 1330348463000000 945 enhancement kindly kindly ckan-v1.6 closed fixed [super] Richer resources - Resource Groups, new fields, improved UI "Super ticket: #1032 This is a meta ticket for changes that are going to happen in resources. * New resource group table. #956 * New kind field in resource. #957 * UI for new kind field. #958 * Resources in REST API ticket:358 * Resources in WUI #1445 * Make Resources first class entity. #922 (duplicate?) Background on this change can be found at: * http://ckan.okfnpad.org/resources - discussion etherpad * Contains commented version of James Gardner's main email: http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html * UseCasesResources " 1296475283000000 1325259350000000 1235 enhancement thejimmyg ckan-future new [super] Search Improvements "Child tickets: * #234 UI Review - Autocomplete package names & tags in search * #193 Searching by time-related field * #191 Searching by modification date * #905 Unable to search with accented characters in package names * #906 Ability to search without accents for accented words * #924 Search box has no search button Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two." 1311182641000000 1311182641000000 1717 enhancement shevski ckan-backlog new [super] Search UX improvements "1. Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search) 2. Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page). 3. Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it. 4. More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu 5. Blue search button should be displayed in line with the search bar, not underneath 6. Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name" 1327603981000000 1330088539000000 1549 enhancement ross ckan-backlog closed wontfix [super] Short link tool "It would be great to have a CKAN extension that allowed users (or CKAN itself) to generate short links to other URIs (both internal and external). Once created, shortlinks made by CKAN should be changeable. This would allow uploaded content to be moved without the user's link changing at all. The tool itself might also be of use as a general link-shortener to users other than the CKAN system itself. Another useful feature would be for this to also collect some simple analytics such as the referrer and client IP for future reference. I'm not yet sure what we would do with the analytics other than some sort of popularity metric. Questions: - Core, or Extension, or Self-hosted? " 1324036998000000 1325474219000000 1370 enhancement toby rgrp ckan-sprint-2012-04-30 closed fixed [super] Social sharing for datasets (and resources) "Add a way to share datasets and see mentions of datasets on twitter (and elsewhere) As a visitor I want to share a link to a dataset I have found. I also want to see how many others have shared this (or mentioned it). * Sharing platform: simplest option is probably twitter but could generalize to e.g. sharethis system which supports, facebook, twitter etc etc. * Will need a config option for relevant API key e.g. ckan.sharethis.apikey * Location: Put this at top of sidebar on dataset view (a better suggsetion?) * Should show how many mentions / shares there are. (very important!) == Discussion == * (Tom and Ira and ...): do we really want this, versus e.g. proper follow extension? * What about bringing conversation back in to CKAN. E.g. show all times this dataset (i.e. its url) was mentioned on twitter. If we do this I think this should be definite +1. * Also, given our users, I think just doing twitter (identica) may be sufficient (how many people want to share links to datasets on facebook?) " 1317422686000000 1336045983000000 1574 enhancement ross ross ckan-v1.7 closed fixed [super] Storage changes including automated save to Webstore "Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore. 1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs [5d] 2. The user is provided the link to the file as currently. 3. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) [5d] - took 6 * It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file. 4. The archiver will continue to move the file into permanent storage as per current practice. == Tickets * #1608 Merge storage into Core [5d] * #1609 Celery task for ckanext-archiver to write to webstore. [5d] (took 6) * #1687: Deploy to datahub [1d] * #1681: Core storage documentation [1d?] * Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc * External: here's how to configure it and here's what you get ... (goes in main sphinx docs) " 1324315633000000 1338202955000000 1698 enhancement seanh seanh ckan-v1.7 closed fixed [super] Tag Taxonomies "Add drupal-like ""taxonomies"" to CKAN. Etherpad with user stories, feature list, design and implementation discussion: http://ckan.okfnpad.org/22 Branch where this is being developed is feature-1698-tag-taxonomies: https://github.com/okfn/ckan/compare/master...feature-1698-tag-taxonomies Tickets related to this have [http://trac.ckan.org/query?status=accepted&status=assigned&status=closed&status=new&status=reopened&group=owner&max=1000&order=milestone&col=id&col=summary&col=status&col=owner&col=milestone&col=keywords&keywords=~taxonomies keyword taxonomies]." 1327415756000000 1338204433000000 1682 enhancement dread dread ckan-v1.6 closed duplicate [super] Translatability for EC "We need to be able to translate: * EC extension (templates, form) * Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions? * Field values in taxonomy (e.g. country names - Eurovoc) * #1665 Research into Eurovoc * Display Taxonomy in different languages * Field values not in taxonomy (e.g. title & description) * Use extra fields e.g. _i18n_title_fr = Le data. * (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.) Timescale to complete this - Mar/Apr. Related work to interface with: * Eurovoc - Sean * i18n in search index - Adria" 1326911899000000 1330038786000000 1526 enhancement icmurray icmurray ckan-sprint-2012-03-05 closed fixed [super] Update and test existing DGU package form "Re-create behaviour of old form without using form-alchemy. And unit test. * [X] write high-level functional tests for the form. Creation / edition / validation. Original estimate: 6 days (3 days x 2) Time spent: 3 days Time remaining: 0 days * [X] write the html Original estimate: 2 days (1 day x 2) Time spent: 1.3 days Time remaining: 0 days * [X] modify the existing validation schema Original estimate: 2 days (1 day x 2) Time spent: 1.3 days Time remaining: 1 days * [X] split resources into types: individual, time-series, and additional. Original estimate: 2 days Time spent: 2.5 days Time remaining: 0 days * [X] contact information is derived from Publisher, but can be altered for each dataset. [#1617] Original estimate: 4 days Time spent: 0 days Time remaining: 4 days (Waiting on publisher integration) * [X] foi contact information is derived from user, but can be altered for each dataset. [#1617] Original estimate: 1 days Time spent: 0 days Time remaining: 1 days (Waiting on publisher integration) * [X] provide stubs for the theme and sub-themes [#1618] Original estimate: 1 days Time spent: 0 days Time remaining: 0 days * [X] provide stubs for the publisher hierarchy [#1619] Original estimate: 2 days Time spent: 0.5 days Time remaining: 1.5 days Note - the groups refactor is essentially ready to be used. And this would be a good test case for it. * [X] don't allow package-type selection when editing a package Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa. (#1620) Original estimate: 0.5 days Time spent: 0 days Time remaining: 0 days " 1323173313000000 1330942383000000 1654 task amercader amercader ckan-v1.7 closed fixed [super] Update Publicdata.eu to the latest CKAN stable version "Tasks include: * #1813 Update ckanext-pdeu (4d) * #1814 Update harvesters (2d) * #1649 Verify ckanext-rdf works with latest CKAN (3d) * #1815 Reenable Sparql endpoint (?) * #1816 Update ckanext-apps (2-3d) " 1326710647000000 1338203166000000 1734 enhancement amercader amercader ckan-v1.7 closed fixed [super] Updates on harvesting for UKLP "Some changes are needed in ckanext-harvest to support the new piece of work in the UKLP harvesters, mainly related to the deletion of existing sources. * #1726 Update harvesting model * #1727 Add ""current"" field to harvesting objects * #1728 Update UKLP harvesters * #1733 Delete (inactivate) harvest sources At some point we also want to support a publisher model for the harvesting: * #1731 Publisher support for harvesting" 1328008474000000 1332167315000000 1600 enhancement rgrp rgrp ckan-v1.7 closed wontfix [super] User guide with integrated help in the WUI "Provide help to users in the WUI, with this linking to a detailed user guide. * User guide (suggest at help.thedatahub.org (or possibly help.ckan.org) * Snippets in the WUI with links to full user guide. * Provide helper methods for creating these links with config options for setting path to user guide so that the link location can vary on install by install basis * General info about publishing/working with/etc data (where we don't duplicate elsewhere) Est: (?) 5-10d == Topics == See http://ckan.okfnpad.org/datahub-help * Step-by-step for adding a dataset including tips * Reuse: http://datapatterns.org/datahub.html * Quick overview of relevant parts of domain model (dataset and resource) * API (perhaps link to main docs) * Command line interface and dpm * Using groups == Why do this == We already have wiki.ckan.org and docs.ckan.org. Why do this? Answer for docs: * This is administrative guide and focused on general CKAN software. * Hard to provide completely generic help. * Integratable with the DataHub site Answer for wiki: * No spam * General feeling that wikis are a little bit loose and untidy. We want this to be smart, clean and useful. * Integratable with the DataHub " 1325257824000000 1333362051000000 1343 enhancement rgrp rgrp ckan-backlog new [super] User related improvements (login, user pages etc) " * Disallow account creation via openid - #1386 * Require email field - #1319 * Require email confirmation to be activated (?) * Improvements to user page (e.g. show activity and more info about user) - #1396 * Modal user login - #1311" 1316017098000000 1318528138000000 1506 enhancement zephod rgrp ckan-sprint-2012-03-05 closed fixed [super] UX improvements to dataset edit, create and search + group + user page (Dec 2011) "== Historical Parent Tickets == * ~~Parent-ticket: #1450~~ * ~~Parent-ticket: #1032~~ == Step Zero == * ~~COMPLETELY REDESIGN resource edit process~~ == Dataset Edit-Resources == * ~~Force format field to be lowercase~~ * ~~Modify backend to serve it up in that format~~ * ~~Subticket #978: Add extra fields to resources in Web UI~~ * ~~Fix display of large datasets (http://test.ckan.net/dataset/europeana-lod)~~ * ~~When submitting forms with errors, open the correct resource pane.~~ * ~~No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10~~ * ~~Resource description field to become the markdown editor we use everywhere else.~~ == Dataset Settings == * ~~Subticket #1648: Clarify edit terminology~~ * ~~Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).~~ * ~~Fix (i.e. allow / show) name editing on dataset edit.~~ * ~~Fix the slugify so it does not overwrite if i change title after fixing name.~~ * ~~Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'~~ * ~~Autocomplete the author/maintainer box (requires API extension?)~~ [WONTFIX what? it's not like that field is linked to users of the site or anything] == Dataset View == * ~~Fix animation glitch in expanding description~~ * ~~Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?~~ * ~~Verify that there are no more (none) placeholder objects (i believe they've been refactored out)~~ == Dataset Create == * ~~""Upload or link data files, APIs and other materials related to your dataset.""~~ * ~~Include the description markdown editor again (keep it small)~~ * ~~Licence should maybe be restored~~ * ~~""(Don't worry if you don't know what your licence is)""~~ * ~~Show the log message field~~ * ~~Beautify~~ * ~~Pre-populate the log message field?~~ * ~~? Show Home Page field when creating dataset~~ [WONTFIX don't push it] * ~~? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation?~~ [WONTFIX doesnt make sense with new resource editor awesomeness] == User pages == * ~~Improvements to user page UX - 1d - #1396~~ * ~~Broken links for non-Gravatar use icons - 0.25d - #1719~~ == Etc == * Take before and after screenshots" 1323088306000000 1330702859000000 1797 enhancement rgrp rgrp ckan-sprint-2012-03-05 closed fixed [super] Webstore v2 - Integrated and based on ElasticSearch "Main tasks: * Integrated seamlessly into CKAN (authorization etc) * Based off ElasticSearch Etherpad: http://ckan.okfnpad.org/webstore = Tasks = * ~~New API point (config + controller) [0.75d]~~ * /api/data/{resource-id} * ~~Authorization [0.5d]~~ * ~~Documentation [0.25d]~~ * ~~Upgrade data viewer to use ES [0.5d]~~ * ~~Convert webstorer to new setup - http://github.com/okfn/ckanext-webstorer [1d] - #1753~~ * See below * ""Create DataStore Resource"" in Resource creation (??) * Just creates a resource with blank url and datastore enabled (webstore_url non null) * ~~Tick box to enable DataStore in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)~~ * Also need to use this info in DataStore extension? * ~~Create nice instructions on Resource page about using the DataStore~~ * ~~Rename webstore -> datastore~~ * ~~Deploy [0.5d]~~ = Details = == Layout on ES == {{{ elasticsearch/{index}/{type-id} index = ckan-{ckan-id} type-id = resource-id (1) }}} == Request path == Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results See as an example: https://github.com/okfn/elastic-proxy == Questions == 1. How do we decide what to use in the viewer e.g. Webstore data versus DataProxy * Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't. 2. How do we decide when webstore is activated * Ans: see previous. 3. Do we (and how do we) allow for resources which are just for webstores * Ans: we have to set webstore_url = True ... on save. Or add a big button saying: ""Create Blank Resource"" ... 4. How does webstorer work ... * CSV import - NS: make sure we add e.g. rownum or __id__ = rownum ... * Separate mini-service (webstorer.thedatahub.org ...) 5. How do we selectively disable recline where not appropriate (e.g. query support makes no sense for DataProxy) * Ans: do not bother at present. Focus on Webstore route. 6. Where does geodata support come in (in Recline or separately) * This is a separate issue == User Story 1 == User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save => Result = New Dataset + Resource + Data API to resource and nice Preview of the data Solution: 1. Allow normal upload process 2. Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database. 3. Update the resource in some way (set webstore_url) = User Stories = == User Story 2 == User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise) => Result = New Dataset + Resource + nice Preview of the data (wherever possible) == User Story 3 == User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store. => Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...)" 1329306185000000 1331412644000000 1601 enhancement amercader rgrp ckan-v1.7 closed fixed [super] Well integrated Geo-spatial support including search "Need thoughts from Adria and James. RP thoughts * Integration into WUI search: #1469 * JS Editor for adding/editing spatial attribute: #1818 * Theme/template hook for adding the map e.g. on dataset page (or elsewhere) * Clear Documentation * Deployment * Announce post * (?) Map page for stats section showing location of datasets Extras (probably separate): * csw support ..." 1325259821000000 1338204029000000 1243 enhancement timmcnamara closed invalid "[thedatahub.org] ""Upload Data"" should provide a nicer" "If someone who is not logged in clicks on ""Upload Data"", they are presented with {{{ Not authorized to upload files. }}} This is in emboldened red text. That is, the system is telling a user that they're doing something illegitimate, even though they're actually following an invitation from the main menu. I propose two options for resolving this: 1. only display the ""Upload Data"" tab if someone is authorised to upload data 2. if someone isn't logged int, provide a nice welcome message thanking them for wanting to upload data, but we only allow registered users to do so ENV: thedatahub.org, CKAN 1.4.2b" 1311577466000000 1328000405000000 2809 enhancement aron.carroll toby demo phase 3 closed fixed _ not defined for modules "eg on confirm delete http://localhost:5000/zh_TW/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada _(....) gives [object Object] not translation" 1344335863000000 1344347814000000 1697 enhancement rgrp ckan-backlog new A Configurable list of states for a Dataset "Currently have 'active' and 'deleted' suggest also: * 'draft' * 'hidden' (Do we need both). Also write out workflows related to these." 1327400630000000 1338204189000000 366 enhancement pudo dread ckan-v1.4 closed fixed A flash message says why you are redirected to the 'user login' page "You get redirected for a number of different reasons - often confuses me let alone the average user! Cost: 1h" 1279128058000000 1300212171000000 26 enhancement somebody johnbywater closed duplicate A registered person creates their own tags for a package 1152551351000000 1152555283000000 1157 enhancement pudo pudo pdeu-1 closed fixed A simple theme for publicdata.eu, CKAN "Change the visual style of CKAN to be more like these sites: * GitHub * Quora * Google Projects This does not include major UX work. " 1306337573000000 1309804029000000 123 enhancement dread dread v0.10 closed fixed Ability to edit Group in WUI "Add Group editing page. If no permissions to change group can't edit group. Also cannot view edit page. Editable attributes: name, title, description No preview needed " 1253708041000000 1254321447000000 848 enhancement pudo ckan-v1.3-sprint-1 closed fixed Ability to hide extras fields on package read for helsinki regional infoshare. 1291714316000000 1291751275000000 906 enhancement thejimmyg Stiivi ckan-sprint-2012-03-05 closed fixed Ability to search without accents for accented words "In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results. This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents. For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCII//TRANSLIT). " 1294939588000000 1330990400000000 1051 defect sebbacon sebbacon ckan-v1.4-sprint-4 closed fixed Ability to set custom favicon Allow deployers to set a URL pointing to their own favicon 1300703160000000 1301305079000000 2497 enhancement toby toby closed fixed abort loses language settings "if not logged in then go to add dataset url the language is lost check if this is a ckan 1.7 issue" 1339067855000000 1339421305000000 2773 enhancement markw markw ckan-v1.9 new About page needs improving The about page for the DataHub (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh. 1343646795000000 1343646795000000 2409 enhancement ross markw ckan-sprint-2012-05-29 closed fixed Accept headers not handled correctly "CKAN returns HTML when an Accept header should make it return RDF/XML: http://lists.okfn.org/pipermail/ckan-dev/2012-May/002237.html " 1337683898000000 1337856979000000 116 enhancement dread dread v0.10 closed fixed Access Control - edited in wui "Based on a section of AccessControl design: WUI gives controls to user and administrator to change permissions on a package." 1253034802000000 1253709460000000 117 enhancement dread dread v0.10 closed fixed Access Control - group core functionality "Based on a section of AccessControl design: Group reads and edits are controlled by access control. WUI and REST interfaces covered." 1253271333000000 1254735855000000 114 enhancement dread dread v0.10 closed fixed Access Control - model "Create in the model basic operation of Access Control. roles table {{{ name | context | action ----------------------- admin| package | edit admin| package | edit-permissions admin| package | read editor| package | update editor| package | read reader| package | read This data is set-up on db init and will have no interface. }}} user-roles table: {{{ username | context_type | objectid | role rgrp | system | n/a | admin visitor | package | * | reader bob | package | geonames | admin visitor | package | geonames | editor visitor | package | geonames | reader john | group | ukgov | admin dread | group | ukgov | editor visitor | group | ukgov | reader This data will be added when someone is given permissions for the system, a package or a group. }}} Pseudo code: class Package def is_allowed(name, action): is_allowed(name, action, context=self) class Group def is_allowed(name, action): is_allowed(name, action, context=self) def is_allowed(name, action, context=None): ''' name: string - a username or IP for 'visitor' action: string - 'read', 'edit', 'delete', 'edit-permissions' context: object - a Group or a Package or None (which means system) ''' # look up user from name. # look up in user-roles table what roles this user has for this context. # for each roles, look up in roles table what actions are allowed. # return True if action is allowed, else False. " 1252494527000000 1253034529000000 115 enhancement dread dread v0.10 closed fixed Access Control - wui constrained by model "Based on a section of AccessControl design: Reading and writing to packages in the WUI and REST API are now dependent on the authz tables." 1253034394000000 1253091426000000 93 enhancement rgrp rgrp v0.10 closed fixed Access control for packages "Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: AccessControl. Cost: 8d " 1249055049000000 1253613274000000 1517 defect zephod zephod ckan-sprint-2011-12-05 closed fixed Accessing resource view sometimes auto-downloads the resource "Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file. For example, being here: http://test.ckan.net/dataset/europeana-lod and clicking on any of the links to the first resource." 1323109647000000 1323175957000000 614 task johnbywater johnbywater closed invalid Acquire local copy of XML schema for remote metadata documents 1284219077000000 1288038898000000 615 task johnbywater johnbywater closed invalid Acquire local implementation of UKLP schematron "We ""just"" need a schematron.validate(metadata) method. :-)" 1284219174000000 1288038926000000 1335 enhancement kindly dread closed fixed Action API - basic Provide an API to the logic layer at /api/action. 1315910021000000 1315910117000000 1438 enhancement dread ckan-future new Action API - parameter discovery/checking "Many actions in the Action API require parameters. What params are needed should be listed and checked. Because currently, if you get them wrong you simply get a useless 500 error. Currently they are listed in the docs, extracted from the code manually. So you could GET /action/api/package_list to receive not only the help text, but a list of arguments. And if you send an extra or missing argument then an intelligent error message can be returned. = implementation = How about some sort of decorator on the action function: {{{ @logic_params(id, offset, limit) def get_package_list(context, data_dict): ... }}} This would do the param checking, and is there a way to extract these params from the function? Or do a registration of the logic function? I'd certainly like to keep the list of the list of params for the function with the function, for ease of reading the code. Another good thing would be to pass in the params named as themselves, rather than having them contained in the data_dict." 1320161890000000 1338205050000000 1439 enhancement dread ckan-backlog new Action API discoverablility "A good service API needs to be discoverable, so you are not always having to refer to the documentation html. Maybe /api/action should return a list of actions available? (Currently this returns a 404.) * It would be nice to sort these into get/create/update/delete. * #1438 Parameters for each of the actions must be discoverable too /api/action/{action_name} should also return the help text / parameters allowable. (Currently this returns 400 error)" 1320161970000000 1325474974000000 1504 defect dread dread ckan-sprint-2011-12-05 closed fixed Action API given blank data causes exception "This is what caused the exception: {{{ curl http://localhost:5000/api/action/package_search -d='{""q"": ""osm""}' }}} The problem is the content-type is formencoding, the '=' is the first character of the request, and so internally the data_dict is set to ' ', which needs to be a dictionary." 1322756748000000 1324035928000000 1322 enhancement dread assigned Action API improvements "Focusing on improving Action API as the v3 API: * have an optional parameter of the data_dict called ""options"". Options would contain items that would get passed into the context. e.g. {{{{""options"" : {""ref_package_by"": ""id""}}. }}} * instead of using API version to change the way packages are referenced, use the ref_package_by. * All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'. * Action API is v3 of api, replacement for v1 & v2. Default for most urls is still v1, but if url is /api/action then default to v3. Next steps: * Add search API (package, resource, * Add Util API * Clarify JSONP still applies * Add doc strings, clarifying parameters" 1315474749000000 1340191088000000 1746 enhancement seanh ckan-backlog closed wontfix Activity streams pagination Currently user, package and group activity streams only return the most recent 15 activities, even though all activities are kept in the db. Do we want to add pagination - to both the API and the HTML pages - to support retrieving older activities? 1328446488000000 1355141062000000 1743 defect seanh seanh ckan-v1.6 closed wontfix Activity streams still link to deleted things For example, after deleting a dataset old activity stream events will still hyperlink to that dataset. 1328212016000000 1328527086000000 1299 enhancement seanh kindly ckan-sprint-2012-01-23 closed fixed Activity streams table migration Migrate tables for activity streams 1314696635000000 1327322739000000 2642 enhancement shevski amercader demo phase 3 closed fixed Adapt harvest frontend to new theme 1341846196000000 1345113645000000 2641 enhancement johnmartin amercader demo phase 5 assigned Adapt spatial widgets to new theme Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now 1341846147000000 1352658854000000 1727 enhancement amercader amercader ckan-sprint-2012-02-06 closed fixed "Add ""current"" property to harvest objects" "Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new ""current"" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes: * Model update (See #1726) * Update queries for getting harvest sources' datasets * Update ckanext-csw queries for getting the objects * Update inspire and ckan harvesters * Document new field" 1328007254000000 1328523641000000 2748 enhancement shevski demo phase 5 new add 'add new resource' button to sidebar "When editing a resource you see the current and any other existing resources in sidebar (e.g. see http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True) Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page" 1343212878000000 1344503744000000 2570 enhancement aron.carroll shevski ckan-sprint-2012-06-25 closed fixed add 'back to dataset' button to resource pages Suggest making this consistent with http://s031.okserver.org:2375/dataset/adur_district_spending/related by adding same button in top right of resource pages 1340039680000000 1340107841000000 2779 enhancement aron.carroll shevski demo phase 3 closed fixed add 'deleted' to deleted datasets that admins can view "Currently, as a logged in admin, you can view deleted datasets just as if they are live datasets. Nothing distinguishes them apart from normal datasets other than they don't show up on the dataset page This is potentially very confusing Can we add some kind of banner / label to deleted datasets to show they have been deleted? & Do you guys need Sam to design this? Related ticket for undeleting datasets - http://trac.ckan.org/ticket/2780" 1343737077000000 1343903455000000 158 enhancement dread dread v0.11 closed fixed Add 'extras' to REST API " * Ensure you can add/change/delete extra fields from packages using the REST API. * Update API docs to cover this. On Package get, ensure separation of extra attributes from other attributes. What is the meaning of posting an incomplete dictionary (updating a package)? Option 1 - if field not present, leave field alone * Delete field if field value is None Option 2 - user must supply all fields * Delete field if field absent Prefer option 1. Document this. Follows on from ticket:152" 1255703259000000 1255949818000000 121 enhancement dread dread v0.10 closed fixed Add 'Group' to main menu And associated page to browse group. 1253694827000000 1253716782000000 2754 enhancement aron.carroll shevski demo phase 2 closed fixed Add + icon to two buttons Specifically the 'add resource' button and the 'add item' button in related section 1343225177000000 1343227446000000 31 enhancement somebody rgrp closed wontfix Add a comment to a package "== As a == Registered user == I want to == Add a comment to a package == Notes == * The comment should be tagged with the username of the person making the comment * Should we allow comments per release as well as per package? '''No''' for the time being. * Comments will be 'flat' and '''not''' threaded (at least for the present) * Comments should be plain text and so html should be escaped (in the future we might look to use some kind of formatter such as markdown)" 1157374505000000 1185472236000000 2638 enhancement toby toby ckan-v1.8 closed fixed add a get from request param helper function 1341844597000000 1342112772000000 956 enhancement kindly kindly ckan-v1.4-sprint-2 closed fixed Add a resource group table. "We need a way to group resources by various different factors. The most important of which is timeseries. The model will look like Package resource_group resource The use cases to this are outlined in this email. http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html " 1296821112000000 1299489084000000 1754 enhancement seanh seanh ckan-sprint-2012-02-06 closed fixed Add activities for extras When an extra is added to or removed from a dataset or is updated, an activity should go in the dataset's and user's activity streams. 1328528318000000 1328528330000000 1291 enhancement kindly kindly ckan-sprint-2011-09-12 closed fixed Add activity and activity detail tables. "Add tables described in http://ckan.okfnpad.org/notifications " 1314267572000000 1315950394000000 1686 enhancement seanh seanh ckan-sprint-2012-02-06 closed fixed Add activity events for when a dataset is added to or removed from a group 1326993692000000 1328527370000000 2615 enhancement seanh seanh ckan-v1.8 closed fixed Add activity stream activity when a user starts to follow something 1340874249000000 1341235002000000 1631 enhancement seanh seanh ckan-sprint-2012-01-23 closed fixed Add activity stream events for new/changed groups 1326304020000000 1326736381000000 1625 enhancement seanh seanh ckan-sprint-2012-01-23 closed fixed Add activity stream events for new/changed users This requires adding a logic function for emitting an activity stream event, and then editing the logic functions for creating or updating users and making them call the new emit event function. This same emit event function can later be used to emit activity stream events for other types of object as well. 1326187794000000 1326736328000000 1666 enhancement seanh seanh ckan-sprint-2012-02-06 closed fixed Add activity stream events for tags "...and show them on the tag pages. This means adding tag_activity_list() and tag_activity_list_html() logic functions, adding the templating to render tags as HTML, and adding the tag HTML into the tag pages. What exactly should go into a tag's activity stream? As with datasets we may want a more complex SQL query, where events from objects related to the tag (datasets, users, etc.) appear in the tag's stream." 1326795985000000 1328527113000000 3024 enhancement seanh closed fixed Add activity stream tab to dataset read page 1353949986000000 1355141087000000 2997 enhancement seanh ckan 2.0 new Add activity streams to dataset pages Add an activity stream tab to dataset pages, like we have on user profile pages. Dataset activity streams are already implemented in the backend. 1350484306000000 1350484306000000 2998 enhancement seanh ckan 2.0 new Add activity streams to group pages Before we do this we should implement smarter group activity streams (i.e. what activities should appear in a group's activity stream?) and following of groups. 1350484394000000 1350484394000000 2999 enhancement seanh ckan 2.0 new Add activity streams to organization pages Before we do this we need to merge the organizations branch into master, we need to implement smarter activity streams for organizations, and we need to implement following of organizations. 1350484457000000 1350484457000000 1667 enhancement seanh seanh ckan-future new Add an extension point for rendering activity streams "Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML. There needs to be an extension point where extensions can register their own rendering functions for particular activity types." 1326796151000000 1338204295000000 782 defect pudo rgrp ckan-v1.3 closed fixed Add and remove users from sysadmin role from the command line "Need to be able to create (and remove) sysadmins easily (used to be able to do this via ini file but this functionality was removed a while back in authz system reorganization it seems). Suggest creating a new paster command as first step. {{{ # user identifier = openid (?) paster sysadmin add {user identifier} paster sysadmin remove {user identifier} }}} Suggest this also be added to main CKAN docs. This should go into metastable as well as default so that it goes into the v1.2 release (can do this via a transplant from one branch to the other)" 1289205287000000 1289218661000000 1673 enhancement ross ross ckan-sprint-2012-02-06 closed fixed Add approval status field to groups "Super #1669 As groups need to be authorised in some cases, particularly publisher profiles, we need a field to denote how this is managed. We should add an 'approval_status' field that will be used to denote its status. By default this will be ""approved"" but this will allow us to override it with a ""pending"" or ""denied"" status. The latter should also change the main status of the Group." 1326802753000000 1328526644000000 1795 enhancement ross ross closed wontfix Add approval_status to Package (as for Group) "The Package model should have a approval_status as the Group model does. See migration 049." 1329234060000000 1338206587000000 91 enhancement dread rgrp v0.10 closed fixed Add author and maintainer attributes to package "Add the following attributes to package: * author, author_email * maintainer, maintainer_email Gives us full compatibility to: http://docs.python.org/distutils/setupscript.html#additional-meta-data Column ordering - should come after name, title, url, download_url." 1249049780000000 1250864156000000 2566 enhancement toby shevski demo phase 1 closed fixed add back buttons to create dataset process 1340039366000000 1340373308000000 2329 enhancement rgrp ckan 2.0 assigned Add back in RSS/Atom links on relevant pages E.g. on dataset page, on revision page, on user page and on search results. 1335442540000000 1346664271000000 2775 enhancement toby aron.carroll demo phase 4 new Add bin/less to paster serve command "Ideally the ./bin/less command would be run when the server is started. * Also it would be good to have a paster command to build the production CSS with {{{ ./bin/less --production }} * The command could also detect missing node binaries and redirect to the documentation." 1343685686000000 1344543962000000 2991 enhancement seanh ckan 2.0 new Add blocks to header.html template Currently if you want to customise the header.html template you have to copy the whole thing, it isn't broken up into blocks like other templates are. 1350385820000000 1350385820000000 1522 enhancement kindly kindly ckan-sprint-2011-12-19 closed fixed Add capacity to member table. Need to add capacities to member tables. 1323172610000000 1324333827000000 2201 enhancement rgrp ckan-v1.7 closed duplicate Add citation info to Dataset and Resource page "Suggest: %title. %author. Retrieved %date. %site_title. For resource: %title = %dataset_title. %resource_name." 1330765080000000 1330765400000000 128 enhancement dread rgrp v0.11 closed fixed Add ckan_url attribute to REST JSON representation of a Package "Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN. Cost: 30m" 1253866713000000 1275694573000000 682 enhancement dread dread ckan-v1.4-sprint-1 closed fixed Add ckanclient tests to buildbot Some work done on this to make tests run independently. 1286386769000000 1297358266000000 2437 task icmurray icmurray ckan-v1.8 closed fixed Add coding standards to CKAN docs "* [ ] Write up python coding standards as discussed at dev-meetup into CKAN docs. (Mostly written-up as part of dev-meetup notes.) * [ ] Collate frontend coding standards (http://aron.github.com/ckan-style/styleguide/) into same doc(s). " 1338196895000000 1343124799000000 2701 enhancement toby shevski demo phase 3 closed fixed add comments to datasets disqus?? 1342620406000000 1343949913000000 1720 enhancement johnglover johnglover ckan-sprint-2012-02-20 closed fixed Add converter functions to convert tags (with a given taxonomy) to/from tags field Part of #1698 1327943471000000 1328625423000000 2462 enhancement johnglover johnglover ckan-sprint-2012-06-25 closed duplicate Add converter to rename resource_type field to type 1338212065000000 1338309596000000 2994 enhancement seanh ckan 2.0 new Add dataset and user popovers throughout the site "If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button. These popovers should appear wherever datasets or users are listed, throughout the site." 1350483964000000 1350483964000000 2743 enhancement toby shevski demo phase 2 closed fixed add dataset button within group "Currently the 'add dataset' button on a group page http://s031.okserver.org:2375/group/another-group-in-the-wall goes to edit group. This is confusing since the interface for this is no way streamlined or helpful. Can we get this to go to the create dataset process instead (ideally with the group pre-chosen) for now? Further on will need to re-think & plan the workflow for publishers which will be a large part of new work" 1343125202000000 1343131092000000 2419 enhancement aron.carroll aron.carroll closed fixed Add dataset form needs hooking up to action/controller Currently for some reason I'm not getting the errors/data dict in the package/new.html template. 1337793401000000 1343220027000000 2665 enhancement aron.carroll toby demo phase 2 closed fixed add dataset stages need to be form buttonable "the stages snippet needs to be inside
    so I can make some links buttons - eg stage 1 -> stage 2 I want to save the dataset not just move to add resource if you can just move it within the form html leaving everything else the same - I can then add buttons and pass back for styling" 1342023971000000 1342088650000000 2905 enhancement johnmartin demo phase 5 new Add dataset URL key behaviour "On step 1 for adding a dataset can we add a ajax ping that checks if the URL that is generated by the JS is taken? Behaviour should be as follows: 1. User types title in box (as current) 2. URL key is regexed by JS (as current) 3. The JS does a ajax call to check if the generated URL key is valid. The ajax callback returns a JSON dump which contains what the URL key should be (e.g. if 'test' was taken it returns 'test-1') 4. If the URL key returned in the ajax is different than the one generated by the JS then: 4a. If the user hasn't clicked the edit button then: it automatically updates the URL key field with the URL key supplied in the ajax 4b. If the user has clicked the edit button then: a form error next to the input appears saying that the URL key is taken and within that error is suggests the URL key returned by the ajax as alternative (the user can click said URL key within the error to automatically choose that one) The URL key isn't reserved by the ajax and upon form submission the usual validation should still apply." 1346855283000000 1346855283000000 2628 enhancement seanh seanh closed duplicate Add docs for upgrading a source install 1341585472000000 1343392717000000 2932 enhancement seanh ckan 2.0 new Add docstring to system_info.py What is the system_info table for? It looks like a way to override config file settings in the database, perhaps so that admins can use a web interface to change site title etc. Could do with a docstring explaining the intended purpose of the table. 1347893429000000 1347893429000000 2921 defect seanh ckan 2.0 new Add docstring to top of lib/extract.py file "I think it couldn't hurt for this module to have a docstring at the top of the file explaining what the module is for. I know from setup.py that it's there to provide the extract_ckan() string extractor function for Babel, but I think looking at the file on its own it's not so obvious. Also a couple of other small fixes: jinja2_cleaner looks like a helper function only meant to be used by extract_ckan(), might make things clearer if it was called _jinja2_cleaner(), just so it doesn't look like a public function the module wants to export. jinja_extensions seems to be a module-level variable that is only used in one function, could be moved into the function, unless you think there might be more functions that want it in future." 1347530407000000 1347530407000000 445 task johnbywater johnbywater ckan-v1.2 closed fixed Add dumps to DGU SLA "Thinking about putting something like this: Dumps Interface (this is NOT going in the API, it is another interface of the service): - resources: Daily Dump Folder Daily Dump File - locations: /dump/hmg.ckan.net-YYYY-MM-DD.[json|csv].zip - data format: Dump Folder is a HTML document containing a table of links to Dump Files Dump File has either JSON or CSV format. - status codes (200, 404, 500) " 1282299380000000 1283250620000000 2594 enhancement johnglover johnglover ckan-ecportal closed fixed Add EU27 to EC ODP geographical coverage field EU27 option seems to have been removed at some point, needs to be added back to geographical coverage vocabulary and tested. 1340638576000000 1340716640000000 2370 enhancement seanh seanh ckan-v1.9 assigned Add examples of all config settings in default config file The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there. 1336042237000000 1340624662000000 423 task dread johnbywater closed fixed Add examples to documentation about uploading data 1281435260000000 1282812768000000 1147 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed Add expired_id to all revision tables. "Revision tables need expired_id to make querying history AND pending changes more efficient. This involves making a session extension and a large table migration." 1305839833000000 1307957556000000 1738 enhancement kindly kindly ckan-sprint-2012-02-06 closed fixed Add extension point to search index, so can modify dataset dict before index. It is useful for extensions to add or change items that go into the search index. Add an extension point for this. 1328031572000000 1330086105000000 2477 enhancement seanh shevski ckan-v1.9 accepted Add extensions section to readthedocs Need a central up to date place for currently supported extensions (closest we have is http://wiki.ckan.org/List_of_Extensions) & their documentation. I suggest this gets added to a dedicated section of docs.ckan.org 1338393244000000 1343124123000000 1741 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Add extention point to dataset view. "Need to add extension point to change the data_dict that passed to the dataset view template. This extensions modify what data gets shown. Tasks include: * Fix up data_dict and templates so that the pkg objects do not get used in the templates. This is to assure that all data passed is modifiable. (2d) (complete) * Add extension point to Ipackage controller (1d) * Test (1d)" 1328105709000000 1329750838000000 1324 enhancement kindly kindly ckan-sprint-2011-09-12 closed fixed Add extra fields to resource table "add extra fields to resource as outlined in http://wiki.ckan.net/Domain_Model/Resource Specifically (all unicode strings except for those noted): * ''name'': a name for this resource (could be used in a ckan url) * ''type'': the type of the resource. One of: file | api | service | listing * ''mimetype'': standard mimetype (e.g. for zipped csv would be application/zip) * ''mimetype_inner'': mimetype of innermost object (so for example would be text/csv) * ''size'': [integer] size of the resource (content length) in bytes. Usually only relevant for resources of type file. * ''last_modified'': [datetime] the date when this resource's data was last modified (NB: ''not'' the date when the metadata was modified). * cache_url: url for cache of object in ckan storage * cache_last_updated: [datetime] timestamp when cached version was created * webstore_url: * webstore_last_updated: [datetime] timestamp when webstore was last updated" 1315578976000000 1315948336000000 166 enhancement dread dread v0.11 closed fixed Add extras editing to ckanclient Ensure logging is working on server. 1256203581000000 1265891199000000 2648 enhancement shevski aron.carroll demo phase 2 closed fixed Add file upload to the demo site 1341921081000000 1342714659000000 755 task johnbywater johnbywater ckan-v1.3 closed duplicate Add filter attribute to harvest source entity 1288040506000000 1294408632000000 756 task johnbywater johnbywater ckan-v1.3 closed duplicate Add filter field to harvest source form 1288040545000000 1294408642000000 1072 enhancement dread dread ckan-v1.4-sprint-5 closed fixed Add filters to authztool It takes several minutes to print the 'rights' on DGU, which is annoying when you only want to grep for a few lines. Much quicker than grepping is to filter in the query. 1302106311000000 1302106474000000 1643 enhancement shevski ckan-backlog new Add fixed tags to thedatahub for better browsing Similar to publicdata.eu, want to have themed areas such as finance, environment, census, etc and country tags 1326393293000000 1326393293000000 2941 enhancement johnmartin johnmartin ckan 2.0 closed fixed Add follower support back into CKAN 2.0 Add the views and functionality of following users and groups into 2.0 1348584670000000 1350560915000000 3005 enhancement seanh seanh ckan 2.0 new Add following/unfollowing of groups: model, API, frontend, tests 1351013744000000 1351013744000000 2563 enhancement aron.carroll aron.carroll demo phase 1 closed fixed Add font based icons to the demo theme "Allowing them to be displayed in many sizes, colours and resolutions. http://www.zurb.com/playground/foundation-icons" 1340035831000000 1341501320000000 2914 defect seanh ckan-v1.8.1 new Add form_to_db_schema_options() to IDatasetForm and IGroupForm This optional method is supported when creating or updating groups and packages and is used in DefaultGroupForm and DefaultPackageForm, but seems to be missing from the interfaces. Should be added with docstring. 1347445795000000 1350303928000000 275 defect dread rgrp closed fixed Add forms renderer for JsonType fixing bugs in admin interface "This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type) Cost: 0.5h {{{ from ckan.model.types import JsonType forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer }}}" 1269122740000000 1281002082000000 2267 enhancement aron.carroll rgrp demo phase 5 closed fixed Add generator=ckan tag in head Add generator=ckan tag in head of templates to identify site as generated by CKAN 1333040018000000 1343229146000000 1409 enhancement zephod zephod ckan-sprint-2011-11-07 closed fixed Add Gravatars to user profiles "Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?) [Reopened: Additional] Use gravatars in the HTML generated by helpers.py to create lists of users. Eg. on user/list and dataset/history." 1319200773000000 1324047722000000 1410 enhancement zephod zephod ckan-v1.5 closed duplicate Add Gravatars to user profiles Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?) 1319200774000000 1319213205000000 122 enhancement dread dread v0.10 closed fixed Add Group authz page 1253694842000000 1253716757000000 2883 enhancement seanh ckan-v1.8.1 new Add high level dev overview to 'For CKAN Developers' section of docs Here's a draft: https://gist.github.com/3414107 1345546355000000 1345629890000000 1434 enhancement seanh dread ckan-v1.8 closed fixed Add i18n strings from extensions Decide which extensions are important. 1320145648000000 1341245889000000 2625 enhancement seanh ckan-v1.9 new Add i18n strings from non-core but supported extensions to ckan.pot file Have to decide which non-core extensions are going to be supported first. 1341236903000000 1341236903000000 1692 enhancement rgrp ckan-v1.7 closed wontfix Add image attribute to Dataset and Group "Add image attribute for Dataset and Group which would be shown in prominent place. For dataset this could be a relevant image and for Group this would be more of a logo. I think this has very high value from a UX point of view. == Issues == * Do we need to support uploading and storing the image (or do users store elsewhere) * Could use something like transloadit service * Use our own storage facility with a reserved directory * Do we need to do resizing etc? * Maybe. Perhaps can prefer this in reasonably small size and then just size down in css for thumbnails etc. " 1327108761000000 1338204395000000 1383 enhancement kindly kindly ckan-sprint-2011-10-24 closed fixed Add IResourceChange IResourceUrlChange Interfaces. Need to add way for extensions to trigger based on resource change and just resource url change. 1318342153000000 1320142305000000 74 enhancement dread rgrp v0.10 closed fixed Add Is It Open links to package pages "On each package page in the openness field if the package is not open provide a link to is it open create enquiry page. Details: * may want to add to the url a query parameter indicating which package this relates to e.g. ?ckan-package=... * http://isitopen.ckan.net/enquiry/create/?ckan-package=... * do we provide link even if package is open (e.g. so that people can make enquiries about e.g. adding open data/content buttons ...) Cost: 0.5h" 1247828218000000 1250182938000000 1197 enhancement timmcnamara ckan-backlog closed wontfix Add JavaScript guide for CKAN "A new library, [http://blog.optimizely.com/introducing-guidersjs-an-open-source-guider-e guiders.js], has been open sourced that seems to be great at unobtrusively introducing new users to features of websites. The library drives Optimizely's website. The !GitHub repo is here: https://github.com/jeff-optimizely/Guiders-JS Some examples: [[Image(http://posterous.com/getfile/files.posterous.com/temp-2011-06-22/BnlgHDvhjpFfqbwvprsIaklqrpdgJyebfuhxajbDhmvnFgzdzzCJdvluJFki/Screen_shot_2011-06-22_at_6.42.21_PM.png.scaled500.png)]] [[Image(http://s3.amazonaws.com/files.posterous.com/temp-2011-06-22/CpvlcinkrxhtnAHfGtEHhxyGkraqlFmCjcnztFuugagBGxmBjCdwuquHpvEB/promo_gmail.png?AWSAccessKeyId=AKIAJFZAE65UYRT34AOQ&Expires=1308801019&Signature=DJWDgj7fmyq7K3ve177H7DPwKdw%3D)]]" 1308801032000000 1340020357000000 2850 enhancement aron.carroll demo phase 4 new Add JSON support for IE7 "The JavaScript just isn't going to work without it. Look into either the newer JSON3: http://bestiejs.github.com/json3/ or the old faithful: https://github.com/douglascrockford/JSON-js/" 1344940692000000 1344940692000000 2392 enhancement icmurray icmurray ckan-sprint-2012-05-29 closed fixed Add language field to dataset "An extra field on datasets called ""metadata_language"". The language that the meta-data is in. Drop-down list needs to be translatable. Default value is ""English""." 1337159501000000 1337270483000000 2705 enhancement johnmartin shevski demo phase 5 closed fixed add link to edit dataset page from edit review page "e.g. on http://s031.okserver.org:2375/en/dataset/afghanistan-election-data/resource_edit/f6331f99-51f6-44d9-95b9-b20f3b74f360 & ensure consistant with sidebar of edit dataset page" 1342621122000000 1352205954000000 813 requirement cygri ckan-v1.3 closed fixed Add link to “Register new package” to the homepage "A link to the “register new package” page would be handy somewhere on the front page. As a frequent editor, I often go to http://ckan.net in order to create a new package. I don't want to hunt around for the link to the “new package” form. The link could go into the bar next to “About / Statistics”. Or into the “More information” sidebar. Neither of these truly make sense, but it's a frequently accessed link and that's reason enough to put it somewhere onto the frontpage. " 1289995253000000 1294411047000000 1647 enhancement shevski ckan-backlog new add links to ckan discuss & dev to thedatahub In the footer as well as more clearly & directly on the About page 1326673852000000 1326707383000000 2704 enhancement johnmartin shevski demo phase 5 closed fixed add list of resources to edit dataset page sidebar when editing a dataset (e.g. here: http://s031.okserver.org:2375/en/dataset/edit/afghanistan-election-data) add sidebar list of resources for user to click & browse through taking you to the relevant resource edit pages 1342620986000000 1352205939000000 95 enhancement rgrp rgrp closed fixed Add manifest support "Support for listing files (manifest) contained within a package. Traditional manifests just list the files. Suggest in addition we have support for optional metadata in form of key, value pairs." 1249981842000000 1251454716000000 573 story johnbywater closed duplicate Add metadata entity to harvesting queue 1284045353000000 1284220987000000 1592 enhancement amercader amercader ckan-sprint-2012-01-09 closed fixed Add metadata_modified and metadata_created to package_dictize output The dict returned by package_dictize does not include metadata_modified and metadata_created. These are really useful properties, so it's worth having them on the standard package dict representation, which is used in several places, like at the template level. 1324488909000000 1326304321000000 2210 enhancement ross ross ckan-sprint-2012-03-19 closed fixed Add method to group for adding members "It is necessary to add a helper method to Group for adding members to that group so that - nobody needs to work with the Member model - It handles existing membership and existing membership with a different capacity" 1331028470000000 1331549369000000 1165 enhancement nils.toedtmann ckan-future new Add multi-site support to ckan "Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3). That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's """", or tracd). Depending on the ""Host:"" header in the HTTP1.1 request, it would choose which local ckan ini file to obey. I see two ways to constitute the map hostname-to-ini-file map: * ckan reads a set of ini files, and each ini file declares which servers names it is responsible for * In a global ini file, there are directives mapping servernames to ini files. In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc." 1306413667000000 1339774466000000 1779 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Add multilingual translation table. A new table with 3 columns should be added. term, term_tranlastion, language_code. This table will be used for all translations, including tags. The table should have indexs on both the term and (term, language_code) combination. 1328548631000000 1329393759000000 295 enhancement dread dread v1.0 closed fixed Add multiple packages when editing a group "In Group edit page, you can only add one package at a time. It would be better to use some scripting to allow adding more than one at once, to make it easier and quicker when you have a long list to add. Ticket is split off from ticket:221 See also ticket:155 (duplicate)." 1271756713000000 1272384758000000 2538 enhancement seanh seanh ckanbuild accepted Add multiple-instance support to ckanbuild "Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command. May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command." 1339775499000000 1340639836000000 2980 enhancement seanh ckan 2.0 closed wontfix Add new groups and organizations functionality into legacy templates From Toby: add the functionality to legacy templates 1350297883000000 1350560252000000 1650 enhancement ross ross ckan-future closed wontfix Add new language/translation of fields for datasets. "Datasets should have a field that specifies the language used in the dataset, this should be a two character ISO 639-1 code (see http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes for list) . Also make sure all tests and migrations are resolved. [2d] Determine the best approach to having 'translation_of' references to show which dataset the current one is a translation of. This may be a translation of field, or a 'copied from' field or another form of package relationship. [2d] " 1326709392000000 1346670010000000 2703 enhancement toby shevski demo phase 2 closed fixed add new resource to existing dataset with add button on dataset page and in 'data & resource' section when there's no data 1342620885000000 1342687332000000 2683 enhancement seanh new Add no-cache header to _tracking API call's response to make sure it doesn't get cached 1342446577000000 1342446577000000 2541 enhancement seanh seanh ckanbuild accepted Add non-core extensions to ckanbuild We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include. 1339775826000000 1340639856000000 3000 enhancement seanh ckan 2.0 new Add number of followers to dataset pages User profile pages now show some interesting stats about the user: num. datasets, num. edits and num. followers. Would be nice to add some similar info to dataset pages. At least num. followers can be done (already implemented in backend). 1350484560000000 1350484560000000 2569 enhancement toby shevski demo phase 2 closed fixed add other resources to sidebar of resource pages Allow user to navigate directly from one resource page to another via resource box in sidebar 1340039608000000 1342085235000000 1012 defect kindly kindly ckan-v1.4 closed fixed Add package revision history to api Revision history information is not accessible, dgu want this. 1298887351000000 1301943113000000 703 story pudo johnbywater iati-3 closed fixed Add package to package group using the package form fieldset 1287403920000000 1289909096000000 911 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Add pager to dcat-tools/rdfsolr 1295266211000000 1296467375000000 869 enhancement rgrp rgrp ckan-v1.3-sprint-1 closed fixed Add plugin interface to allow editing and extending of config by extensions "Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc). Allowing plugins to edit the config dictionary on load would allow this (and more!). Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)" 1291989395000000 1292235262000000 2861 enhancement toby shevski demo phase 4 new Add qa breadcrumb can we add in the breadcrumb navigation to qa pages so that you can go back to http://s031.okserver.org:2375/qa from sub pages such as http://s031.okserver.org:2375/qa/dataset/five_stars 1345114046000000 1345483270000000 1694 enhancement seanh seanh ckan-sprint-2012-02-06 closed fixed Add quick functional test(s) for activity streams HTML rendering The activity streams feature is already thoroughly tested via the logic layer and API calls, but there needs to be a quick test of the rendering code, e.g. fetch a user's page (thus testing that ckan doesn't crash while rendering it) and then assert that the expected activity streams HTML is in the fetched HTML. 1327322248000000 1328465254000000 1159 enhancement pudo pudo pdeu-1 closed fixed Add RDFa to CKAN package pages "RDFa can be used as a simple way to expose linked data or at least sameAs the API version of the data but we need to make sure we do not expose anything different from the ""official"" representation." 1306337907000000 1307615133000000 92 enhancement rgrp rgrp closed wontfix Add RDFa to package pages 1249049927000000 1297344859000000 2604 enhancement amercader amercader ckan-v1.8 closed fixed Add Requests as requirement for tests "Requests (http://python-requests.org) is really useful to deal with http requests and saves a lot of time. I propose we include it as a requirement for the tests for the time being, and even for core if necessary at a later stage." 1340790229000000 1340968458000000 3025 enhancement amercader amercader new Add requests to core requirements Because yes please 1354052124000000 1354052124000000 2316 enhancement kindly ckan-sprint-2012-04-30 closed fixed add resource status api to see the state of celely tasks "add resource_status_show action function to see the state of celery tasks. Need to get information from both out task status table and the celery_taskmeta table." 1335016037000000 1338205426000000 1547 enhancement amercader amercader ckan-sprint-2012-01-23 closed fixed Add search methods (before and after) to the IPackageController interface "Time estimate: 1d Add a two search method to the IPackageController interface so extensions can: * Modify the query being sent to Solr {{{ def before_search(search_params): return search_params }}} * Modify the results returned (`after_search`) {{{ def after_search(search_results, search_params): return search_results }}} Please see this pad for implementation details: http://ckan.okfnpad.org/search" 1323778996000000 1330084379000000 2769 enhancement toby shevski demo phase 3 closed fixed add site wide related page "Can we add the related dashboard (http://trac.ckan.org/ticket/2347) to demo - central place to show off all the related stuff added. " 1343323567000000 1343734083000000 1763 enhancement seanh seanh ckan-v1.7 closed wontfix Add some simple model-level tests for activity streams 1328536285000000 1338204708000000 2298 enhancement icmurray icmurray ckan-v1.9 new Add sort-by controls to the search results page "Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page. * What should the sortBy controls/widget look like? * Which fields should be exposed? The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master. [1] https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c" 1334569162000000 1340635947000000 232 enhancement rgrp rgrp closed invalid Add strapline to CKAN logo "Strapline (?): Sharing, discovery and reuse of data and content Need to consult on this. Cost: 1h" 1262975291000000 1297246297000000 2776 enhancement aron.carroll shevski demo phase 3 closed fixed add styled text field for header title "to use on demos instead of the ckan logo & tag line. Design here: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63387712/comments#190457381 (First, text only one at the bottom)" 1343724603000000 1343824375000000 953 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Add tagline/description to template and set in configuration Have a tagline / description are under main title and set if from ckan.site_description config variable. 1296809808000000 1296809834000000 1363 enhancement johnglover kindly ckan-sprint-2011-10-10 closed fixed Add task_status table to store qa/archiever/webstore information that does not need to be versioned. "{{{ table: task_status id entity_id entity_type task_type key value last_updated state unique(entity_id, task_type, key) example keys: last_task_id }}}" 1317219617000000 1318256600000000 3008 enhancement seanh ckan 2.0 closed fixed Add tests for dashboard activity stream auth We need to test that only logged in users can see their own dashboard activity streams, and they can only see their own dashboard activity streams not those of other users. 1351600860000000 1355141103000000 1036 defect johnlawrenceaspden johnlawrenceaspden ckan-v1.4-sprint-4 closed fixed Add tests for three functions in ckan/model/user.py "create a new test file ckan/model/test_user.py add tests for the following three functions in ckan/model/user.py number_of_edits, number_administered_packages, search merged in in changeset 0046f83aedcf " 1300127840000000 1301304575000000 258 enhancement rgrp dread v1.0 closed fixed Add uuids into package in REST Add uuids into package in REST interface. 1266954722000000 1273596163000000 65 enhancement rgrp rgrp v0.9 closed fixed Add version attribute to package "It is essential packages have a version attribute to assist in tracking releases etc (and to allow compatibility with other packaging systems e.g. distutils which is being used by datapkg). cost: 2h" 1245262864000000 1246434569000000 895 defect memespring ckan-backlog new Add version number (or simular) to css/js includes query string "Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g. mycssfile.css?v=12345678" 1294343382000000 1339774593000000 1824 enhancement seanh ckan-future new Add vocabulary pages "For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page. We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar." 1329845089000000 1338204958000000 2200 enhancement seanh seanh ckan-future new Add vocabulary_id option to tag_show() logic action function Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag. 1330533678000000 1338204900000000 2781 enhancement aron.carroll shevski demo phase 3 closed fixed add way to edit homepage text to simple theming "demo.ckan.org has a small field for adding text on homepage current text: ""Welcome to the CKAN demo Try out standard CKAN functionality in a sandbox environment. Search for datasets directly from the homepage or by navigating to the Datasets search page where you can facet by tags, groups and format."" Want to make this editable for demos" 1343738523000000 1344510922000000 1095 defect kindly kindly closed fixed add way to pass in schema to logic layer. We need a way to pass in schemas to the logic layer to deal with edge cases. 1303221854000000 1310134959000000 294 enhancement thejimmyg dread closed duplicate Add/remove extra fields in Package edit form Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources. 1271756591000000 1291830960000000 3007 enhancement seanh ckan 2.0 new Adding a dataset creates multiple activities Adding a new dataset creates multiple activity stream activities, e.g. seanh created the dataset foo, seanh add the resource bar to the dataset foo, and seanh updated the dataset foo are all created when I add a new dataset. I wonder if these can be collapsed into a single activity. 1351516507000000 1351517019000000 2662 defect toby shevski demo phase 2 closed fixed adding colons in tags results in server error on demo "changing a tag from ""country-uk"" to ""country:uk"" and updating gives server error otherwise updating metadata / adding tags seem fine either we should allow colons or form should return a message indicating what characters are allowed" 1342017594000000 1342018898000000 2843 defect seanh seanh ckan-v1.8 closed fixed Adding existing datasets to organizations is broken "On the organization read page there is only an 'Add Dataset' button, not a drop-down with add new dataset or add existing dataset as on the default group page. On the organization edit page you do have the drop-down, but if you click add existing dataset nothing happens except it appends #datasets to the URL" 1344859137000000 1347270569000000 155 enhancement dread dread v1.0 closed duplicate Adding multiple packages to a group "Ability to add multiple packages to a group in one go (e.g. with 'add' link which makes drop down menu appear - so can add one after another - then submit simultaneously) Use a bit of javascript to add more dropdowns. Suggested by Jonathan Gray " 1255621779000000 1271760041000000 1140 defect dread ckan-v1.5 closed fixed Adding the package to the group is not search indexed "To reproduce: 1. {{{paster db init}}} 2. {{{paster create-test-data}}} 3. In Web UI: create new group 'tilo', which includes package 'annakarenina' 4. {{{curl http://localhost:5000/api/search/package?groups=tilo}}} results in 0 results (WRONG) 5. {{{paster search-index rebuild}}} 6. {{{curl http://localhost:5000/api/search/package?groups=tilo}}} results in 1 result" 1305718290000000 1312537257000000 470 task dread johnbywater ckan-v1.3 closed fixed "Adjust API key's HTTP header name on DGU ""live"" and ""test""" 1282310854000000 1292587187000000 2742 enhancement johnmartin toby demo phase 5 closed fixed admin authz styling "http://localhost:5000/ckan-admin/authz you need to be an admin paster sysadmin add -c dev.ini like #2741 we need other admin links you may want to make the inputs use your form macros - I've just done a basic port also we will need to get all the tables done but looks like we need to get auth groups working for that" 1343124423000000 1352206081000000 2876 enhancement toby toby demo phase 4 closed fixed Admin Config changes are not forced need to make sure these update everywhere when done 1345208451000000 1345470265000000 2798 enhancement toby toby demo phase 4 closed fixed Admin Config reset "this want to a) be a link on the left of the form like deletes b) needs a confirm like delete c) wants to work with the autoform" 1344012605000000 1345540010000000 467 story johnbywater ckan-v1.3 closed duplicate Admin configures CKAN to expect API key in named HTTP header 1282310562000000 1294411681000000 829 defect rgrp dread ckan-v1.4-sprint-6 closed wontfix Admin CRUD broken Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception. 1290695043000000 1303838115000000 1565 enhancement Rufus Pollock jilly mathews ckan-future new Admin dashboard finished? Is testing complete and ready for release? 1324293092000000 1324293092000000 2741 enhancement johnmartin toby demo phase 4 closed fixed admin index styling "http://localhost:5000/ckan-admin you need to be an admin paster sysadmin add -c dev.ini we need to get links to other admin pages etc - I think this will want to be flexible so extensions can add extra links etc - but that can come later" 1343124308000000 1352205988000000 134 defect rgrp dread v0.11 closed fixed admin interface is only available to sysadmins 1254735314000000 1255430998000000 1075 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-6 closed fixed Administrative dashboard - Edit Authorization related to System object "Roles on System object are important because admin role on system equates to being a 'sysadmin' (i.e. able to do anything). * Make users sysadmin (either as separate action or as part of editing roles on system object) * /authz subpage for editing roles on system object * Add and update user roles * Add and update authz group roles * List actions associated to roles at top of page (extra points for checkbox table with editability) * Document on http://wiki.ckan.net/Authorization what roles on System object 'mean' esp sysadmin role on System == Related Tickets == * super ticket: #833 * authz lib improvement and authztool refactor #1050" 1302279799000000 1303227982000000 422 story dread ckan-v1.3 closed fixed Administrator imports meta-data into CKAN "Requirements: * cater naturally for multiple packages with multiple resources * international encoding * large number of packages" 1281433490000000 1292586586000000 1185 defect timmcnamara ckan-backlog new Administrators can't delete packages from web UI "Administrators have ""View"", ""Edit"" and ""History"" tabs. However, I can't see a way to delete a package from the web UI. Version: CKAN.net as of today" 1308111818000000 1339774289000000 302 enhancement johnbywater dread v1.0 closed fixed Advertise a package's feed visually "Visual display of a package's feed in the Package Read (/package/xyz) and Package History (/package/history/xyz) pages. Motivation: each package's Atom feed is mentioned in the tag, which Firefox displays with a feed icon, but the other 3/4 of users may not discover it. The Repo History (/revision) page has a visual link the feed. PS It would be nice to display the feed icon in both these places too." 1272370397000000 1272453821000000 2261 enhancement toby kindly ckan-sprint-2012-04-02 closed fixed All links need to option to change where language is placed in url. "For the ecportal case we need links to be like the following (%host)/open-data/en/data/(.*)[[BR]] (%host)/open-data/fr/data/(.*) " 1332762702000000 1332840112000000 83 enhancement rgrp rgrp vdm-0.6 closed fixed "Allow ""ignored"" fields on versioned objects" "Allow 'ignored' fields on versioned objects (i.e. attributes which are not 'versioned'). This is not hard to do as we already have most of the necessary mechanisms set up in the Revisioner object. Cost: 2h " 1248339384000000 1267648301000000 1749 enhancement seanh ckan-future assigned Allow creating activity details through API Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts. 1328465406000000 1338213933000000 1188 enhancement nickstenning ckan-backlog new Allow diffing against initial (blank) package version "Currently the history page only allows diffing between different versions of a package, but there doesn't appear to be any easy way to see the changes introduced by the first version of a package. I'm requesting the ability to diff against a ""blank slate"" initial state of a project, so I can see the content of the first project commit. Not sure if this is a vdm feature, so I'm putting this ticket in against ckan." 1308153160000000 1339774275000000 652 requirement pudo ckan-v1.2 closed fixed Allow for addition and removal of groups on package/edit Much more natural place to do this, even at the risk of overloading the package edit form. 1284998134000000 1284998201000000 2956 enhancement shevski johnmartin ckan 2.0 assigned Allow for resource editing in CKAN 2.0 On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually. 1349277100000000 1349346442000000 808 enhancement pudo rgrp ckan-v1.4-sprint-2 closed fixed Allow i18n language choice on site (and possibly set automatically) "Allow choice of language on ckan instance. * Configuration variable listing language choices (could try doing this automatically but recommend against this) * Switcher option in UI (perhaps on user profile page) * Default it automatically based on browser settings (?) Think this is high value (and cost is low) Cost: low (4h) " 1289944394000000 1297783658000000 2618 enhancement toby toby demo phase 4 closed fixed allow nested js resources 1340878063000000 1347901864000000 2688 enhancement ross new Allow ordering of groups in WUI Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name. It would be better if it could be sortable by name (or reversed) or by package_count (or reversed) 1342520875000000 1342520875000000 313 enhancement rgrp dread v1.1 closed fixed Allow packages to be specified by IDs in REST interface "If a package name changes, a simple CKAN client may not be aware of this (not monitoring the push notifications, revisions or feed), so it is preferable to refer to the package by its (invariant) ID. It is still useful to refer to a package by its name though, so both should be valid arguments in the REST interface. " 1273606248000000 1275404524000000 2785 enhancement johnmartin aron.carroll demo phase 5 assigned Allow resources to be re-ordered "Not sure where this functionality should be added, possibly in one of the sidebar widgets when editing a resource? Ira, what are your thoughts?" 1343816523000000 1346235916000000 274 defect rgrp rgrp closed fixed Allow searching by any extra field "At the moment only support a very limited set of extra fields (see docs for details). Should support arbitrary extra fields (note no need to put these in special full text index though ...) Cost: 1h" 1269034562000000 1287402800000000 1112 enhancement dread dread ckan-v1.4-sprint-7 closed fixed Allow searching for all packages "{{{ GET api/search/package?q= }}} returns all packages. This is so you can filter them e.g. by openness, which is not currently possible." 1304007852000000 1304358643000000 1550 enhancement ross ckan-backlog assigned Allow simple auth via the API It should be possible to pass userid/username and api key and obtain a response from CKAN for external services that use CKAN auth. Those services shouldn't be talking to the DB directly. 1324049369000000 1346670055000000 233 enhancement rgrp rgrp v1.1 closed fixed Allow simple site-specific customization/overriding of templates "Options: 1. Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works) 1. Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist). Can do this using: " 1263000396000000 1273080352000000 2842 enhancement seanh ckan-v1.8 closed fixed Allow sort ordering of dataset listings on group pages 1344855695000000 1344855852000000 361 enhancement rgrp rgrp datapkg-0.7 closed fixed Allow specification of the default index in configuration 1277803305000000 1291135756000000 2581 enhancement toby toby demo phase 2 closed fixed allow state = 'draft' pass some stuff through the context 1340287298000000 1342074635000000 1634 enhancement seanh seanh ckan-v1.7 closed duplicate Allow users to follow/unfollow activity streams of other users, datasets and groups "This is a fairly big feature to add. Analysis: http://ckan.okfnpad.org/27 This ticket is related to #1635 (email notifications for activity streams) which is about managing a list of activity streams for which the user receives email notifications. There are decisions to be made about how the two will work together." 1326304542000000 1334587390000000 796 defect pudo ckan-v1.3 closed invalid AlphaPage for non-latin languages "The AlphaPage implementation assumes that all packages start with a latin character. This is not true for Russian, Chinese, Arabian and Persian languages etc. We should either use a unicode solution (i.e. display a full class of characters based on the current locale) or use generic paging instead. " 1289296662000000 1291640040000000 247 enhancement dread rgrp v1.0 closed fixed Alternative formats links on package page """Alternative formats"". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f). * How do we know if RDF available? Ans: see details in ticket:90. Cost: 1h" 1265890326000000 1271248813000000 453 task thejimmyg johnbywater closed invalid Analysis and planning for UKLII 1282299725000000 1294411374000000 2312 enhancement ross ross ckan-future closed duplicate Analysis of how datasets could belong to users instead of Groups "** DUPLICATE OF #2548 Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset. To resolve this it would be great if a dataset could be attached to a user directly - find out how." 1334663770000000 1340188765000000 1257 enhancement dread ckan-backlog new Anti-Spam tools "We are getting more and more spam on ckan.net and we need to improve our strategy of combating it. It is bad because google ranks who we link to (which we do want for legitimate links), and our front page contains the latest package edits, so spam is immediately very visible. Spam: * Package creation * Package edit * User creation * Group creation Systems to consider: * Automatic * captcha * bayesian scoring * Sysadmin * a tool to quickly analyse and remove packages, edits and users * Helpful users * button to click indicating spam, leading to any/all of: * add TODO item (which emails package admin and sysadmins) * quarantine the package/user so it doesn't show up on the front page and a warning is shown when you view it General thoughts: * We should learn from the wikipedia tools, as they are the experts * This is a serious problem that is only going to get worse and is a threat to the whole site." 1312283565000000 1339774129000000 2968 defect seanh ckan 2.0 new Anyone can access organization members page The button will not show if you are not authorized but browse to /organization/members/foo and you can edit the members, it does stop you when you try to save your changes, but you shouldn't be able to get to the page at all 1350296355000000 1350297070000000 2647 enhancement aron.carroll toby demo phase 5 closed fixed api bullshit button make it as a snippet calling thing like aron would love 1341920530000000 1343229185000000 1663 enhancement seanh seanh ckan-sprint-2012-02-06 closed fixed API call for creating an activity streams event The create_activity() logic function already exists, but validation and authorization need to be added before it's hooked up to an API call. 1326794844000000 1328013153000000 1495 enhancement kindly seanh ckan-sprint-2012-01-23 closed fixed API call for getting a user's public activity stream An API call that returns a user's public activity stream in JSON format. 1322495549000000 1327576099000000 1494 enhancement seanh seanh ckan-sprint-2012-01-09 closed wontfix API call for getting a user's public activity stream as rendered rtext "This could be implemented as a separate API call, or the rendered text versions of the activities could be added into the JSON returned by the existing API call. This requires setting up templates for rendering activity streams items and detail items as nice, human-readable text. There are some open questions, e.g.: Do we want the entire activity stream rendered as a block of plain text? As HTML? Or do we want a list of JSON objects, where each object contains its textual and/or HTML representations as fields? Activity stream items and their related detail items are separate objects that each have their own textual representations. For a mockup of the kind of text messages we want, see: http://datahub.pudo.org/pudo but note that this ticket is just for creating the text snippets themselves, not rendering then in an HTML page or RSS feed. Also the mockup only show activity items and not their detail items." 1322495447000000 1326109757000000 1637 enhancement seanh seanh ckan-v1.6 closed fixed API call for getting the list of activity detail items for a given activty stream item (and add test cases for it) 1326304817000000 1326737169000000 2198 enhancement zydio closed fixed API documentation is missing Storage Metadata API info Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on [https://github.com/okfn/ckanext-storage/blob/master/README.rst Storate Metadata API]. 1330421743000000 1339771453000000 1688 enhancement dread dread ckan-sprint-2012-02-06 closed fixed API for changing the permissions on an object "We need an API for changing the authorization (permissions) on a model object. This was neglected in #1253 because of various changes to authz were looming. But six months later it still hasn't been done. Suggest this is just moving _add_user_object_role from ckan/lib/base.py to the logic layer, but no doubt it will be more complicated. Estimate: 3 days" 1327061809000000 1328289101000000 1310 defect dread dread ckan-sprint-2011-09-12 closed fixed API GET of a revision that has been approved causes exception "e.g. this particular revision on ckan.net: http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c (most others have not been approved) Causes 500 error." 1315234899000000 1315820419000000 1214 defect dread dread closed worksforme API improvements (following javascript use) "Notes on the CKAN API from Aron: * Server returns text/html for errors even when ""Accept: application/json"" header is set. * DELETE package request requires a Content-Length? Should be 405 Method Not Allowed? * Tag returned as a JSON object when updating but as a string when requesting. Same with ""extras"" content. * How to remove key from ""extras"" object? Passing None as described in the docs is invalid JSON and using null doesn't work. * Tags return package names rather than ids when querying GET /tag/{id} * Doesn't return rendered_text property on package update. * Relationships use ""object"" key rather than id or package_id." 1309975222000000 1314029628000000 667 defect thejimmyg dread ckan-v1.3 closed wontfix API is slow "See message and script: http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html 1500 requests shouldn't take hours." 1285885228000000 1297069121000000 2942 defect dread dread closed wontfix API POST barfs on interesting Content-Type headers "When POSTing to the API, if specified, the 'Content-Type' header must be blank or 'application/x-www-form-urlencoded'. Otherwise we get an error like: ""Bad request - JSON Error: Could not extract request body data: Bad content type: \'; charset=utf-8\'"""" The problem is that this is a very reasonable header to send. Indeed requests 0.14 sends this particular header. This affects all versions of CKAN. This is due to webob/requests.py:1248 being pretty basic." 1348593156000000 1348611144000000 482 story thejimmyg johnbywater ckan-backlog closed wontfix API Rate Limiting 1282422756000000 1298284158000000 1174 enhancement pudo pudo pdeu-1 closed wontfix API Representation Registry "As CKAN grows, more and more optional representations of packages will become used. Besides RDF (which is the motivation for this ticket), support for DSPL, home-grown XML, or CSV listings is imaginable. To properly support this CKAN should have an extensible representation registry that can be extended when new output types become available (and without changing the API in the process). This needs to integrate in two places: * We need to add support for file format extensions to the package handler and REST API. * We need to add HTTP content type negotiation (Accept headers). To support this we should have a registry with two registers: * to map format extensions to mime types (e.g. ""json"" -> ""application/json"") * to map mime types and entity types to converter functions (e.g. 8""application/json"", ckan.model.Package) -> func(obj, mime_type) ) This should be set up on load_environment so that IConfigurer plugins can feed into it. The registry should then first be added to _finish in the REST API (which needs to be refactored to be passed the {format} part of the URL if one is given. Based on the format part and HTTP headers, an appropriate representation can be generated by the registry and then be returned to the user. Forwarding of requests to the regular WUI controllers with Accept headers set or a format specified can be implemented in a separate effort. " 1307050992000000 1307615200000000 1798 defect dread closed fixed API search in non-q fields has exception for unicode characters "You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q. For example: {{{ http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93 }}} This ""N-dash"" (Unicode character 2013) causes this exception: {{{ Module ckan.controllers.api:460 in search << if ver in u'12': # Otherwise, put all unrecognised ones into the q parameter params = convert_legacy_parameters_to_solr(params) query = query_for(model.Package) results = query.run(params) >> params = convert_legacy_parameters_to_solr(params) Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr << for search_key in non_solr_params: value_obj = legacy_params[search_key] value = str(value_obj).replace('+', ' ') if search_key == 'all_fields': if value: >> value = str(value_obj).replace('+', ' ') UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128) }}} This problem affects CKAN 1.5 and 1.5.1 only." 1329395420000000 1329395560000000 1116 defect dread dread ckan-v1.4-sprint-7 closed fixed api search loses boolean q options filter_by_openness and filter_by_downloadable options don't work when specified as URI parameters. (They do work in qjson parameters) 1304086823000000 1304358664000000 377 enhancement glen@… ckan-future closed wontfix API Should return JSON in all cases "When the API returns any response it should always retunr JSON. For example a 404 response should return something like {{{ {'status': 404, 'message': 'package somename not found'} }}} When errors are encountered when creating a package: {{{ {'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be blank', 'description':'license code must be an integer']} }}} Not authorised: {{{ {'status': 503, 'message': 'You are not authorised to create this package.'} }}} The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created. " 1280268158000000 1338206349000000 1001 enhancement rgrp rgrp ckan-v1.4-sprint-4 closed fixed API should use normal user credentials if available "When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key. The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...). This is critical to incorporating any ajax editing into the frontend. As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author)." 1298489705000000 1301310351000000 1780 enhancement kindly kindly ckan-backlog closed duplicate Api to add translations. "Add api to translation to the term_translation table. * add to logic 2d * test 2d" 1328548908000000 1328578479000000 1781 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Api to add translations. "Add api to translation to the term_translation table. * add to logic 2d * test 1d" 1328549031000000 1329393822000000 788 story johnbywater johnbywater ckan-v1.3 closed duplicate API user deletes harvest source Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?). 1289211196000000 1294410007000000 473 story johnbywater ckan-v1.2 closed API user discovers correct header for sending API key 1282312203000000 1283248736000000 471 story johnbywater ckan-v1.2 closed API user sends API key in correctly named header 1282312088000000 1283248729000000 472 story johnbywater ckan-v1.2 closed API user sends API key in incorrectly named header 1282312108000000 1282932740000000 2433 defect toby toby ckan-v1.8 new API uses name not id for some version 3 calls see https://github.com/okfn/ckan/pull/5 1337957648000000 1340099820000000 3002 defect amercader ckan-v1.8 new API v1/2 'legacy' search parameters must be escaped before they are put into a Solr query string "Just to track @tauberer patch on Github. Would be nice to write a test for it. Probably going to 1.8.1" 1350639142000000 1350639142000000 1412 defect rgrp seanh ckan-sprint-2012-02-06 closed worksforme application.js crashes when viewing a dataset with no resources "Create a new dataset, don't add any resources to the dataset yet, view the dataset in ckan, if you have firebug enabled you'll see the error ""resource is undefined"" at line 699 of application.js." 1319450625000000 1328226385000000 1276 enhancement johnglover johnglover ckan-sprint-2011-09-12 closed fixed Apply weighting to search fields in solr "For example: * name and title should have a higher weight than the notes field" 1313414353000000 1315948394000000 1573 enhancement David Raznik jilly mathews ckan-future new Apps and Ideas Estimate 2 weeks for someone to finish and test. 1324294593000000 1324294593000000 2778 enhancement aron.carroll toby demo phase 3 closed fixed apps dashboard needs some design love "apps dashboard available at http://localhost:5000/apps in main dev branch I have got this to work in the demo however a) it is a bit ugly b) the template could do with some sorting eg the dropdowns should be being created in a nicer way " 1343734040000000 1343841130000000 2829 enhancement johnglover toby ckan-v1.9 new Archiver fails on 403 http response "Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end {{{ $ paster archiver update -c ../ckan/development.ini 2012-08-09 11:01:37,636 INFO [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources) 2012-08-09 11:01:37,671 INFO [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13 2012-08-09 11:01:37,900 INFO [ckan.lib.base] /api/action/package_show render time 0.043 seconds Traceback (most recent call last): File ""/home/toby/okfn/pyenv/bin/paster"", line 8, in load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')() File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 104, in run invoke(command, command_name, options, args[1:]) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 143, in invoke exit_code = runner.run(args) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 238, in run result = self.command() File ""/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py"", line 98, in command response = app.post(api_url + '/package_show', data) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 262, in post expect_errors=expect_errors) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 243, in _gen_request return self.do_request(req, status=status) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 406, in do_request self._check_status(status, res) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 439, in _check_status res.body)) paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show) {""help"": ""Return the metadata of a dataset (package) and its resources.\n\n :param id: the id or name of the dataset\n :type id: string\n\n :rtype: dictionary\n\n "", ""success"": false, ""error"": {""message"": ""Access denied"", ""__type"": ""Authorization Error""}} }}}" 1344508484000000 1344508484000000 2736 enhancement ross ross ckan-v1.9 closed wontfix Archiver fixes "We need to check the archiver to make sure it works as we would expect after the changes in #2732: * Receive notification of a file that needs processing * Check the file's DB entry and upload the file to the configured remote service * Update the URL of the file (both in the file's DB entry and the resource) * Once we're sure the file is safely archived, we should mark the file as being archived so that we know it can safely be deleted at some point. Maybe this could/should be implemented as a service rather than a celery task?" 1343121237000000 1346670097000000 402 task pudo pudo ckan-v1.3 closed duplicate Archiving worker to back up package resources from a CKAN instance "Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system. * Naming scheme? * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? * What happens if names change * File: filename? hash? * Store hash back on ckan instance? The caching worker will consumer update notifications and fetch packages. Extra points for: * Properly checking for source file modification (Last-modified, Etag) * Using PIP VCS Backends for retrieval * OFS/S3 Storage " 1281018912000000 1296467635000000 426 task dread dread ckan-v1.2 closed fixed Ask list for suggestions improving API docs 1281436574000000 1282925305000000 1413 enhancement seanh shevski ckan-backlog closed fixed Ask users to 'add email address' when logged in "I'd like us to display a one-off banner that prompts users who log into thedatahub.org to go and update & their profile & fill in their email address (if we could do it just for those who don't have an email address, then even better) That is, once they log in, they see a banner which says: Please update your profile [here] and add your email address, so you can receive notifications, be able to reset your password and get updates Then when we build notifications or if we decide to survey people, we can actually contact them. Email address should be required as standard DR: Also, mention their Full Name too - v. useful for selecting correct user in group curation" 1319453108000000 1330020486000000 729 bug pudo closed fixed Assets to be loaded from assets.okfn.org, not m.okfn.org Move from hetzner box to s3. 1287736519000000 1288012854000000 450 task dread closed fixed Assist with proxying of static media for forms John did this a while ago. 1282299573000000 1287997487000000 452 task dread closed fixed Assist with using package update error form 1282299647000000 1287997540000000 195 enhancement rgrp rgrp v0.11 closed fixed Association listings should be alphabetically sorted (e.g. a group's packages) "Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name. Cost: 2h" 1258475448000000 1258585500000000 2709 enhancement icmurray markw new Atom feeds are undocumented There doesn't seem to be any documentation yet for Atom feeds. 1342624310000000 1342626212000000 1120 enhancement tsm ckan-backlog new Atom feeds of each tag "Tags could/should have an Atom feed. This would mean that every edit to relevant packages could be easily monitored. See [1]. [1] http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001162.html" 1304309285000000 1339774568000000 272 enhancement rgrp rgrp closed fixed Atom/RSS feeds for individual tags and groups "I want to be able to easily check when a package is added (or removed?) from a given tag or groups. To do this I would like access to an Atom/RSS feed for a given tag or group. * This would allow people to embed feed widgets in other places" 1268988652000000 1340631775000000 787 task thejimmyg thejimmyg ckan-sprint-2011-09-12 closed fixed Auth API "Auth Proposal Use case: We'd like an authenticated and authorized Drupal user to be able to edit/delete packages using the API To do this a user would need a CKAN API key so we need some way of Drupal telling CKAN who a user is so that they can get their API key. Proposed Implementation ----------------------- A user visits the CKAN API key page to get their key. Because CKAN is at catalogue.data.gov.uk (a subdomain of the Drupal site) it can read Drupal cookies. If there is no ``DRXtrArgs`` or ``DRXtrArgs2`` cookie, we know the user isn't signed in so we redirect them to Drupal to sign in. WISHLIST: It would be really nice if we could pass the URL to redirect back to Drupal so tha Druapl can send them back to the CKAN API key page Either way, they get back to CKAN and now the cookies exist. When the first HTTP request header is sent, CKAN will read the Drupal session ID and then call a Drupal API, server to server. TODO: Implement an API on the drupal server which accepts a Drupal session ID as an argument and returns the username and credentials, but only if the request if from the CKAN server (perhaps we specify an API key in the request) If the session is valid CKAN will set its own auth cookie and show them the page with the API key. CKAN only considers a user signed in if both the CKAN cookie *and* Drupal session ID are present. If at any time they sign out of Drupal the Drupal session disappears so they will be signed out of CKAN too. Now the user has an API key they can use the standard CKAN command line API tools. The API key is the only thing the user will need to use the CKAN API. This poses a problem. What if a user is removed from Drupal but still has a CKAN API key? Two solutions: 1. API keys will be set to only be valid for the length of a Drupal session so that a user will only be able to use the write API for a little time after credentials are revoked 2. CKAN provides an API to allow Drupal to revoke keys Option 1. seems easier to me. If we choose this we will write a ``get_api_key`` command line tool so that Drupal user can write things like this: :: datapkg list_packages --api_key `get_api_key` The ``get_api_key`` function will prompt for username and password and then perform the steps necessary to get an API key." 1289211035000000 1315821118000000 855 defect pudo dread closed invalid Authenticator tests The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests. 1291723967000000 1311176988000000 2811 defect seanh ckan-v1.8 closed fixed Author doesn't show on dataset read page Edit a dataset and add an author, dataset read page still says 'author not given'. If you fill out both the author and author email fields then it works. 1344415861000000 1349869110000000 2587 enhancement shevski toby demo phase 1 closed fixed author/maintainer not shown if email missing is this the expected behaviour seems crap to me, should we just see the author but just no link 1340369450000000 1340376975000000 1027 enhancement pudo pudo ckan-v1.4-sprint-3 closed fixed Authorization checks on all controller actions We want to have authz checks on all controller actions so that we can lock down CKAN to a login-only mode. 1299666256000000 1299682082000000 835 enhancement pudo pudo ckan-v1.3 closed fixed Authorization CLI and Documentation "As a sysadmin I want to take CKAN out of wiki-mode into normal mode. In normal mode visitor and logged in users cannot create or add packages. Suggest put this as an internal command in authz system and add to cli as paster command." 1291029108000000 1291644820000000 647 requirement pudo pudo ckan-v1.3 closed fixed Authorization Groups for group-based access control allow group-level authorization on all system entities 1284889418000000 1287391981000000 1052 defect dread dread ckan-v1.4-sprint-4 closed fixed Authz holes "No authz on: * Group creation/edit/listing * Package relationship create/edit/delete" 1300709144000000 1300895410000000 1050 enhancement thejimmyg johnlawrenceaspden closed invalid Authz lib improvement and refactor of ckan/lib/authztool.py "Refactor ckan/lib/authztool.py so that the relevant methods are independent of the command line interface. The extracted methods should live in a new file ckan/authz.py. authztool.py should probably move into cli.py and will just do command line parsing and printing and use ckan/authz.py. The updated web gui for authz will also use this code. Tests should be made. There's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests. all to go on a branch feature-1050-refactor-authtoolz == Optional extras == * Rename ckan/authz.py to ckan/lib/authz.py or even ckan/logic/authz.py " 1300451937000000 1315394117000000 1253 enhancement amercader amercader ckan-sprint-2011-10-28 closed fixed Authz refactor We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system. 1312199491000000 1313401367000000 1038 enhancement dread dread ckan-v1.4 closed fixed Authz tool - operate on all packages at once Add 'package:all' to authz tool to allow mass changes of authz. 1300212788000000 1300212841000000 2323 enhancement ross ross ckan-backlog closed fixed Auto-complete in organizations "Apparently auto-complete in Organizations may not work. Check and fix. See https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5" 1335274354000000 1340188400000000 803 story thejimmyg johnbywater closed invalid Auto-generate next migration version unit and test stub files As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually. 1289486678000000 1314031451000000 2984 refactor seanh ckan 2.0 new Auto-populate context with user, model, session, etc. "From Toby: make the context auto-populated with things like user/model/session if not user supplied - especially for extensions (pre 2.0) maybe via a helper or else a decorator on the action - helper seems nicer maybe done via get_action()" 1350298101000000 1350298101000000 2404 enhancement rgrp ckan-future assigned Auto-screenshot for related items "As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere Implementation: (?) Use phantomjs or something like http://snapito.com/ Investigate for 1.8, will be needed for TDH Oct." 1337583347000000 1346669526000000 2696 enhancement aron.carroll shevski demo phase 2 closed fixed autocomplete for resource format in create/edit dataset forms 1342619938000000 1343138454000000 2695 enhancement aron.carroll shevski demo phase 2 closed fixed autocomplete for tags in dataset creation / edit form 1342619891000000 1343138240000000 816 enhancement johnglover cygri ckan-sprint-2011-10-28 closed fixed Autocomplete for the resource format field "The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource. This would help improve data quality and consistency." 1289996574000000 1319812324000000 815 requirement cygri ckan-v1.3 closed duplicate Autocomplete for the search field The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page. 1289995951000000 1294410951000000 308 enhancement rgrp rgrp closed duplicate Autocomplete package names & tags in package search "Extracted from ticket:216. Dubious of its merit." 1273050549000000 1275302577000000 332 defect pudo rgrp closed fixed Autocomplete tag on click "During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box. " 1275302887000000 1280743320000000 2807 enhancement aron.carroll toby demo phase 3 closed fixed autocomplete.js error "@aron, edit dataset has a js error http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada TypeError: options is undefined [Break On This Error] this.lookup(options.term, options.callback); autocomplete.js (line 231)" 1344266228000000 1344270190000000 25 enhancement johnbywater johnbywater v0.6 closed fixed Autocompletion (and suggestion) of package tags when adding tags to a package "== Notes == * Currently have a text box where users can add space separated tags * Tags should auto-complete in a manner similar to delicious * would also be good to have a list of suggested tags. This could either be: * [simpler] a list below the tag box chosen by existing popularity or * [more complex] a list of suggestions that updates as you add tags (based again on popularity but 'filtered/reranked' to reflect choices so far). * For first attempt to should probably just go for the simpler option." 1152551313000000 1204131522000000 2681 enhancement aron.carroll shevski demo phase 2 closed duplicate autofill on resource format "This is pretty key for standardization. Also would like us to change the grey text to be: e.g. CSV, JSON, [more examples... i have no idea what they key ones are tbh]" 1342440406000000 1343135874000000 213 enhancement dread dread v0.11 closed fixed Automate CKAN deployment fabric script to automate ckan deployment 1260811707000000 1265891061000000 1398 enhancement kindly rgrp ckan-sprint-2011-11-21 closed fixed Automated conversion of resource data into webstore " * Possible sibling / requirement: #1397 (resource archiving) * Setup test environment. 1d * Make task to upload files to webstorer. 3d * This includes writing back webstore_url and updated date. 0.5d * Type guessing of file format. 1d * Make cron to upload all un webstored resources. 0.5d * Testing for errors by running cron 2d * Make nice way to deploy celery 1d. * Documentation. 1d * Deployment. 1d " 1318529724000000 1321827970000000 186 enhancement rgrp rgrp closed duplicate Automated upload to archive.org s3 "(Follows on from ticket:107). We want to provide facility for users to automatically upload material. * Create an upload page * Store the material on archive.org using s3 - http://www.archive.org/help/abouts3.txt * Main issue is how we cache large datasets ..." 1257803430000000 1296341182000000 776 enhancement rgrp dread closed duplicate Avoid generating vdm warnings "We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided? {{{ 2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object 2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object 2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object 2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object 2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object 2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object }}}" 1288372797000000 1297066840000000 730 task rgrp closed duplicate Back up package data from all CKAN packages to storage.ckan.net "Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net. * Naming scheme? * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? * What happens if names change * File: filename? hash? * Store hash back on ckan instance?" 1287737109000000 1291139609000000 544 requirement pudo ckan-v1.3 closed duplicate Backport facet browsing to CKAN 1.2 "This is in IATI, would be nice to have in generic CKAN. * http://bitbucket.org/okfn/iati/changeset/378431974c76" 1283267292000000 1291638966000000 1007 defect pudo pudo closed wontfix Backport WDMMG translations for extensions i18n "Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class. cf. https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16 " 1298630621000000 1340626203000000 329 defect rgrp dread v1.1 closed fixed Bad dates cause exception on Gov form "== Reproduction == Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message. Affects all versions of CKAN. == Why it's happening == The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught." 1275060617000000 1275079189000000 2512 enhancement toby ross ckan-sprint-2012-06-25 closed fixed Bad link on demo site. On the group read page http://s031.okserver.org:2375/group/nhs the add dataset button goes to the edit group page instead of the add dataset url. 1339518597000000 1339576809000000 844 defect wwaites wwaites closed fixed bad mail config ckan.net/ckan.org "ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example: {{{ B30E6D9F 2873 Sat Dec 4 14:05:40 MAILER-DAEMON (delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out) no-reply@ckan.net }}} both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null" 1291556753000000 1296340486000000 1263 enhancement dread dread closed fixed Bad name for logic action in API causes Exception "A bad action name: e.g. curl http://test.ckan.net/api/action/user_showaa -d '{}' causes 500 response. Occurs only on 1.4.3 beta." 1312479020000000 1314029279000000 2231 defect amercader amercader ckan-sprint-2012-04-02 closed fixed Base harvester broken on 1.6.1 "On its current version, ckanext-harvest passed a custom schema to package_create and package_update to allow defining an id for new packages (the same as in remote ckan instances) [1] Recent changes in the logic layer have made that the context object is no longer directly read from the context, but only get from plugins [2]. This makes the schema sent from ckanext-harvest to be ignored and the harvesting fails because the id field is not expected {{{ ValidationError: {'Id': u'The input field id was not expected.'} }}} We need a way to either: 1. Be more flexible accepting schemas on the logic functions 2. Modify the harvester plugin to implement the necessary interface [1] https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115 [2] https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38" 1331815499000000 1333374735000000 57 enhancement rgrp rgrp v0.8 closed fixed Basic datapkg integration with CKAN "Using datapkg a user can: * list packages on CKAN * get info about package on CKAN * register and update a package on CKAN " 1238750826000000 1239018717000000 1552 enhancement dread dread ckan-sprint-2011-12-19 closed fixed Be able to set password on the command line when creating users (instead of using the prompt) - needed for deployment 1324056178000000 1324056240000000 1471 enhancement seanh dread ckan-v1.8 closed fixed Be compatible with new ubuntu release in April "Nils flagged up that Ubuntu launches another LTS in April which we should probably move to. This ticket is to get the ball rolling with what we do. We can always include in our package the exact package versions of deps that we want, but it is good to be in line with the release if possible. Here's how to see the packages included with it: http://packages.ubuntu.com/search?keywords=python-pylons&searchon=names&suite=all§ion=all Pylons 0.9.7 -> 1.0-2 Requires a small amount of work SQLAlchemy 0.7.3-2 OK Babel 0.9.6-1 OK Genshi 0.6-2 Perfect Another alternative is to move to another web framework, especially minding that Pylons is not being further developed." 1321552345000000 1340810270000000 1665 task seanh seanh closed fixed Begin doing research into eurovoc How big is it? How are we going to store it? etc. 1326795828000000 1329742600000000 2319 enhancement rgrp ckan-v1.8 closed wontfix Better auto-complete for groups on dataset edit page Use jquery chosen? 1335211353000000 1340624083000000 2924 defect seanh ckan 2.0 new Better docs for trans js command, and add to release process "Add a better docstring to for trans js command explaining what it does and why, and how to use it. Also add this command to the CKAN Release Process as it's needed there" 1347530671000000 1347530671000000 2931 enhancement seanh ckan 2.0 new Better docstring for app_globals.py "''' The application's Globals object ''' is not very informative. " 1347891194000000 1347891378000000 2922 defect seanh ckan 2.0 new Better docstring for CKANInternationalizationExtension "I'm unsure about what's going on here. As I understand it, when we run `python setup.py extract_messages` it's going to use the extract_ckan() function from ckan/lib/extract.py to process the HTML files. For the HTML files that are jinja2 templates, extract_ckan() will call jinja2_cleaner() which will call regularise_html() on the strings. So the strings are regularised when they are extracted from the source files. But then we have the parse() method of CkanInternationalizationExtension also calling regularise_html(). I don't get what's happening here. Why do the strings need to be regularised twice? My guess is that CkanInternationalizationExtension is used when the strings are extracted from the templates at runtime, and they need to be regularised at this time in order to match them against the regularised strings in the mo files to find the translations to output? Maybe CkanInternationalizationExtension needs a better docstring saying what it does?" 1347530507000000 1347530507000000 1290 enhancement dread dread ckan-backlog closed fixed Better error when blank database "When installing CKAN, when doing ""paster serve development.ini"", lots of users encounter the error for every request: {{{ ProgrammingError: (ProgrammingError) relation ""user"" does not exist }}} This is because the database tables have not been created - they have forgotten or missed the ""paster db init"" step. Can we provide a better error to say that the database is not initialised yet? == Implementation options == 1. At the start of every request we reflect the database tables and check they are there. This is rather expensive! 2. Uncached requests to the home page start with a cheap database query. If there is an exception then return this error about database setup. I really like Drupal's page for this that has in large letters that the site is currently off-line. Below a line, in small letters, there are developer suggestions on what is wrong and where to look to fix it." 1314264255000000 1314270894000000 2254 enhancement toby toby closed fixed better hooks for templates "TD look at issues and then email ckan-dev meeting notes from 21-3-2012 general ideas non specific Problem: * We want to remove the Transformer usage due to performance and its close tie to genshi * Want a standard way for core code and extensions to put code into the templates where *templates request it* (templates control insertion rather than extensions ...) * Stacking versus overwriting on entry points * Relationship to h.requires('jquery', version) - http://codex.wordpress.org/Function_Reference/wp_enqueue_script Options: * placeholders or helper methods * {{user _info}} {{sub_menu}} is this equivalent to * [xyz arguments] -> h.load_snippet('xyz', ...) * ${h.snipit()} " 1332341333000000 1338210316000000 2852 enhancement toby shevski demo phase 3 closed fixed better message for trying to refresh inactive sources in harvest "Currently if you try to refresh (i.e. schedule the harvester to re-run) an inactive source (one who's state is 'withdrawn' - e.g. http://s031.okserver.org:2375/harvest/3feea629-32d8-4431-9957-9f31d32864f1) you get the error message: ""An error occurred: [Can not create jobs on inactive sources]"" This could be better and not red, e.g. and info box (maybe yellow) saying that ""Cannot re-run harvesting on inactive sources. Please first change the status to 'active'""" 1345048317000000 1345106515000000 2480 enhancement markw ckan-v1.9 new Better message when dataset has no resources "If a dataset has no resources the resources list currently says '(none)'. Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'" 1338453093000000 1339771086000000 511 requirement dread ckan-v1.3 closed worksforme Better warnings and errors when using API 1282754677000000 1297075354000000 2711 enhancement johnmartin toby demo phase 5 closed fixed bin/less dies for toby it would be nice if this didn't fall over when the dir is removed maybe a linux only issue? 1342631508000000 1352205968000000 363 defect kindly dread ckan-backlog closed wontfix Blank revisions "Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they? They appear on the 'Recently changed' list on the home page with an empty 'Packages' column." 1278947772000000 1310125872000000 1581 enhancement mark.wainwright@… johnglover ckan-future new Blog post about Google Analytics extension for CKAN "The CKAN Google Analytics extension has been updated to work with the latest version of CKAN, could make for a nice blog post. Can ping John Glover in January for any details required. Key link is: http://thedatahub.org/analytics/dataset/top though this should probably move to be under stats (e.g. http://thedatahub.org/stats/usage)" 1324402800000000 1325474274000000 2747 enhancement toby shevski demo phase 2 closed fixed Breadcrumb should use title not URL of dataset "For example here: http://s031.okserver.org:2375/en/dataset/new_resource/gold-prices The breadcrumb shows ""gold-prices"" which is not the title of the dataset" 1343211742000000 1343215679000000 1527 enhancement icmurray icmurray ckan-sprint-2012-01-23 closed fixed Break DGU package edit form into sections " - use javascript to selectively hide/show parts of the form - there's no validation between steps at this stage. It's still a ""big save button at the end""." 1323174829000000 1327589456000000 1258 enhancement kindly kindly ckan-sprint-2011-10-28 closed fixed Bring purge revision into ckan repo from vdm In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory. 1312289539000000 1319812452000000 986 defect wwitzel3 thejimmyg ckan-v1.4-sprint-2 closed fixed Broken link report from the ckanext-qa code "Should have the following features: * A list of all packages with broken links on the resource * Under each packages a description of any resources with broken links to include: * Link * Description * Format * First found to be broken date [If you can't do the last one yet, don't worry]" 1297683994000000 1297812401000000 1719 defect rgrp dread ckan-sprint-2012-02-06 closed fixed Broken links for non-Gravatar use icons - 0.25d "Super ticket: #1506 e.g. http://thedatahub.org/user The users with Gravatar have their nice user icons, but the majority of users (without Gravatars) have 'broken link' symbols. Same problem wherever users are shown, such as the dataset history pages. Broken on 1.5.2b - currently on thedatahub.org." 1327938788000000 1328541630000000 5 enhancement johnbywater johnbywater v0.3 closed fixed Browse list of packages and select one to view "== As a == Visitor == I want to == Browse a list of packages resulting from a search or browse request (see other use cases) == So that == I can select one of the packages to view in more detail (-> viewing an individual package ticket:6) == Notes == * When browsing a list of packages you should be able to see summary information about the package such as title (though this may be shortened in order to conveniently fit the list * The list should be broken up into pages so that the number of packages per page should be kept to a reasonable number (<= 50). Response time should be kept reasonable " 1152549884000000 1185473622000000 183 enhancement rgrp rgrp closed worksforme Browse packages by rating At moment order packages by title. 1257534606000000 1290604779000000 520 requirement pudo pudo iati-1 closed fixed Browseable web interface onto the data * e.g. find/browse by country and by publishing entity and by donor 1282893270000000 1283538080000000 1417 defect dread dread ckan-sprint-2011-10-24 closed fixed Browser language detection doesn't work "In Firefox: * Edit | Preferences | Content | Languages - put 'de' at top language * Clear cookies * Load thedatahub.org - it should be in German, but it's in the default language - English." 1319539010000000 1319651617000000 1457 defect jilly.mathews ckan-future new Bug with DataNL instance " ""when logging into http://register.data.overheid.nl/ with OpenID, the /user/me page gives a 404. CKAN version 1.3.4."" n the manual it says an API key kan be created via http://test.ckan.net/user/me /. However, when I try the corresponding http://register.data.overheid.nl/user/me, I get a 404 error (not found). " 1320925041000000 1320925041000000 1193 enhancement kindly kindly closed fixed bug with new formalchemy causing all resouces to be deleted "The new formalchemy stores dict in a new format. " 1308525655000000 1309768960000000 2318 enhancement seanh rgrp ckan-sprint-2012-04-30 closed fixed Bug with Portugese translation and Javascript "If you switch to Portugese and try to add a resource on dataset edit it will fail. This is because there is a string translation in js_strings.html that has quotes in it. When this is inserted into the file this causes a js exception which then prevents any further js processing. Specifically: {{{ CKAN.Strings.youHaveUnsavedChanges = ""Você tem alterações não salvas. Certifique-se de ter clicado ""Salvar Alterações"" abaixo antes de sair desta página.""; Uncaught SyntaxError: Unexpected identifier }}} To fix is simple i imagine: need to escape "" correctly in js_strings.html (or translations used). I have temporarily patched this live in order to allow the hackday here to continue." 1335033889000000 1335774178000000 2777 enhancement icmurray new bug: user attributes 1343726363000000 1343726363000000 314 defect johnbywater johnbywater closed fixed Bugs getting revisions from the REST API "Bug report regarding getting revisions: Getting revisons by ID (on the latest ID) GET ""http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff"" Returns a whole load of stuff (it's also quite slow - about 10 seconds) I'm expecting it to return just an empty array - am I doing something wrong here - if so could you clarify correct use of the API? GET ""http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45"" Returns the empty string - I'd expect an empty array ie [] GET ""http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45"" Returns an internal server error 500 - I think it should probably be ""bad Request"" 400 (the date is invalid) " 1273743755000000 1276523983000000 1106 defect rgrp rgrp ckan-v1.4-sprint-6 closed fixed Bugs related to routes arising from API refactor + removal of default routes "Various bugs I've been encountering: * Autocomplete of tag names no longer works (no longer works on http://ckan.net/). Appears to be due to no longer have a routes for apiv2 (i'm seeing failing calls to: http://ckan.net/apiv2/package/autocomplete?callback=callback&incomplete=a) * Incorrect url generated for API in page footer (e.g. http://ckan.net/rest/get_api) due to use of old 'rest' rather than new 'api' Latter issue was masked by existence of 'default' routes: {{{ map.connect('/{controller}', action='index') map.connect('/:controller/{action}') map.connect('/{controller}/{action}/{id}') }}} Having these is, I think, bad practice as it is better to be explicit and we should therefore remove asap. In addition I think we should be cautious about 'default' routes in core such as: {{{ map.connect('/api/rest/:register', controller='api', action='list', conditions=dict(method=['GET']) ) }}} As it makes it harder for extensions to introduce their own APIs (here one could perhaps add something at /api/rest/{my-object} but only by using before_map rather than after_map). " 1303747360000000 1303834069000000 2877 enhancement kindly rgrp assigned Bugs with datastore v2 "In progress 1. ~~[major] q does not seem to work reliably. e.g. using the setup from this gist https://gist.github.com/1930806 and doing a ?q=DE yields no results (does not work with ""q=de"" either)~~ * ~~q=second does work ...~~ 1. [major] q does not work with 2 values (see below) 2. Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT * Also weird fact that limit is returned but as as as string - should it not be an integer 3. Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...) === Multiple query values === Try a query such as: ""second UK"" and you will get 500 error: http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&q=second%20UK {{{ ProgrammingError: (ProgrammingError) syntax error in tsquery: ""second UK"" 'select ""_id"", ""id"", ""date"", ""x"", ""y"", ""z"", ""country"", ""title"", ""lat"", ""lon"", count(*) over() as ""_full_count""\n from ""4f1299ab-a100-4e5f-ba81-e6d234a2f3bd"" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',) }}} === Suggestions === Filter support: should think in more detail about this (may want to follow recline style) Simple filters in query parameters would be nice too ..." 1345250002000000 1346320395000000 1161 requirement pudo amercader pdeu-1 closed duplicate Build a simplified theme for PDEU "We need to offer a strongly simplified version, read-only of CKAN under publicdata.eu, with a focus on its role as search engine instead of a data catalogue. This ticket relates to work on the PDEU theme only!" 1306407835000000 1306408026000000 172 enhancement rgrp rgrp v0.11 closed fixed Build ckan documentation using sphinx and upload "Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible. NB: improving the documentation is another matter (as is integrating e.g. existing api docs). Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net): * http://knowledgeforge.net/ckan/ckan/doc/ * (Other options: http://www.ckan.net/doc/ http://doc.ckan.net) " 1256489019000000 1257532331000000 534 defect johnbywater dread ckan-v1.2 closed fixed buildbot config - database machine moved "Configure buildbot to use postgres db which has been moved to eu5.okfn.org pudo said: Could you perhaps also comment on http://knowledgeforge.net/okfn/tasks/ticket/466 with a hint on how to update this properly? I only edited /home/buildslave/okfn/full/build/buildandsmoke/buildandsmoke.ini which was the only occurence of the DB credentials I could find there." 1283165568000000 1288002762000000 437 bug dread ckan-v1.2 closed fixed Buildbot test failures - ascii codec "On today's buildbot: http://buildbot.okfn.org/builders/buildbot-test/builds/201 2 failures about ascii (ignore other 2)" 1282223640000000 1288004009000000 2713 enhancement aron.carroll toby demo phase 3 closed fixed button colour not set properly in variables.css "to reproduce run paster color fuchsia -c development.ini then go to http://localhost:5000/dataset/new Add button is blue gradient at the top this should be overridden in variables.less when added send this ticket to me and i'll update the color command to change it" 1342706428000000 1343656865000000 1542 enhancement dread ckan-backlog new Buttons to purge spam datasets and groups "A sysadmin should be able to easily examine a suspect group or package, determine if it was created by a spammer (as opposed to being a legitimate object that has been graffitied by a spammer) and purge it. The existing two-stage revision delete is currently unreliable and perhaps too laborious. Olav and Richard have needs along this line." 1323364930000000 1339774000000000 537 task wwaites wwaites closed duplicate Caching and Performance improvement "There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests. Introducing caching has to be done carefully and should be done in a ''standards compliant manner''. == General strategy == * Where possible, cache output within the pylons app (beaker). * Facilitate external caching in an end-user's web browser or a caching proxy * Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes). * When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data. == Tasks == These tasks should be broken into sub-tickets: * caching of parts of templates that are expensive to render (package list, tag list, group list) * caching of entire output using beaker particularly for API read operations. * need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches. * cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: http://www.mnot.net/cache_docs/#CACHE-CONTROL * Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well * Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner == Future Work == * Investigate ckanclient library maintaining a local cache as a web browser would * Investigate using a CDN like Google Storage or Amazon for serving cached data." 1283184362000000 1311178929000000 841 enhancement kindly dread ckan-v1.4-sprint-4 closed duplicate Caching docs (as a whole) "Documentation article on caching / improving performance. (To complement configuration docs.) * Different sorts of cache - beaker style, etags, package_dict in search results(?) * How each one affects performance * How to turn them on/off and configure them * Is it possible to bypass each of them in the browser or with wget/curl?" 1291308879000000 1300364333000000 668 defect thejimmyg Colin Calnan closed invalid Caching issues on API v1 "It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues. Steps to test: 1. Modify a dataset on datadotgc.ca, redirects to CKAN 2. On save, redirects to http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100 3. You can see that the Dataset has not updated correctly. Run a check on the API v1 - http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present 4. Check the v2 of the API - http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present. 5. Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either." 1285953542000000 1311176649000000 1223 enhancement pudo pudo closed fixed Caching of static files StaticURLParser can have caching - use it 1310573854000000 1310573893000000 930 defect wwaites dread closed fixed call_timing files created for every request "On a production server we produce literally millions of little files saying how long every single request took to process. They are here: {{{ hmg.ckan.net.2/pylonsdata/call_timing }}} They were added by ckan/lib/base.py in cset:da438a9085d3. I don't believe anyone uses these stats." 1296061721000000 1297066292000000 1115 defect johnlawrenceaspden closed wontfix can have two authzgroups with the same name "If you've got edit permission on an authzgroup, then you can change its name to be the same as another existing authzgroup. This causes some strange UI effects at worst, and probably causes worse problems somewhere else. Is there any reason why changing the names of existing authzgroups should be allowed? And if so, name collisions should presumably be guarded against in both the name-changing and creation functions" 1304085120000000 1324054704000000 1356 enhancement kindly amercader ckan-sprint-2011-10-10 closed fixed Can not recreate a deleted extra If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine). 1317034180000000 1318279617000000 2832 enhancement shevski demo phase 4 closed duplicate can't add a dataset to more than one group "add to groups is a drop down menu where you can only choose one needs a new UI & logic allowing user to add new groups & potentially remove from other lists" 1344521472000000 1344542984000000 951 defect adrian.pohl@… closed invalid Can't add a package to group "I can't add a package (e.g. http://ckan.net/package/ub-konstanz) to a group (e.g. http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is ""history"") nor on the group page." 1296726886000000 1314031006000000 2718 enhancement toby shevski demo phase 4 new can't add dataset to more than one group "trying to add a dataset to another group means it's no longer part of the first group http://demo.ckan.org/dataset/edit/afterfibre" 1342780550000000 1344544203000000 2266 defect dread dread ckan-sprint-2012-04-02 closed fixed Can't delete all of a package's resources over REST API Nothing happens if you set resources=[] or resources=null. 1332932504000000 1332932634000000 1479 defect dread dread ckan-sprint-2011-12-05 closed fixed Can't edit a user with a unicode email address " 1. Register User with an email address with a unicode char (e.g. u'\u044e') 2. View the User in the UI (/user/) or with 'user_show' Action API Exception: {{{ Module ckan.controllers.user:98 in read << try: user_dict = get_action('user_show')(context,data_dict) except NotFound: h.redirect_to(controller='user', action='login', id=None) >> user_dict = get_action('user_show')(context,data_dict) Module ckan.logic.action.get:488 in user_show << check_access('user_show',context, data_dict) user_dict = user_dictize(user_obj,context) if not (Authorizer().is_sysadmin(unicode(user)) or user == user_obj.name): >> user_dict = user_dictize(user_obj,context) Module ckan.lib.dictization.model_dictize:189 in user_dictize << result_dict['display_name'] = user.display_name result_dict['email_hash'] = user.email_hash result_dict['number_of_edits'] = user.number_of_edits() result_dict['number_administered_packages'] = user.number_administered_packages() >> result_dict['email_hash'] = user.email_hash Module ckan.model.user:59 in email_hash << if self.email: e = self.email.strip().lower() return hashlib.md5(e).hexdigest() def get_reference_preferred_for_uri(self): >> return hashlib.md5(e).hexdigest() UnicodeEncodeError: 'ascii' codec can't encode character u'\u044e' in position 17: ordinal not in range(128) }}}" 1321960486000000 1321961592000000 1419 enhancement dread ckan-sprint-2011-11-07 closed invalid Can't log in via OpenID "I couldn't log into theDataHub with OpenID today. I tried both Google ID and MyOpenID. Both times the login on the remote auth server went fine, but when it returns you to theDataHub you get error ""Login failed. Bad username or password.""" 1319543013000000 1319796164000000 662 defect sebbacon johnbywater ckan-v1.4 closed fixed Can't put entity that is returned by posting to package register "It's because Package carries several out-of-band values, which are snagged on the way back out. Entity get response also can't be posted. However, post response can be re-posted (because it isn't the same as the register-post/entity-get responses. An issue for CKAN too. Sub-ticket of #961 (form, validation, model sync meta-ticket) and depends on that work." 1285410546000000 1301076463000000 1081 defect johnlawrenceaspden johnlawrenceaspden closed fixed can't remove user from authz group I've found that if I make an authorization group I sometimes can't remove myself from it. I've no idea why. I can add and remove other users. I'll investigate, just making a note of it here. 1302541056000000 1303489474000000 2918 enhancement johnmartin ross ckan 2.0 closed fixed Can't remove users from organizations "When you remove someone, without adding them, the text box at the bottom (which should probably autocomplete) is empty, and this causes problems on the server. Ideally when you add a user (select from the autocomplete) it would add another row to the table, defaulting the user to editor and setting the names to user__{{X}}__name and user__{{X}}__capacity where X is $('tr').size()" 1347455572000000 1347970735000000 1374 defect dread dread ckan-sprint-2011-10-24 closed fixed Can't switch to English if default is non-English e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English. 1317893975000000 1319648746000000 1577 defect rgrp dread ckan-backlog new Can't upload file with foreign chars in filename "Looks like uploading a file with foreign characters fails due to encoding reasons. {{{ URL: http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:123 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckanext.storage.controller:2 in auth_form Module ckan.lib.jsonp:26 in jsonpify << Very much modelled after pylons.decorators.jsonify . """""" data = func(*args, **kwargs) return to_jsonp(data) >> data = func(*args, **kwargs) Module ckanext.storage.controller:301 in auth_form << method = 'POST' authorize(method, bucket, label, c.userobj, self.ofs) data = self._get_form_data(label) return data >> authorize(method, bucket, label, c.userobj, self.ofs) Module ckanext.storage.controller:79 in authorize << if method != 'GET': # do not allow overwriting if ofs.exists(bucket, key): abort(409) # now check user stuff >> if ofs.exists(bucket, key): Module ofs.remote.botostore:53 in exists << if bucket is None: return False return (label is None) or (label in bucket) def claim_bucket(self, bucket): >> return (label is None) or (label in bucket) Module boto.s3.bucket:87 in __contains__ << def __contains__(self, key_name): return not (self.get_key(key_name) is None) def startElement(self, name, attrs, connection): >> return not (self.get_key(key_name) is None) Module boto.s3.bucket:144 in get_key << response = self.connection.make_request('HEAD', self.name, key_name, headers=headers, query_args=query_args) # Allow any success status (2xx) - for example this lets us # support Range gets, which return status 206: >> query_args=query_args) Module boto.s3.connection:388 in make_request << if isinstance(key, Key): key = key.name path = self.calling_format.build_path_base(bucket, key) boto.log.debug('path=%s' % path) auth_path = self.calling_format.build_auth_path(bucket, key) >> path = self.calling_format.build_path_base(bucket, key) Module boto.s3.connection:88 in build_path_base << def build_path_base(self, bucket, key=''): return '/%s' % urllib.quote(key) class SubdomainCallingFormat(_CallingFormat): >> return '/%s' % urllib.quote(key) Module urllib:1222 in quote << safe_map[c] = (c in safe) and c or ('%%%02X' % i) _safemaps[cachekey] = safe_map res = map(safe_map.__getitem__, s) return ''.join(res) >> res = map(safe_map.__getitem__, s) KeyError: u'\xed' CGI Variables AUTH_TYPE 'cookie' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' HTTP_ACCEPT_ENCODING 'gzip,deflate,sdch' HTTP_ACCEPT_LANGUAGE 'en-US,en;q=0.8' HTTP_CACHE_CONTROL 'max-age=259200' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; ckan_user=elenaibp; ckan_display_name=""Elena Mondo""; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; __utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10.1324298316; __utmc=119670349; __utmz=119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' HTTP_HOST 'thedatahub.org' HTTP_REFERER 'http://thedatahub.org/dataset/edit/budget-library-czeck-republic' HTTP_USER_AGENT 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7' HTTP_VIA '1.1 localhost (squid/3.0.STABLE19)' HTTP_X_FORWARDED_FOR '87.114.74.190' HTTP_X_REQUESTED_WITH 'XMLHttpRequest' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' REMOTE_ADDR '193.34.146.142' REMOTE_PORT '55419' REMOTE_USER u'elenaibp' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [''] REQUEST_METHOD 'GET' REQUEST_URI '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' SCRIPT_URL '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' SERVER_ADDR '193.34.146.146' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
    Apache/2.2.14 (Ubuntu) Server at thedatahub.org Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.14 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1324298703.071357, '_creation_time': 1324293077.4139669} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan.net' mod_wsgi.reload_mechanism '1' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 8) paste.cookies (, 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; ckan_user=elenaibp; ckan_display_name=""Elena Mondo""; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; _ _utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10...)|utmcmd=(none)') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'} repoze.who.identity repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multiprocess' wsgi.file_wrapper wsgiorg.routing_args (, {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'}) }}}" 1324317659000000 1325473564000000 3019 defect seanh ckan 2.0 new Cannot delete dataset extras Deleting extras in the web interface is broken 1352918678000000 1352918678000000 1659 defect dread dread ckan-sprint-2012-01-23 closed fixed Cannot logout if CKAN mounted at non-root url "If you set WSGIScriptAlias to mount CKAN at a URL other than / then you cannot logout without adjusting the OpenID logged_out_url to match in who.ini config. e.g. [plugin:openid] ... logged_out_url = /sub/dir/user/logged_out Note: all the other URLs in who.ini should not have the /sub/dir/ - it is just this one that doesn't take account of the mounting point. The solution is to fix-up the repoze.who OpenID plugin to take account of the mounting point." 1326716302000000 1326747205000000 1431 defect dread dread ckan-v1.5 closed fixed Captcha field - foreign chars cause exception "During registering a user, the user inputs foreign chars into the captcha field. {{{ URL: http://thedatahub.org/user/register ... Module ckan.lib.captcha:22 in check_recaptcha << remoteip=client_ip_address, challenge=recaptcha_challenge_field, response=recaptcha_response_field)) f = urllib2.urlopen(recaptcha_server_name, params) data = f.read() >> response=recaptcha_response_field)) Module urllib:1267 in urlencode << for k, v in query: k = quote_plus(str(k)) v = quote_plus(str(v)) l.append(k + '=' + v) else: >> v = quote_plus(str(v)) UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 0: ordinal not in range(128) }}}" 1320078849000000 1320084104000000 2725 enhancement toby shevski demo phase 5 new Case sensitivity on tags "My feeling is that 'country-US' and 'country-us' should be the same tag. However currently tags with caps are treated differently see http://s031.okserver.org:2375/en/dataset/test-dataset with TEST and test - there also get indexed twice in the search page" 1342949667000000 1343030773000000 480 requirement thejimmyg johnbywater ckan-v1.4 closed fixed Catalogue service shall conform to specification "Common requirements for running CKAN behind a (e.g Wordpress or Drupal) front-end: 1. Unrestricted total read-only access to catalogue API for general public (e.g. voluntary organisation). - monitored by API key - not monitored by API key 2. Restricted total read-write access to catalogue API for authorized clients (e.g. front-end system, bulk upload clients). - restricted by CKAN access controller - restricted by HTTP Auth - restricted by IP address 3. Restricted total read-write access to catalogue Web UI for authorized users (e.g. site admins). - restricted by CKAN access controller - restricted by HTTP Auth 4. Restricted partial read-write access to catalogue Web UI for authorized users (e.g. group admins). - restricted by CKAN access controller - restricted by HTTP Auth [[Image(fig128130.png)]]" 1282422612000000 1300281551000000 486 requirement johnbywater ckan-v1.3 closed duplicate Catalogue service shall notify and query SOLR service 1282425790000000 1291639321000000 488 requirement johnbywater closed wontfix Catalogue service shall notify RDF service 1282426021000000 1320930240000000 1616 defect amercader amercader ckan-sprint-2012-04-02 closed fixed Catch exceptions when rebuilding the search index Right now if an exception is found while reindexing, the whole process stops and the remaining datasets are left out of the index. The process should continue after logging the exception. If more than a certain number of exceptions occur in a row, the process should stop. 1325844669000000 1332327635000000 1809 enhancement johnglover johnglover ckan-sprint-2012-03-05 closed fixed Catch request exceptions in archiver link_checker task Some request exceptions are currently not being caught (see the celery log on thedatahub for examples) 1329746267000000 1330528828000000 1609 enhancement ross ross ckan-sprint-2012-01-23 closed fixed Celery task for ckanext-archiver to write to webstore. "From super Storage changes - #1574 - and http://ckan.okfnpad.org/newstorage we determined that ckanext-archiver should have a celery task for grabbing local file uploads and writing to webstore == Analysis == When I upload a file to CKAN: * End up with file in permanent storage * IF file is ot type ... csv,xls,xlsx,sqlite,.sql * End up with new db in webstore * Where? {username}/{resource-id}/... * If a single table: name it after the file name (appropriately slugified) * A resource *always* corresponds to a 'database' in webstore ... * In Data Explorer have ""Sheets"" tab ... * Resource url = /dataset/{x}/resource/{y}/link -> cached_url ..." 1325582253000000 1327057030000000 326 task dread dread v1.1 closed fixed Centralise importation of json library Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers. 1274784223000000 1274789296000000 760 task johnbywater johnbywater ckan-v1.3 closed duplicate "Change ""CSW Get Records"" request class to accept and used given CSW filter" 1288040993000000 1294409111000000 758 task johnbywater johnbywater ckan-v1.3 closed duplicate Change API documentation to indicate harvest source entity has filter attribute 1288040643000000 1294409053000000 2315 enhancement dread dread ckan-sprint-2012-04-30 closed fixed Change Cookie expiry "Change login cookie from a default expiry of 50 years to 2 years. You can also uncheck a 'remember me' checkbox on the login form for the cookie to just last the session. Background conversation on ckan-dev: DR: I wonder if anyone objects to the expiry of the login cookie to be changed from 50 years to 2 years? 50 years might be appropriate for thedatahub.org, but for government sites it seems (to me) to be too lax. Toby: is this the repoze.who cookie? If so that seems sensible to me. Rufus: Definitely agree. I would also like to see introduction of a standard ""remember me"" checkbox (set to true by default). At the moment a login lasts forever (until you logout) automatically." 1334919449000000 1334919522000000 997 defect kindly kindly closed duplicate change create on cli to upgrade When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. 1298287981000000 1298288665000000 998 defect kindly kindly closed fixed change create on cli to upgrade When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. 1298287981000000 1300364423000000 2679 enhancement icmurray icmurray ckan-v1.9 new Change default behaviour of TemplateController.view to 404. "The current behaviour of TemplateController.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file. Although this may be a feature that some instances want. In general, it leads to: - 500s when attempting to access a normal template (eg - http://datahub.io/importer/preview) - A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template). Solution: - Change the controller to 404 - Ensure there's a way for existing ckan instances to override that behaviour should they need it." 1342436133000000 1342436133000000 1149 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed Change domain object modification plugin to use Session extension. This should make it more efficient as it currently does a lot of repeating work. i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1. 1305969863000000 1306090663000000 2737 enhancement toby shevski demo phase 2 closed fixed change help/prefilled text in format field "Should be basic format (which is what the datastore wants & what we display to users) e.g. ""CSV, XML, JSON etc"" http://s031.okserver.org:2375/dataset/new_resource/" 1343121262000000 1343136141000000 2634 enhancement toby toby ckan-v1.8 closed fixed change log - helper function limited in 1.8 make sure documented 1341828248000000 1342085656000000 461 task dread johnbywater ckan-v1.2 closed fixed Change ONS data importing to work via API " * Move script out to ckandgu repo * Change script to convert xml into package dicts * Test (against test.ckan.net, hmg.test.ckan.net) * Deploy" 1282303411000000 1283250478000000 198 enhancement rgrp dread closed fixed Change package and tag ids to uuids "See how we did it already for other things. Note: on ckan.net older PackageRevision.id might not be identical to Package.id but this may need sorting at this point." 1258980613000000 1266837606000000 752 task johnbywater johnbywater ckan-v1.3 closed duplicate "Change package attribute names used by Gemini harvesting to DGU ""v.4""" 1288039205000000 1294408472000000 126 enhancement dread dread v0.10 closed fixed Change package state in the WUI (delete and undelete) "As a Package Admin I want to change the state of the package. In particular I wish to delete and undelete it. (NB: this is quite separate from ""purging"" objects which is the term we shall use for irrevocable removal of an object from the domain model). * Only Package Admins (and sysadmins) should be able to change state == Implementation Suggestions == * 'delete' action should be renamed to 'change-state' (NB: this requires a db migration ...) * Have new package formalchemy form (created via inheritance?) to incorporate state attribute. Suggest this is rendered as a dropdown (and may be simple object rendering of state, i.e. do NOT need to change it to a single name such 'active'). * This form should then be used when the user satisfies is_authorized(..., model.Action.CHANGE_STATE) instead of the usual fieldset " 1253789571000000 1254740244000000 1478 enhancement kindly kindly ckan-sprint-2011-12-05 closed fixed change package_group to members table "This will comprise of the following steps. * model migration. * simplify revisioning * Remove any sqlalchemy relations from groups. * change logic functions estimate 2d" 1321959973000000 1323172654000000 2246 enhancement johnglover johnglover ckan-sprint-2012-04-02 closed fixed Change published_by metadata field to reference group instead of a custom extra * probably needs a new converter, as needs to be usable via API as 'published_by'. 1332243036000000 1332864871000000 2923 defect seanh ckan 2.0 new Change regularise -> regularize The function is called regularise_html(), can't remember what file it's in. 1347530582000000 1347530582000000 252 enhancement dread johnbywater closed invalid Change revision object so that it has parent(s) attribute 1266519767000000 1296477560000000 1534 enhancement rgrp ckan-backlog new Change revisions to record userid rather than username "The use of username is problematic because username's can change. * Change all revision creation code to use user id (simplest is to change c.author field in lib/base.py (?)) * (?) Add a field ipaddr for ip address of anonymous users? (or just keep putting this in author field on Revision and then acception that those won't match when we do a look up against user table) * Change user view page to look up against user id rather than name * Perform migration on existing Revision objects * Match should probably be against both openid and username when searching Revisions' author field (especially true on CKAN where some people have already changed their username from being their openid)" 1323278790000000 1338205050000000 1819 enhancement kindly kindly ckan-sprint-2012-04-02 closed fixed change search index code to use package_dictize instead of obj.to_dict Search index code needs to use package_dictize so that it can get the new vocablurary information and is more consistant with the rest of the system. This is in preperation for their translation being put in the search indexer. 1329770555000000 1338193886000000 2206 enhancement johnglover johnglover ckan-sprint-2012-03-19 closed fixed Change site header to match latest ODP template 1330958095000000 1331046486000000 62 enhancement dread rgrp v0.10 closed fixed Change tags to contain any character (other than space) Requires us to url encode the tag names when displaying them ... 1240585095000000 1250181376000000 414 task johnbywater dread ckan-v1.2 closed fixed Change the Apache and Varnish ports Ask Paul for a new machine for testing. Then one for varnish-live and one for varnish-test. 1281431639000000 1288003770000000 925 defect dread ckan-backlog closed fixed Change the search box icon to remove the down arrow Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed? 1295867593000000 1323168588000000 725 story pudo pudo iati-3 closed fixed Change to allow anyone (logged in) to create a publisher "With a pending state set (""unapproved"") " 1287584630000000 1289296038000000 69 enhancement rgrp rgrp v0.9 closed fixed Change to text-only license field and use external license repo "Switch from license domain object to a simple license field and use license list from new centralised license repo: * This will '''require''' a migration Cost: 4h (plus migration ...)" 1245687449000000 1246437494000000 1653 enhancement toby ross ckan-sprint-2012-03-05 closed fixed Change URLs for multilingual site "To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session. * Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs. * Write Middleware + update url_for to take account of the language. [2d] * Document the language setup, and how to replicate it. [1d] " 1326710590000000 1329845387000000 898 defect rgrp dread closed fixed Changes stored indefinitely "Every change to every object is being stored in memory, which could add up to quite a lot of memory. This fixes it by making sure the objects are in a weakref. https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f" 1294659490000000 1294662408000000 1135 enhancement kindly rgrp assigned Changeset model for vdm "Move to Changeset model for vdm. A changeset model is like an Audit-Log model in which we just record Changesets with Change-Objects rather than have Revision-Objects for each Object that is revisioned. This change would also incorporate significant simplication of vdm." 1305209986000000 1340632267000000 2959 defect icmurray icmurray ckan 2.0 new Changing a Group's name through the action api disassociates it from its datasets in the index "Repro: - Create a new Group, named ""test-group"". - Add a dataset to it. - Verify the dataset belongs to the group by visiting the Group's read-page - Update the Group through the action api (group_update), using the uid in the ""id"" field, and a new name in the ""name"" field. - Visit the group's read-page. The list of datasets will be empty. This was an issue when editing a Group through the web interface, which was fixed in [1]. However it only fixes the issue in the group controller. [1] https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335" 1349363935000000 1349363935000000 1829 defect dread dread ckan-sprint-2012-03-05 closed fixed Changing back to English prints the flash message in the previous non-English language "On the homepage click ""francais"" and then ""English"". The flash message reads ""Le langage a été fixé à: français"" when it should say ""The language is now: English""." 1330000660000000 1330001990000000 1791 defect dread dread ckan-sprint-2012-02-20 closed fixed Changing locale on /dataset/new causes exception "When you are on the /dataset/new page and you try and change locale then you get a 500 error. This is because it adds the '__cache' parameter, to ensure any proxy cache in the chain does not just send the cached page. e.g. http://127.0.0.1:5000/dataset/new?__cache=37713707 " 1329134556000000 1329138315000000 415 task dread ckan-v1.2 closed fixed Chase Talis about loading RDF from CKAN. 1281431656000000 1288003954000000 2632 enhancement toby toby ckan-v1.9 closed fixed check allowed helpers exist in lib helpers make sure the allowed function lists functions exist or throw an error. 1341827811000000 1342087027000000 660 requirement dread closed invalid Check CKAN instance works As an admin I want to check a CKAN instance works having just upgraded it or configured it. 1285348463000000 1311183115000000 659 enhancement nils.toedtmann dread closed fixed Check CKAN instance works automatically Auto way to check web and API interface of a CKAN instance basically works. Several gotchas can be quickly determined, such as logging in, search not working. Needs to be configurable per site basis. 1285348333000000 1311183031000000 1089 enhancement dread dread ckan-v1.4-sprint-6 closed fixed "Check for ""--ckan"" when running nosetests" (because if you forget, you get difficult to understand errors, and more than one person has tripped up on this) 1302631189000000 1302631733000000 612 task johnbywater johnbywater ckan-v1.3 closed duplicate Check given XML schema validates given metadata document 1284218750000000 1294408188000000 1814 enhancement amercader amercader ckan-sprint-2012-03-19 closed fixed Check publicadata.eu harvesters "Estimate 2d Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras). Also check non-CKAN harvesters (specially scrappers) to see if they are still working. Make a list of current harvesters with status and potential ones." 1329757408000000 1332152596000000 194 defect rgrp dread v0.11 closed fixed Check star ratings aren't influenced by search engine crawlers rel=nofollow or robots.txt ? 1258471512000000 1265284389000000 2651 enhancement icmurray ross ckan-v1.8 closed fixed Check support for TSV which doesn't appear to work well. "TSV support doesn't seem to work very well, may be the mimetype ( text/tab-separated-values ) See http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96 " 1341923318000000 1343209784000000 2476 defect seanh johnglover ckan-sprint-2012-06-25 closed wontfix Check that translating lists of strings is being tested in multilingual tests Check that lists of strings are being correctly translated. See https://github.com/okfn/ckan/commit/f1d68c3d2d4d25a0c0f8a89a68940643fc19b156 1338378078000000 1339151396000000 2751 enhancement toby toby demo phase 5 new check translations for full demo site need to check everything gets translated - sean did this before so will have info 1343216443000000 1344243046000000 617 task johnbywater johnbywater ckan-v1.3 closed duplicate Check UKLP schematron validates given metadata document 1284219298000000 1294408164000000 1470 defect dread amercader ckan-sprint-2011-11-21 closed fixed Check user name in the profile form 1321446143000000 1324473955000000 2795 enhancement toby demo phase 5 new Check validation of HTML, CSS, JS Ensure that we are being standards compliant 1343903128000000 1343903128000000 871 defect nils.toedtmann closed invalid Check whether localhost-only exim installtions need upgrading too "The infamous [http://www.exim.org/lurker/message/20101207.215955.bb32d4f2.en.html exim bug] only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers. We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not. BTW did we already run a rootkit checker like [http://rkhunter.sourceforge.net/ Rootkit hunter] on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could. " 1292264117000000 1296340558000000 991 defect dread ckan-v1.3 closed fixed Checkbox defaults to True "Form for new package has CheckboxExtraField checked, when the value is False. (as used in ckanext-dgu package v3 form)" 1298035175000000 1298037717000000 2837 enhancement aron.carroll toby demo phase 3 closed fixed checkbox styling Is this done I know checkboxes are a pain but it looks a bit out of place compared to the other items. 1344605993000000 1344851271000000 2943 enhancement dominik new Chrome does not resize preview Chrome does not resize iframe after a full refresh/ on first load 1349089686000000 1349090759000000 1171 enhancement mark.wainwright dread ckan 2.0 assigned Citation instructions on dataset and resource view pages "Some sort of citation helper. Something small on the dataset and resource page that would show how to cite. wwaites: Some related thoughts on this from opb: http://homepages.inf.ed.ac.uk/opb/papers/ssdbm2006.pdf timclicks: I'm looking at Dataverse for the first time[0]. It seems very popular in the social sciences. I noticed that there is a recommended citation for each dataset. For example, [1] is has this one: ""Targeted Input Programme (TIP) 2000-01"", http://hdl.handle.net/1902.1/SSC-MWI-TIP2000-01-M1 V1 [Version]"" == Implementation == Add a small box at bottom of dataset / resource page (or in sidebar on dataset page) with title ""Cite this"" with contents like: %title. %author. Retrieved %date. %site_title. For resource: %title = %dataset_title. %resource_name. Could also add export to ref managers (e.g. to bibtex) but that is for later." 1306920799000000 1347358705000000 3016 enhancement johnmartin johnmartin ckan 2.0 new CKAN 2.0 template tweaks Just a ticket to keep track of a few suggested template changes. 1352813417000000 1352813417000000 1377 defect zephod zephod ckan-sprint-2011-10-10 closed fixed Ckan admin repair "Integrating ckanext-admin into core has thrown up a number of problems: * Look & feel does not match the rest of the site * Tests are not passing * On the trash page, clicking 'undelete' triggers a purge * Using the purge functionality is dangerous; deleting and purging the latest revision will corrupt a dataset (& several corrupt datasets have been found on thedatahub.org) * Trash page can contain nested form tags in certain cases (breaking test harness & form redirection) " 1318240018000000 1318245795000000 1551 enhancement ross ross ckan-backlog closed fixed CKAN auth for webstore changes Webstore should use auth api ( #1550 ) for authenticating users accessing webstore rather than talking directly to the CKAN database. We also need it to suppose /user/ urls and /userid/ for accessing databases. 1324049966000000 1346662048000000 698 task Stiivi thejimmyg ckan-v1.3-sprint-1 closed fixed CKAN Data API v1 "This proposal is to discuss adding a new API for proxying certain spreadsheet data via JSON-P to make it possible to build simple browser apps directly off the API. See the attached proposal for information." 1287073433000000 1293649815000000 1467 defect thejimmyg thejimmyg ckan-sprint-2012-01-09 closed worksforme CKAN dumps dgu miss certain publisher information Pawel knows about this so David Read, Pawel and I need to find time to discuss it. 1321376042000000 1326120319000000 333 enhancement dread v1.1 closed wontfix CKAN front end requirements for package notifications "== Use case: new package == 1. An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package. 2. User: clicks 'new package'. 3. CKAN presents the package/new form to the user. 4. (After a couple of previews) User: clicks 'commit'. 5. Notification message goes from CKAN to the front-end detailing the new package. 6. The user is redirected back to the front-end web page displaying the list of packages, which contains the new one. The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6). In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages? == Use case: CKAN imports packages == 1. CKAN administrator runs a script that adds 100 new packages into CKAN. 2. CKAN sends notification message to front-end to report the new packages/revisions. 3. Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages. 4. The front-end queries CKAN for each new package one-by-one. 5. A new user request to the front-end will include the info about the new packages. The package addition could be achieved in 1 revision, 100 revisions or some compromise: * If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'. * If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN. This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages." 1275324042000000 1275407987000000 837 enhancement rgrp ckan-backlog new CKAN integration with freebase gridworks / google refine "Thread: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html === Scenario 1 === 1. User installs Refine and CKAN extension for refine 2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance) 3. They edit the dataset on Refine 4. On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package) NB: for the dataset sync back some form of ""CKAN"" storage would be required (we already have storage.ckan.net running but a closer integration would be required) === Scenario 2 === 1. User visits a package on CKAN.net (or another CKAN instance) 2. There is a button on the page ""View and edit this dataset in Google Refine"" 3. Click button -- ask them if they have Google refine installed * Yes: instructions for loading dataset into refine * No: load dataset in hosted version of google refine (we could run this) 4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset." 1291140609000000 1339774605000000 1003 enhancement rgrp rgrp ckan-v1.4-sprint-3 closed fixed CKAN Javascript library and demonstration web interface "A plain javascript library for interfacing with CKAN would be very useful (why? see below!). It would also be nice to have a pure html + javascript web interface to CKAN both for its own sake and to act as a demonstrator for the library. Why? * Development of bespoke interfaces -- much easier to edit html + javascript than to change ckan core * E.g. for specific communities e.g. geodata, science * Specialized tasks - multi-package editing * Very easy deployment and integration (e.g. can drop in to getthedata or other sites) " 1298490086000000 1300100411000000 1804 defect toby dread ckan-sprint-2012-03-05 closed fixed CKAN mounted at URL - changing language problem "e.g. http://189.9.137.65/dados/ clicking on Deutsch link is {{{http://189.9.137.65/dados/locale?locale=de&return_to=%2Fdados%2F&hash=1dc17c315c419df850da0dd3599eefa9da76fbeb}}} and redirect goes to {{{http://189.9.137.65/dados/dados/?__cache=97995106}}} so /dados/dados/ when it should be /dados. Affects CKAN 1.6b only (not yet released)." 1329484956000000 1330347185000000 441 requirement dread dread ckan-v1.3 closed duplicate CKAN read-only state "When performing maintenance on CKAN it may be necessary to make CKAN obviously read-only, telling the users and restricting access to 'edit' pages. Examples of use: * Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done. * A CKAN is used just for distributing metadata and so is always read-only. Updates may still arrive through direct db manipulation, e.g.: * another (but writable) CKAN instance is connected to the same db * restoring database dumps from another CKAN db * Should a security be breached, all editing could be stopped " 1282227314000000 1292586309000000 576 defect sebbacon wwaites ckan-v1.3 closed fixed CKAN Requires Old Version of SQLAlchemy "Requires 0.4. 0.5 is a maintenance branch, 0.6 is current. VDM appears to work correctly (all the tests pass) with 0.5. CKAN does not -- requires more investigation to determine exactly why. It would actually be nice to be able to run with 0.6 or 0.5, though 0.6 will require some changes to VDM as well" 1284141573000000 1294753848000000 847 enhancement pudo rgrp ckan-v1.3-sprint-2 closed fixed CKAN search using SOLR backend "This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010). 3 key aspects of this: * Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN) * Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable * Asynchroously: via worker with queue * #324 Search indexing using notifications * #669: Refactor Solr (indexer) to become a generic worker * #874: Extract Solr search backend into an extension * In core support using SOLR backend as well as postgres for queries * #317: Make search pluggable (?) * Integrate into WUI (if changes needed) * #672 - Facets in CKAN search results Extras: * Solr index testing tool #431" 1291639273000000 1295259902000000 952 task wwaites pudo ckan-backlog closed invalid CKAN should run under nginx/uswgi " second part of #908" 1296730498000000 1310125204000000 1615 enhancement thejimmyg ckan-v1.6 closed worksforme CKAN Should work behind a proxy server "This would allow deployment via Nginx or Apache using proxy to Paster, uWSGI. At the moment CKAN isn't aware of the proxy's IP address so when you perform an action which does a redirect (such as adding a package), CKAN redirects you to the *internal IP* not the external *proxy IP*. We would like this work to facilitate testing within VMs as part of our new build infrastructure. It would also be nice if CKAN worked when mounted at a path other than /. That could be dealt with in another ticket because it isn't a problem at the moment." 1325687841000000 1328888870000000 1518 defect rgrp markbrough ckan-sprint-2011-12-19 closed fixed CKAN Upload fails if filename has spaces in it "E.g. uploading a file with spaces in it: OECD Monthly Exchange Rates.zip Gives a 404 Not Found response to the following file: http://test.ckan.org/api/storage/metadata/2011-12-05T193046/OECD%20Monthly%20Exchange%20Rates.zip The upload wheel keeps spinning and the user is not informed that the upload has failed. Analysis: turns out that google storage (possibly s3 but not checked) replaces with ' ' in keys with '+' on upload. This breaks things because we try and look up metadata about upload using the filename/key we put in but of course that does not exist because google has changed name. Fix is trivial: replace ' ' in keys / filenames with '-'. " 1323114236000000 1330020742000000 1384 task rgrp shevski ckan-backlog new CKAN wiki needs updating to refer to thedatahub.org instead of ckan.net and datasets instead of packages Most articles still refer and link to ckan.net, wiki.ckan.net and to packages 1318414077000000 1318414077000000 2682 defect seanh seanh ckan-v1.8 closed fixed CKAN's internal tracking counts each view twice, needs unit tests "CKAN's internal tracking seems to count each page view twice, the problem appears to be with the SQL in the update_tracking() method in ckan/lib/cli.py. The internal tracking feature needs some tests, and some of the code could maybe do with some more explanatory comments, e.g. what is the intended difference between count and running_total?" 1342446402000000 1343225636000000 1084 task wwaites wwaites ckan-v1.4-sprint-6 closed fixed ckan.net RDF links changed "need to make some changes for the links to semantic.ckan.net. it should use http://semantic.ckan.net/record/ now append .rdf, .ttl, .nt, .dot, .json (even .html for an ugly table) to taste (or just leave off the suffix and let content negotiation take care of it) the base url is changed, but it now uses id not name. see for example: * http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33 * http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33.ttl" 1302616717000000 1304934534000000 2471 enhancement johnmartin ross closed invalid ckan.org bug "If you go to http://ckan.org/features/geospatial/ there is a problem with the pop up tag shown (and zoomed in) - missing status and description. " 1338229749000000 1352206679000000 1461 defect pudo closed fixed CkanClient doesn't submit auth headers for GET requests e.g. package_register_get. 1321354037000000 1321359503000000 867 enhancement dread dread ckan-v1.3-sprint-1 closed fixed ckanclient raises exceptions "To be more pythonic, raise exceptions when ckanclient gets status which isn't 200. " 1291915162000000 1299866685000000 1314 enhancement dread ckan-backlog assigned ckanclient search - generator improvements "Apparently the search generator always makes two requests, even if you don't want to see the search results, which might be slow. Can this be optimised? Maybe we should also provide a second search function that doesn't use the generator - the original simple search function (that leaves the user to deal with limit & offset)." 1315395410000000 1340191233000000 1207 enhancement dread dread closed fixed ckanclient.package_entity_get should raise more specific exception When package does not exist in ckan catalogue, ckanclient.package_entity_get should raise more specific exception, such as CkanNotFoundError instead of generic CkanApiError. 1309515582000000 1311325343000000 1641 enhancement amercader amercader ckan-sprint-2012-01-23 closed fixed ckanext-archiver: Content-length header not reliable to check if resource has been modified "The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the ""Content-Type"" and ""Content-Length"" headers differ from the values stored to see if the resource needs to be updated [1]. The ""Content-Length"" header, although widely used, is not mandatory and some servers don't provide it, e.g.: {{{ $ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG HTTP/1.1 200 OK Cache-Control: private Transfer-Encoding: chunked Content-Type: text/xml Vary: Accept-Encoding Server: Microsoft-IIS/7.5 Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly X-AspNetMvc-Version: 3.0 content-disposition: attachment; filename=AFG_IATI_12012012.xml X-AspNet-Version: 4.0.30319 X-Powered-By: ASP.NET Date: Thu, 12 Jan 2012 12:36:43 GMT }}} Also worth noting that [http://docs.python-requests.org/ requests], the python library that uses ckanext-archiver, sets an ""Accept-Encoding: gzip"" header by default, which depending on the configuration of the remote web server, may prevent the ""Content-Length"" server from being sent, e.g.: {{{ $ curl -H ""Accept-Encoding: gzip"" -I http://iatistandard.org/published-temp/adb-activities.xml HTTP/1.1 200 OK Date: Thu, 12 Jan 2012 12:12:46 GMT Server: Apache Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Content-Type: application/xml curl -I http://iatistandard.org/published-temp/adb-activities.xml HTTP/1.1 200 OK Date: Thu, 12 Jan 2012 11:56:23 GMT Server: Apache Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT Accept-Ranges: bytes Content-Length: 2686720 Vary: Accept-Encoding Content-Type: application/xml }}} All this can lead to some resources never getting updated, and of course the size property of the resource not being set. As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it). [1] https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72" 1326376420000000 1326376777000000 2533 defect seanh seanh ckan-v1.8 closed fixed ckanext-harvest is broken with ckan 1.8b "Looks like recent CKAN cleanups broke some imports: https://gist.github.com/2935639" 1339754707000000 1339755433000000 2435 defect seanh seanh ckan-v1.8 closed worksforme ckanext-qa doesn't work with CKAN 1.7+ 1337963228000000 1340810257000000 2873 defect toby danieljohnlewis demo phase 4 assigned ckanext-qa: dataset summary incorrect " For example see: http://s031.okserver.org:2375/qa/dataset/five_stars Which has: Example dataset as a score of: 4 After examination of the dataset ( http://s031.okserver.org:2375/dataset/example-dataset ) it is made up of a CSV and an HTML file. " 1345126429000000 1345208283000000 2534 defect amercader seanh ckan-v1.8 closed duplicate ckanext-spatial is broken with CKAN 1.8b "Looks like recent cleanups in CKAN broke an import in the extension: https://gist.github.com/2935688" 1339754762000000 1340631124000000 2912 defect seanh ckan-v1.8.1 new ckanext/organizations: Neither public or private initially selected when adding dataset When adding a dataset with the organizations extension on, neither the Public or the Private radio button is initially selected. Public should be selected. 1347271076000000 1350303897000000 1648 enhancement shevski ckan-backlog closed fixed Clarify that additional info = extra fields + add guidance "Super ticket: #1506 Need to decide which term to use and then have the same for editing as well as viewing a dataset. In creating/editing a dataset, want more explanation about adding extra fields (probably as a tooltip or similar).. i.e. that this let's you add extra custom metadata such as 'location: uk' which is then searchable etc" 1326674843000000 1330632344000000 2244 enhancement toby toby ckan-future new clean up auth "* cache admins etc * aim to move check_access so independent of logic function * general spruce up" 1332177771000000 1338205402000000 2874 enhancement rgrp rgrp assigned Clean up bin directory Full of obsolete material 1345190508000000 1345190515000000 2472 enhancement amercader kindly ckan-v1.8 closed fixed clean up datastore controller to get rid of datastore_url enables checks "Datastore_url enabled checks are useless. We should probably change this to datastore_url = 'active' and only update the flag to when a attempt at a post is made. This will need to be removed from the form at some point." 1338293407000000 1341222195000000 3030 enhancement toby new clean up helper functions 1355834038000000 1355834038000000 2417 enhancement toby aron.carroll closed fixed Clean up output for dataset search results "Currently due to the data structure the search result filters are output incorrectly. Repeating the facet with each tag. See http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa It outputs: Tags: fibre Tags: terrestrial It should be: Tags: fibre terrestrial" 1337793213000000 1337864155000000 1803 enhancement toby toby ckan-sprint-2012-03-19 closed fixed clean up routes 1329483497000000 1332333091000000 2572 enhancement toby toby ckan-v1.9 new clean up stats plugin attempt to disengage the stats plugin from core as much as possible 1340105054000000 1340899524000000 2536 enhancement toby toby ckan-v1.9 closed fixed cleanup activity streams 1339765323000000 1347901915000000 2229 enhancement kindly kindly ckan-sprint-2012-03-19 closed fixed Cleanup plugin system after some test failed to run. The logic test did not have __init__. This caused lots of tests to fail because there were mock extensions that ran automatically in them. Fix plugin system so this can work. 1331721611000000 1332163408000000 2257 enhancement toby toby ckan-v1.9 new cleanup template vars "look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc" 1332513307000000 1340097071000000 1362 defect johnglover johnglover ckan-sprint-2011-10-10 closed fixed Clearing the database should also clear the search index When paster db clean is run, the search index should also be cleared. 1317121861000000 1318256546000000 1026 enhancement dread dread ckan-v1.4-sprint-3 closed fixed cli for creating users It's handy to be able to create users using the cli (e.g. dgu migration) 1299604652000000 1299605128000000 1067 enhancement dread dread ckan-v1.4-sprint-5 closed fixed CLI for loading/dumping complete databases Use 'db dump' and 'db load' for 'pg_dump' and 'psql -f' of a database. Use pylons config to find out database options. 1301645463000000 1302186503000000 2401 enhancement dread dread ckan-sprint-2012-05-29 closed fixed CLI for time/speed profiling To enable you to easily track down what is taking all the time when you make a request. 1337269571000000 1337273042000000 1114 enhancement dread dread ckan-v1.4-sprint-7 closed fixed CLI for viewing search index of a package To see what lexemes are generated for debug purposes. 1304078353000000 1304085484000000 1790 enhancement dread ckan-future new Click to delete tags, rather than have all existing tags in the tag text box "From Pablo: Editing the tags field is clumsy when there are too many tags. Could show existing effectively as tags (like delicious), then allow clicks to delete. New tags added via text box." 1328888674000000 1328888674000000 2689 enhancement aron.carroll shevski demo phase 2 closed fixed clicking on 'save & add another' doesn't act as expected "Clicking on 'save & add another' button here http://s031.okserver.org:2375/dataset/new_resource/yo takes you to step 3 instead of leaving you on step 2 when you haven't added any data. Instead it should show an alert/ message that no data has been added & that you need to link to a file or upload something Also, can upload be option 2 instead of 3?" 1342538126000000 1342617293000000 322 enhancement dread dread v1.1 closed fixed Client interface for Notification Service "=== Use cases === * Register for package changes * Register for all revisions * Notified of a package change * Notified of a revision * Deregistration * Configuration of port in pylons config === Design === * Default port: 5672 (standard for AMQP) * Exchange name: 'ckan' * Exchange type: topic exchange (most flexible) * Routing keys: (see below) === Routing detail === Routing key format: ""OBJ_TYPE"" (NB tags should be identified by their name, not ID) Example routing keys * 'package' - Package edited/created * 'resource' - Resource edited/created * 'revision' - Any change * 'db.clean' * 'db.rebuild' Example queue bindings that clients may use: * * - no filtering - client receives all notifications * package - only changes to packages * revision - all revisions * db - all database operations === Versioning === Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps? === Documentation === * How to use * simple example of an external client?" 1274720042000000 1277722821000000 2835 enhancement aron.carroll demo phase 5 new Client module needs a template loading method "{{{ Client#getTemplate(name, params, success, error); }}} Where params, success and error are optional arguments. test/index.html already has an implementation called loadFixture()." 1344532233000000 1344532233000000 853 enhancement wwaites wwaites ckan-v1.3-sprint-1 closed fixed Client upload to storage without having primary storage keys "Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys. This would lead to an extension to OFS. This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN). * master ticket #852" 1291723063000000 1294594581000000 1164 enhancement amercader amercader pdeu-1 closed fixed Cloropleth Map of European Data Availability for PDEU A nice map in the homepage showing the availability of data across Europe 1306408824000000 1308647224000000 2253 enhancement toby toby closed wontfix CMAP [super] "Somewhere for CMAP stuff not in other tickets need to create some general tickets * template changes * general demo server setup " 1332341133000000 1340038490000000 55 enhancement rgrp rgrp v0.7 closed fixed Code to migrate data from v0.6 to v0.7 using dump and load "Associated to ticket:51 (upgrade CKAN to new vdm) and ticket:54 (dump/load) need to convert v0.6 data for v0.7. Obvious way to do this is via alteration to data load method." 1223908240000000 1223909891000000 2890 enhancement seanh ckan-v1.8.1 new Collect data previews and data store docs in one chapter "Currently there is this page: http://docs.ckan.org/en/latest/data-viewer.html which covers Recline Data Explorer and other kinds of data preview in CKAN. It is under the Publishing Datasets section in the documentation. I had to to a search for 'recline' to find it. Separately there is this page: http://docs.ckan.org/en/ckan-1.7.1/datastore.html which covers datastore, datastorer, and the data api. I suggest collecting this together in one chapter called 'Data Previews'. If I understand it right the general gist would be: CKAN has builtin previews of data resources on resource pages, enabled by default. Images, Google Documents, and web page resources will be loaded into embedded iframes for preview. Text-like files will be displayed raw. CSV or Excel files uploaded to CKAN will be previewed using Recline Data Explorer. Additionally, you can enable CKAN's DataStore, requires you to install ElasticSearch and nginx and put datastore.enabled=1 in your ini file. Lets you use the Data API to query data. Does having DataStore enabled mean you get preview of more types of resources? Any resource that's available via the Data API will be previewed using Recline, You can install ckanext-datastorer, and then CSV and Excel files _linked to_ as CKAN resources will be previewed using Recline also. Requires celeryd." 1346149236000000 1346175867000000 354 defect johnbywater johnbywater closed invalid Collect together requirements and top-level design for user/package 'groups' "Collect together requirements and top-level design for user/package 'groups': existing tickets, Rufus spec, Sean spec, meeting notes (dread) email, based on existing user authz stuff. http://knowledgeforge.net/ckan/trac/wiki/AccessControl Do we add these into user-role table somehow or new table? To present this to team" 1277131335000000 1282908983000000 1218 enhancement dread minspamboks@… ckan-sprint-2011-10-28 closed fixed Colour the History tab icon "Change the color of the ""History"" tab icon to yellowish, like the rest of the icons in the other tabs (""View"" and ""Edit"", ""Authorization""). == Reasoning == When you view a data package, for instance http://ckan.net/package/thesaurus-w, you will see ""View"", ""Edit"", ""History"" tabs on the top. ""History"" tab has a black-and-white icon which makes it look like an inactive/disabled tab (since the text is also grayed out when the tab is not selected). This is not a major issue, but it is a little bit confusing for the users. This icon exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). The simple solution would be to change the color of the ""History"" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. ""View"" and ""Edit"")." 1310375768000000 1310389390000000 1519 enhancement johnglover shevski ckan-sprint-2012-01-09 closed wontfix combine stats and analytics extensions into one in UI as well as deployment "Makes more sense to only have one comprehensive stats/analytics extension, so when people are looking to add a stats extension they won't have to add two which may be confusing (is one an old version of the other? why do I have to add two? what's the difference? etc) User-wise we need a way to display our stats with google analytics in the same place" 1323169033000000 1324317373000000 1133 defect johnlawrenceaspden closed worksforme command line rights manipulation doesn't work "It appears that the command $ paster rights add russianfan admin warandpeace has no effect, even though $ paster rights remove russianfan admin warandpeace works fine. This may be specific to something I've done, could someone confirm? If it's the case more generally, then I'm assuming this behaviour is untested? Tests should probably be added." 1305054948000000 1324057072000000 1559 enhancement rgrp jilly mathews ckan-sprint-2012-04-02 closed fixed Comments Extension / Disqus Polish off comments extension dev and test. estimate 2 days. 1324291720000000 1332242129000000 1182 defect timmcnamara ckan-backlog new "Comments from deleted packages appear in ""Recent Comments"" feed" "When a package has been deleted, say for spam moderation, comments still appear in the recent comments section. This is a problem because non-admin users will be shown a warning that they're not authorised to view the package if they click on the link. At CKAN.net currently, this affects the most recent comment." 1307658251000000 1339774319000000 1415 enhancement thejimmyg nils.toedtmann ckan-sprint-2011-12-05 closed fixed Comments on current status of ckan deb packages "This is a scratch pad ticket with some comments on the current status of our ckan deb packages. I know that some of it is the deb packaging roadmap anyway, please forgive me if i mention them here again. Rufus and me re-deployed some community ckan instances onto s022 (see http://trac.okfn.org/ticket/926). We followed the documentation http://docs.ckan.org/en/latest/install-from-package.html * Deb package version number: the version of the deb package is ""python-ckan 1309471251~149be76faabc+lucid-1"", and it's hard to guess from there that it contains a ckan 1.4.2a * When is 1.4.3/1.5.x expected as deb package? * There was a bug in the DB upgrade script /usr/share/pyshared/ckan/migration/versions/029_version_groups.py (line 150) which looks like it was fixed 1.4.1==>1.4.2 but was nevertheless present in this deb package. * The current script /usr/bin/ckan-std-install * does not set the Apache ServerName according to the $INSTANCE variable * automatically configures a ckan extension named after $INSTANCE * depends on local postgres * could be replaced with ""/usr/bin/ckan-deploy --name=ckan-std --domain=ckan-std.localhost (see next point) * (i think this is exactly James' plan): have more generic deployment script /usr/bin/ckan-deploy as part of python-ckan which takes arguments like * --domain=cc.ckan.net * --aliases=$list-of-domains * --name=cc (defaults to ""domain"") * --no-db (does not configure a DB) * --sql-alchemy=$DB_CONFIG_STRING (also runs ""paster --plugin ckan db upgrade --config"") * --extension $list-of-extesions * ..." 1319457069000000 1323167941000000 296 enhancement johnbywater johnbywater closed duplicate Commit CKAN revisions to changeset system 1272279521000000 1294407032000000 310 defect dread rgrp v1.1 closed fixed Commit message box looks wrong in edit page since edit style overhaul "Suggest move this below the label and make full width of screen and only 3/4 rows high (more like a wiki site). * Also change label to: Edit summary (Briefly describe the changes you have made) * Remove: you can markdown formatting here. * Move author: if you have not signed in smaller and closer (like markdown instructions are nwo). * Change commit -> save * Remove ""please save"" just have the bullet points" 1273348714000000 1279300525000000 987 defect pudo pudo closed duplicate Common harvesting framework "We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include: * CSW/WFS for INSPIRE/UKLII (yields CKAN packages) * Catalogue scraping for LOD2 experiments (yields RDF graphs) * Atom/DCat for LOD2 production (yields RDF graphs) * OAI-PMH for http://datadryad.org/ and other dspace (yields CKAN packages) We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios. In general, such a system would have the following stages: * Source selection: find what to download/scrape/harvest/parse * Index retrieval (i.e. package index) * Item retrieval (i.e. package entity) * (Optional: Serialization) * Normalisation * Loading/Merging into CKAN Exisiting harvesters are at: * CSW: https://bitbucket.org/okfn/ckanext-csw/src/ * Scraper+CKAN: https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/" 1297684756000000 1311177705000000 3013 defect dominik dominik new common-error-messages is unreadable "Since the update of the doc theme, the page became unreadable. http://docs.ckan.org/en/latest/common-error-messages.html" 1352553505000000 1352553505000000 1048 enhancement dread dread ckan-v1.4-sprint-4 closed fixed Complete making groups versioned " * Deleting a group changes state to 'deleted' rather than purging it * Adding authz tests for deleted groups" 1300387655000000 1300702752000000 1557 enhancement David Rasnik jilly mathews ckan-future new Complete Webstore Preview Extension "Finish any work out standing on web store preview extension to be able to package and release. Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted." 1324291253000000 1324291253000000 1021 enhancement pudo pudo ckan-v1.4-sprint-3 closed fixed Config option to disable OpenID HRI don't like federation, want to login normal way only. Make this a config option and perhaps even mess with runtime repoze config 1299492920000000 1299518828000000 1428 defect dread dread ckan-sprint-2011-11-07 closed fixed config options not used or documented Need to ensure config options in the deployment.ini_tmpl match up with doc/configuration.rst. Remove any unused ones too. 1319803594000000 1319804773000000 2886 enhancement ross ckan 2.0 new Configurable related items "Related items (Apps & Ideas) have a collection of types, which are currently fixed. Some of these types are ambiguous, such as ideas in that they may not have a link - and if they do it is likely to be to a blog post - another existing type. Whilst the URL is required, and we think this should stay required, we should also allow users to change the types found in Apps & Ideas to a shortened list that suits their requirements." 1345625718000000 1345625718000000 1656 enhancement ross closed duplicate Configuration for reverse proxying "Provide configuration for reverse proxying that will correctly handle the mapping of a URL to a sub-folder (using X-SCRIPT-NAME) * Analysis of the best solution [1d] * Implement reverse proxying in tandem with #1653 [2d] * Document and store the configuration files. [1d] " 1326711575000000 1326711659000000 1657 enhancement ross ross closed wontfix Configuration for reverse proxying "Provide configuration for reverse proxying that will correctly handle the mapping of a URL to a sub-folder (using X-SCRIPT-NAME) * Analysis of the best solution [1d] * Implement reverse proxying in tandem with #1653 [2d] * Document and store the configuration files. [1d] " 1326711580000000 1338206640000000 400 task johnbywater johnbywater ckan-v1.2 closed fixed Configure DGU Public API with TSO Have TSO configure data.ov.uk/api/catalogue. Have TSO register catalogue.data.gov.uk. 1281000956000000 1288003690000000 525 requirement pudo pudo iati-2 closed fixed Confirm and approve publishing entity account " * So some kind of confirmation or approval by someone more senior? * Alternative: just have an account request. * Priority: 3" 1282894040000000 1285595152000000 2808 enhancement aron.carroll toby demo phase 3 closed fixed Confirm delete dialog badly placed on small screen "If you make your browser window short - say half screen height on a page with delete eg http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada the dialog box is partially off the screen - it should try to stay on screen" 1344333474000000 1345023693000000 2726 enhancement toby shevski demo phase 5 new confusing logic on data preview formats "1. If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this 2. If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28" 1342949927000000 1343030906000000 2415 defect kindly markw ckan-v1.9 new Confusing message on attempting to register new account with upper case "On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message: ""The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"" What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...' But even better would be to make the login case-insensitive and allow capitals to be entered. " 1337779189000000 1341268304000000 2813 enhancement toby markw demo phase 5 new Confusing sidebar on demo dataset page "On a dataset page on demo.ckan.org, the left sidebar is confusing. * It starts with some random links. Actually they are links to groups which the dataset is in, but this isn't clear. * The sidebar elements that are actually part of the dataset are 'Datset extent' and 'License', so these should be right at the top (if they belong in the sidebar at all), instead of which they are right at the bottom in the junk part of the page (i.e. probably lower than the bottom of the main page, and hence lower than anyone will scroll)." 1344420206000000 1344445419000000 2287 enhancement seanh ckan-future new Consistent datetime and time period strings localization "Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support. For example, on the user index page right now there are time period strings like ""less than 1 month"" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly). ISO date format everywhere might be a good idea. Need to consider local timezone issues. Python standard library and Babel may have useful helpers for this. Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page?" 1334254123000000 1339411445000000 995 defect kindly kindly closed wontfix consolidate caching Try and minimise the amount of different caching methodologies, especially for api calls. 1298283499000000 1311179009000000 673 task johnbywater closed Construct and send CSW GetRecordById request 1286214712000000 1286786758000000 759 story johnbywater johnbywater ckan-v1.3 closed duplicate Construct and send filtered CSW GetRecords request 1288040753000000 1294408652000000 1088 defect wwaites ckan-v1.4 closed fixed content-type autonegotiation is wonky "in ckan/controllers/package.py around line 130 it does some strange things... perhaps replace with https://github.com/wwaites/autoneg and handle redirection of these content types: {{{ application/rdf+xml application/turtle text/plain text/x-graphviz }}} " 1302630261000000 1303035487000000 2838 enhancement seanh ckan-v1.9 new Context variables accepted by action functions need to be documented "I was doing this: {{{ context = {'model': base.model, 'session': base.model.Session, 'user': toolkit.c.user or toolkit.c.author, 'extras_as_string': True} group_dict = logic.get_action('group_show')(context, {'id': group_id}) }}} in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass {{{'extras_as_string': True}}} in the context. I don't think this or other context variables are documented anywhere." 1344630911000000 1344630911000000 2552 enhancement ross ckan-future assigned Controlling access to features "== Requirements == To provide a freemium service it is necessary to be able to provide differing __levels__ of functionality based on the '''type''' of user (see #2550). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks. Initial implementation should focus on limiting access to datastore disk space. == Interface == These changes are currently only for the data hub and should be kept as much as possible within the data hub extension. == User Stories == * As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited). * As a system administrator I don't expect to need to manage the levels of users '''or''' the features that this applies to. == Tasks == [ ] Clarification of requirements/analysis [ ] Tests [ ] Code [ ] Model [ ] API [ ] UI [ ] Documentation == Estimates == " 1340018770000000 1346669544000000 2322 enhancement ross rgrp ckan-sprint-2012-05-29 closed fixed Convenience offset on end of resource urls: /raw, /api, /viewer "== Raw == /raw or /download (latter may be better) {{{ /dataset/{id}/resource/{resource-id}/raw => redirects to resource url (download) }}} == API == {{{ /dataset/{id}/resource/{resource-id}/api[/....] => either redirects or directly serves /api/data/{resource-id} }}} == Viewer (Explorer?) == Do data viewer but without anything else showing (full-screen?). this would not be just the data explorer but other cases too (including iframe etc ...) {{{ /dataset/{id}/resource/{resource-id}/viewer Resource (pre)-viewer without anything else (perhaps include resource title) }}} ----- Assigning for kindly for review. These would take about 20m to implement and would be *very* convenient from my experience of using the data API. ---- Implementation at enhancement-2322-convenience-urls" 1335236581000000 1344086640000000 2803 enhancement aron.carroll aron.carroll demo phase 3 closed fixed Convert all JS translations to use keywords 1344251817000000 1344256274000000 2343 enhancement johnglover johnglover ckan-sprint-2012-05-15 closed fixed Convert all non-free metadata fields to vocabulary fields " * convert non-free fields to vocab fields (eg: interoperability_level) * store values as JSON files in the ecportal repo * translate output on dataset view page * add tests" 1335876562000000 1335969506000000 1192 task annapowellsmith annapowellsmith ckan-sprint-2011-10-28 closed fixed Convert CKAN Sphinx docs into admin/reference manual "As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into: * An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install. * A Reference Manual which is the primary source of reference for CKAN software - this includes API docs. The current chapters of the Sphinx docs should be moved as follows: * index.rst - copy some stuff from README.txt, keep short * README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst * [NEW] install.rst - new file on installation * [NEW] theming.rst - move over from wiki * api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide) * i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file * design.rst - REMOVE (can copy some over if you can be bothered to http://wiki.ckan.net/Vision (all philosophical stuff should go!)) * loaders.rst: move to wiki.ckan.net/Using_Loaders * feeds.rst: move to User Manual (CKAN_Feeds or just Feeds) * importer.rst: remove (don't even copy) * forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) http://wiki.ckan.net/Customizing_Forms * forms-integration.rst: remove * model.rst: leave but move to reference (only for core devs) * load-testing.rst -> move to http://wiki.ckan.net/Load_Testing * distributed.rst -> remove * admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority * deb.html -> go into reference (query james gardner on list about moving this to wiki?)" 1308324593000000 1312191462000000 1472 enhancement rgrp rgrp ckan-sprint-2011-11-21 closed fixed Convert ckanclient to git and github from mercurial 1321624905000000 1321721874000000 100 enhancement rgrp rgrp closed fixed Convert existing data on disk to a datapkg distribution "(2008-10-29) convert existing directory on disk to a datapkg distribution (PythonDistribution by default). This would be presented in the CLI as a convert command. Details: * Add a metadata file (setup.py), perhaps prompting for input * List all existing data in that directory Cost: 4h " 1249986628000000 1318181227000000 70 enhancement rgrp rgrp v0.9 closed fixed Convert from py.test to nosetests Cost: 1h 1246434753000000 1247827053000000 85 enhancement rgrp rgrp closed fixed Convert state from an object to an enumeration "No real benefit of having State object as opposed to a simple text field with enumerated values and this second option is both simpler and more flexible. Main hassle here is that it requires migration in all projects that use vdm. If we're going to do this we should do it sooner rather than later. Cost: 2h (+ 1h for a conversion method)" 1248339662000000 1263206391000000 832 enhancement rgrp rgrp ckan-v1.3 closed fixed Convert stats to a plugin "Convert stats section to a ckan plugin. Stats need not be in core and making it a plugin would make it easier to extend and expand." 1290765915000000 1296334980000000 76 enhancement dread rgrp v0.10 closed fixed Convert to use formalchemy for all forms "Current form stuff is not very good (uses formencode). Switch to formalchemy would improve this, especially on validation. As an extra we could utilize the formalchemy pylons admin interface (pretty much for free). Cost: 12h == Details == 1. Replace htmlfill and formencode extract in controllers/templates with formalchemy 1. Crude and simple 2. Suppress unwanted fields (revision, state, all revisions) and sort out ordering to be similar to before. 3. Sort out tag field with a special renderer 2. Validation - testing definitely required. 3. Fix up description and pretty css etc" 1247829041000000 1251301765000000 2930 defect seanh ckan-v1.8.1 closed fixed convert_from_extras() returns qupted strings from API "Use an IGroupForm plugin to add a custom metadata field to groups using convert_to_extras() and convert_from_extras(), when calling group show the value comes back quoted, e.g. '""my_value""' Should add tests to example_igroupform and others that setting and getting the custom fields works through the action API." 1347639864000000 1361988592000000 1300 enhancement rgrp rgrp ckan-sprint-2011-11-07 closed fixed Core changes to base theme "Core changes to theme in order to make it easier to re-theme: * page_heading * side bar menu must be switchable (left to right) (through config or css) * Add optional_footer to complement optional_head * And adopt rule for template writers that all extra js must go in optional footer * That way we could move optional js to the bottom of the page to improve page load times. * Re-add support for body-class ... (seems to be removed by #1108) * more divs. * better labelling for css (and javascript)" 1314862223000000 1319794520000000 1681 enhancement ross ross closed fixed Core storage document "Document all of the changes and usage for #1574 so that we can mark it 'done done'. Make sure we document any setup issues/processes and the settings.py setup required for CKAN so that Mark can announce next week and it should be easily deployable/testable. Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc External: here's how to configure it and here's what you get ... (goes in main sphinx docs)" 1326904939000000 1338202930000000 2317 enhancement rgrp ckan-v1.7 closed wontfix Corrections to dataset creation form for v1.7 " * Add home page url * Remove group option (seen it confuse several users) - perhaps replace with tags (?) * Check form looks ok in portugese (seems to be a problem with css) " 1335022061000000 1340033433000000 1271 enhancement rgrp rgrp ckan-sprint-2011-10-28 closed fixed CORS support "CORS - http://www.w3.org/TR/cors/ - support. This is what you do in Apache. Should do this in lib/base.py or similar. {{{ Header always set Access-Control-Allow-Origin ""*"" Header always set Access-Control-Allow-Methods ""POST, PUT, GET, OPTIONS"" Header always set Access-Control-Allow-Headers ""X-CKAN-API-KEY, Content-Type"" # Respond to all OPTIONS requests with 200 OK # This could be done in the webapp # This is need for pre-flighted requests (POSTs/PUTs) RewriteEngine On RewriteCond %{REQUEST_METHOD} OPTIONS RewriteRule ^(.*)$ $1 [R=200,L] }}}" 1313241839000000 1313753663000000 798 task dread dread ckan-v1.3 closed fixed cospread importer using API Move existing getdata/cospread.py into dgu repo and use the importer/loader mechanism. 1289475362000000 1294916973000000 3014 defect seanh ckan 2.0 new Crash when deleting a non-empty vocabulary "From Knud Möller: when I try to delete a non-empty tag vocabulary via the API (through HTTP), I get an internal server error. Checking the logs, this turns out to be a consistency error raised by sqlalchemy: Error - : (IntegrityError) update or delete on table ""vocabulary"" violates foreign key constraint ""tag_vocabulary_id_fkey"" on table ""tag"" DETAIL: Key (id)=(21421955-7560-467c-af30-9f790b73e6ae) is still referenced from table ""tag"". 'DELETE FROM vocabulary WHERE vocabulary.id = %(id)s' {'id': u'21421955-7560-467c-af30-9f790b73e6ae'} URL: http://33.33.33.10:5000/api/action/vocabulary_delete The error makes sense, but I'm wondering if it would be useful to extend the API to also allow the deletion of non-empty vocabularies, possibly via a parameter (not sure what best practice in API design is). At the very least, it would be cool if the error message coming back in the response had more information in it." 1352803808000000 1352803808000000 15 enhancement johnbywater johnbywater closed invalid Create a file "== As a == Authenticated user == I want to == Create a file associated with a release == So that == The release has a new file associated with it == Notes == * Permissions are as for package updating (see ticket:10) * file attributes are described in file updating: ticket:16) * there are no attributes that are required to be supplied to create a file " 1152550770000000 1250619139000000 2553 enhancement toby toby demo phase 2 closed fixed Create a get snippet api action "just passes template name ?template_name=..." 1340020613000000 1342090078000000 2831 enhancement aron.carroll ckan 2.0 new Create a limited subset of markdown that's supported "Allowing people to use the full range of markdown results in extremely messy output across the site. I'd suggest limiting support to only a subset of common use cases. * Allow all inline elements, this allows bold, italic, code and links. * Allow lists. Disallow * Horizontal Rules * Headings * Block quote and code (this may turn out to be useful and so could be included) This way you get markdowns paragraph handling and a few inline styles without breaking the entire layout of the page. Here's the full syntax if anyone is interested http://daringfireball.net/projects/markdown/syntax" 1344512467000000 1344512467000000 1108 enhancement zephod pudo ckan-sprint-2011-09-12 closed fixed Create a more modern theme for CKAN "CKAN looks a bit aged, it should be styled more modernly and some elements could be re-arranged: * Collect user info in top bar * re-add the logo to ckan.net * Remove tags from main menu, replace with /sitemap.xml Inpiration: quora.com, github.com, Google Projects, Google Refine, etc. == CKAN.net or CKAN general theme? == To be decided. Suggest we start with ckan.net specific and then backwards integrate (?). Existing ckan.net theme repo: https://bitbucket.org/okfn/ckan-ckan.net" 1303830790000000 1315140879000000 130 enhancement dread rgrp v0.10 closed fixed Create a New Group via the WUI "== As a == User == I want to == Create a new group via the WUI == Details == * locate at /group/new/ * You must be logged in to create a group * Group creator is automatically given role of group admin * Edit screen is same as edit screen ..." 1254130212000000 1254735501000000 12 enhancement johnbywater johnbywater closed invalid Create a release associated with a package "== As a == Authenticated user == I want to == Create a release associated with a package == So that == The package has a new release == Notes == * Permissions are as for updating a package ticket:10 * for details of release attributes see: updating a release ticket:14 " 1152550407000000 1204135071000000 1167 task nils.toedtmann closed fixed Create a standard CKAN system image for Amazon EC2 (AMI) "We should create a public AMI with CKAN pre-installed and configured such that users can easily create their own EC2 machine with a running CKAN to play with. There are three phases: 1. [nils] Deploy an empty EC2 instance to become the CKAN image master instance 2. Install a CKAN and give it a standard configuration. 3. [nils] Create a AMI from the CKAN image master instance and publish it. I am happy to do first and last. Who is installing and configuring CKAN? Unfortunately AMIs are specific to region, architecture and storage type. We cannot maintain too many images, so a number of choices have to be made: * Which distribution/version? '''Ubuntu 10.04 LTS''' * Which architecture/instance-type? I suggest 64-bit/t1.micro * Which region? I suggest us-east-1 and maybe eu-west-1 * Which storage type? '''EBS''' (way easier to make an AMI from than instance-store) * Install CKAN from deb packages via mercurial/virtualenv? I assume the latter because the AMI is targeted to developers? ---- [http://alestic.com/2009/06/ec2-ami-bundle 1] [http://uec-images.ubuntu.com/releases/10.04/release/ 2]" 1306423016000000 1311178516000000 327 defect pudo pudo closed wontfix Create a web hook worker for CKAN " This will be useful as an example consumer of the new queue notifications (#325). * A user can register any URL to be notified upon an event. * Possible extensions: GET with ID only vs. POST with serialized object * E-Mail notifications are also hooks, essentially. We need to have a UI module that allows for the hook CRUD and some feedback (e.g. non 200 status codes)" 1274807361000000 1296467361000000 393 task johnbywater johnbywater v1.1 closed fixed Create aggregation feature to aggregate QoS log 1280446811000000 1280496962000000 508 task dread dread ckan-v1.2 closed fixed Create branch ultrastable " * Create a branch named ultrastable that represents latest version of 1.0 ''as found on dgu'' * Going forward ultrastable will act as stable - 1 (and either ultrastable or stable will be on the rock-solid production systems such as dgu) * Do '''not''' deprecate the stable or metastable branch for the present * Ask for recommendations on mailing list about best practice for branching (e.g. versioned based versus a simple HEAD versus stable) and how this affects standard operations such as applying bugfixes. * For useful discussion of how to do backporting and applying bugfix patches see: http://www.python.org/dev/peps/pep-0374/#backport * NB: in general do '''not''' always have to apply in stable/ultrastable and merge upwards for bugfixes -- can use transplant extension as well. " 1282663663000000 1282908795000000 2357 enhancement aron.carroll aron.carroll demo phase 5 accepted Create build script for front-end resources Should minify and concatenate JavaScript and CSS as part of #2354 1335886181000000 1343220524000000 661 task dread ckan-v1.3 closed fixed Create CKAN site tester "From Rufus: {{{ 'blackbox' testing command (read-only only would be fine) -- very useful for upgrade on sunday On blackbox thinking of something like: paster test-site http://hmg.ckan.net {package-name} Or (maybe better): paster test-site --site-spec {myconfig.json} my-config is a json file with info for testing (could overload --config rather than site-spec): site-url: http://hmg.ckan.net/ package-name: package-title: ...etc ... Then: Test front-page Test /revision/ Test /package/{page} -- check title Test /package/history Test /api/rest/package/ ... Test /api/search/package ... etc }}}" 1285348558000000 1288271266000000 1139 enhancement lucychambers lucychambers ckan-backlog closed fixed Create CKAN Theme Gallery "Take screenshots of existing ckan instances esp those mentioned and put on flickr in ckan or ckan-theme group so we can create a gallery ... (both to illustrate theming but also to show ckan instances that are around -- could add to http://wiki.ckan.net/Instances)" 1305645859000000 1306941356000000 943 enhancement pudo pudo ckan-v1.4-sprint-1 closed fixed Create CKAN Wiki Set up a Mediawiki to contain info on CKAN.net (community) conventions etc. 1296469547000000 1297076498000000 171 enhancement rgrp rgrp datapkg-0.7 closed fixed Create consolidated config for CLI Create consolidated set of config by merging config from .datapkgrc and command line options and use that instead of mixture of command line options and other stuff ... (likely to be less buggy!) 1256488011000000 1297210925000000 1590 enhancement amercader amercader closed fixed Create customized feeds for the IATI Registry "We need a way to track changes on the registry (datasets edited or updated), globally and on a per country/publisher/etc. basis. RSS and Atom feeds are really popular, and after closing #191 and #1498 crating them from the search results should be fairly easy. The following URLs are pretty self-explanatory: {{{ http://localhost:5000/feed/registry.rss http://localhost:5000/feed/country/AF.rss http://localhost:5000/feed/publisher/worldbank.rss http://localhost:5000/feed/organisation_type/10.rss }}} As we need to implement custom wrappers for countries, publishers, etc, we might as well offer a fully customizable feed, e.g.: {{{ http://localhost:5000/feed/custom.rss?q=activity_count:[* TO 100] http://localhost:5000/feed/custom.rss?publishertype=primary_source }}} Apart from the actual feeds, there will be a small amount of work at the template level to add the links to the suitable pages (and maybe a generic page showing all available feeds) " 1324486965000000 1326711608000000 2697 enhancement johnmartin shevski demo phase 5 assigned create dataset validation Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check) 1342620035000000 1346235925000000 2447 enhancement aron.carroll aron.carroll demo phase 5 closed fixed Create demo add dataset page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62234769/comments Implementation: http://s031.okserver.org:2375/dataset/new http://s031.okserver.org:2375/dataset/new_resource" 1338210835000000 1343924125000000 2450 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo add group form "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments Implementation: http://s031.okserver.org:2375/group/new" 1338211136000000 1339670997000000 2445 enhancement toby aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo dataset add related page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments Implementation: http://s031.okserver.org:2375/dataset/adur_district_spending/related/new Needs a new endpoint at /dataset/DATASET/related/new this should have a form containing the following fields. * Title (required) * Type of item (Application|Visualisation) * Description * URL (required) * Image URL When submitted if an item is created it should redirect (303) back to /dataset/DATASET/related with a flash message saying ""Related item was successfully created"". If failed to create it should redirect (303) back to the form and populate the error messages, see add dataset for examples (https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/package/new_package_form.html) " 1338210652000000 1338982754000000 2443 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo dataset data (resource) page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62308209/comments Implementation: (to be added once test server is working)" 1338210382000000 1341824663000000 2446 enhancement johnmartin aron.carroll demo phase 4 assigned Create demo dataset history/comparison page "This is a low priority for the demo site. Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62414120/comments Implementation: http://s031.okserver.org:2375/dataset/history/adur_district_spending" 1338210738000000 1352658819000000 2442 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo dataset overview "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61752357/comments Implementation: http://s031.okserver.org:2375/dataset/adur_district_spending" 1338210249000000 1339670612000000 2444 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo dataset related page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments Implementation: http://s031.okserver.org:2375/dataset/adur_district_spending/related" 1338210559000000 1339670232000000 2528 enhancement aron.carroll aron.carroll demo phase 1 closed fixed Create demo edit dataset page "Implementation: http://s031.okserver.org:2375/dataset/edit/5b451cab-8671-4513-91c4-257784c1f484" 1339686982000000 1340617176000000 2453 enhancement aron.carroll aron.carroll demo phase 5 closed fixed Create demo edit user profile page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments Implementation: http://s031.okserver.org:2375/user/edit" 1338211351000000 1343924178000000 2596 enhancement toby aron.carroll opendatasuite 1 closed fixed Create demo extension Should have all supported extensions enabled, those that don't work should be disabled and ticketed. Also needs a customised homepage. 1340642315000000 1345209485000000 2456 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo forgot password page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments Implementation: http://s031.okserver.org:2375/en/user/reset" 1338211604000000 1339670080000000 2449 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo group page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments Implementation: http://s031.okserver.org:2375/group/test-group" 1338211060000000 1339670041000000 2448 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo groups overview page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments Implementation: http://s031.okserver.org:2375/group" 1338210960000000 1339669678000000 2440 enhancement toby aron.carroll demo phase 1 closed fixed Create demo homepage "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments Implementation: http://s031.okserver.org:2375" 1338206405000000 1340373461000000 2454 enhancement toby aron.carroll demo phase 5 closed fixed Create demo login page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments Implementation: http://s031.okserver.org:2375/en/user/login" 1338211420000000 1343317795000000 2455 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo register page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998362/comments Implementation: http://s031.okserver.org:2375/en/user/register" 1338211541000000 1338998214000000 2461 enhancement toby aron.carroll demo phase 4 assigned Create demo revisions page "I believe this is to be merged in with site activity. Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts Implementation: http://s031.okserver.org:2375/en/revision" 1338212055000000 1344543838000000 2441 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Create demo search page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments Implementation: http://s031.okserver.org:2375/dataset/" 1338210155000000 1339670118000000 2458 enhancement aron.carroll aron.carroll demo phase 3 closed fixed Create demo statistics page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998513/comments Implementation: http://s031.okserver.org:2375/en/stats" 1338211906000000 1344262256000000 2457 enhancement johnmartin aron.carroll demo phase 4 assigned Create demo tags list page "This includes the tag page as well for now. Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments Implementation: http://s031.okserver.org:2375/en/tag" 1338211735000000 1352658878000000 2451 enhancement johnmartin aron.carroll demo phase 5 assigned Create demo user list page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments Implementation: http://s031.okserver.org:2375/user" 1338211213000000 1352658902000000 2452 enhancement aron.carroll aron.carroll demo phase 5 closed fixed Create demo user profile page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments Implementation: http://s031.okserver.org:2375/user/aron" 1338211295000000 1343924143000000 1273 requirement amercader ckan-backlog new Create docs for API v3 1313412083000000 1313412083000000 1475 enhancement kindly kindly ckan-sprint-2012-01-23 closed fixed Create documentation showing how to make an extension that use celery. 1321826681000000 1327576068000000 527 story pudo rgrp closed fixed Create domain model and load sample data 1282899025000000 1293097531000000 2518 enhancement johnglover johnglover ckan-sprint-2012-06-25 closed fixed Create EC ODP CKAN release branch " * branch from 1.7.1 * apply CKAN patches that are current part of installation instructions * create i18n patch so custom i18n directories can be specified via a config option" 1339581818000000 1339595863000000 48 enhancement rgrp johnbywater ckan-v1.4-sprint-1 closed fixed Create favicon Because CKAN's worth it! 1201202789000000 1297077492000000 2372 task aron.carroll aron.carroll closed fixed Create frontend style guide "This should cover formatting and best practices for HTML, CSS and JavaScript Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch. " 1336071043000000 1343220160000000 743 story johnbywater closed wontfix Create harvest source registration via Web UI 1288012860000000 1294412807000000 1821 enhancement kindly kindly ckan-sprint-2012-03-05 closed fixed create internal multilingual extension that brings together all multilingual features "This extension point needs to combine. * Making sure the correct data get put in search index. * Make the search queries are modified to weight multilingual search correctly. * View pages, including search results are translated. * Bring together all search schema modifications." 1329771267000000 1330990111000000 2542 enhancement seanh seanh ckanbuild accepted Create jenkins job to run ckanbuild, and run tests It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests. 1339775888000000 1340639863000000 2502 enhancement toby aron.carroll ckan-sprint-2012-06-25 closed fixed Create Jinja block for common helpers Most important is {% snippet %} but {% url_for %} would be great too. 1339084936000000 1339494853000000 383 task johnbywater johnbywater v1.1 closed fixed Create middleware feature to generate QoS throughput log 1280347595000000 1280854578000000 757 task thejimmyg johnbywater ckan-v1.3 closed duplicate Create migration script to add harvest source filter attribute to existing tables 1288040584000000 1296593448000000 392 task johnbywater johnbywater v1.1 closed fixed Create monitoring feature to generate QoS availability log 1280446769000000 1280496140000000 391 task johnbywater johnbywater v1.1 closed fixed Create monitoring feature to generate QoS responsiveness log 1280446714000000 1280854592000000 2375 enhancement toby aron.carroll demo phase 5 closed fixed Create new base theme "As designed by Sam Smith -> https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/ Also to be used as the demo theme although I think they will diverge at some point. At the moment they should be implemented as one and the same." 1336469009000000 1343317873000000 398 task johnbywater johnbywater ckan-v1.3 closed fixed Create Public API documentation for DGU (draft) 1281000828000000 1294407372000000 522 requirement pudo pudo iati-2 closed fixed Create publishing entity account " * Publishing entity emails system editors * System editor creates publishing entity account and emails publishing entity with details * Priority: 4/5" 1282893668000000 1287392999000000 964 enhancement rgrp rgrp datapkg-0.8 closed fixed Create Resource Downloader and make it pluggable In downloading packages we need to download resources and we need a ResourceDownloader object for this. these should be pluggable so that we can add support for different types of resources. 1297211004000000 1297211237000000 2580 enhancement kindly toby demo phase 1 closed fixed create resource validation "david I've got the add resource working but it is happy to add empty resources which implies that it isn't doing validation correctly. I'm not sure what the validation rules are so hard to test this - I'm coming through as an api call let me know if i can help" 1340285208000000 1342092353000000 359 requirement pudo rgrp datapkg-0.7 closed fixed Create Storage with pluggable extensions "Storage represents something you can upload to (download is done in ticket:406). * storage sub package with defined interface (maybe just use OFS) * Only support uploading single files for the moment (though can do multiple files). * Relationship to build (ticket:306) * No support for syncing e.g. hg/git repos -- that's up to you as a user * Upload command At end of this we can deprecate Repository. Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication." 1277750654000000 1291135692000000 574 story johnbywater closed duplicate Create UKLII package with attributes from remote metadata record 1284045805000000 1284222410000000 111 enhancement rgrp dread v0.10 closed fixed Create user object "Object properties: * id (uuid) * apikey (uuid) -- migrated from the apikey table * name (username = openid) /account is being renamed to /user in the wui." 1252315994000000 1252331626000000 403 requirement pudo closed fixed Create web service for OFS storage "Create a thin web service that will allow users to upload files to OFS on S3 without knowing S3 credentials. This could also provide static urls for all stored resources. * relates to #307" 1281346110000000 1287392451000000 1104 defect dread johnlawrenceaspden ckan-v1.4-sprint-7 closed fixed create-test-data doesn't index the packages it creates "With the default test data created by paster db clean paster db init paster create-test-data going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy But none of those words ""Wonderful"", etc produce search hits. In fact as far as I can tell, nothing produces any search hits. That isn't true on ckan.net, where searching seems to work. " 1303494635000000 1303920791000000 2390 enhancement dread dread ckan-sprint-2012-05-29 closed fixed create_test_data for multiple users, rights It would be useful (well for DGU, but no doubt for core too) to have be able to create multiple users with full properties using one commit. Also to be able to create rights (UserObjectRoles). 1337098459000000 1337100788000000 513 task dread closed wontfix Creating a package with API should warn if there are incorrect fields "When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK. " 1282754802000000 1302774329000000 432 defect dread dread ckan-v1.2 closed fixed Creating package over REST gives 500 error "This occurs when CKAN is run with mod_wsgi (not under paster). This is because of a unicode header being creating in rest.py." 1282065101000000 1282552819000000 1484 defect dread dread ckan-sprint-2011-12-05 closed fixed Creating package with duplicate tags causes exception "e.g. {{{ curl http://localhost:5000/api/action/package_create -d '{""name"":""test5"", ""tags"":[{""name"": ""tag6""}, {""name"": ""tag6""}]}' -H 'Authorization:tester' }}} Also broken in Web UI (adding a package). Affects API and Web UI in 1.5 and I think all versions from 1.4.1" 1322052046000000 1322059006000000 164 enhancement wwaites dread v1.0 closed fixed Creating RDF data from CKAN "Related to: ticket:90 Link to RDF version of CKAN data and ticket:163 (regularly convert CKAN data to RDF). Convert ckan data to RDF with suitable ontology and put on semantic.ckan.net." 1256052643000000 1271251422000000 175 enhancement rgrp dread v0.11 closed fixed Creation date "'created' attribute on User, Group and Rating datetime object: default=datetime.datetime.now() See example on revision Requires migrate script" 1256754070000000 1265291037000000 1127 CREP sebbacon closed fixed CREP0001: Formalise new feature discussion and definition using CREPs "'''Proposer:''' Seb Bacon[[BR]] '''Seconder:''' Rufus Pollock[[BR]] [[BR]] == Abstract == When adding major new features to CKAN, a longer, more formal discussion will improve software design quality and documentation, better engage the wider community, and ensure the core team are up to date with latest developments. I propose a formal process (CREP -- CKAN Revision and Enhancement Proposal) for making this happen. == The Problem == The current workflow for introducing major new features into CKAN is very informal, typically based around one person's great idea, which they've discussed with one or two other people in the team. The originator of the idea is typically the only person with access to all the input they've had through such discussions. Often, the only location of this information is in that person's head. However, there is a lot of experience embodied in the CKAN community which should be drawn on before making large design decisions. This will lead to better software. Additionally, building consensus in the community around a proposal before implementation ensures positive community engagement and buy-in to new features, making them more likely to be a success. We aren't great at documenting new features. Documentation after coding is complete is an unrewarding experience for most programmers. Requiring skeleton documentation before code is written is a good discipline that can form the basis of better documentation in the future (e.g. by a writer rather than a programmer). == Specification == Minor features don't require a CREP, and can just be entered in the issue tracking system as a bug or feature. As a rule of thumb, a feature is major if it will take more than a day to implement, or is likely to involve matters of opinion in its design. A developer may decide that a CREP is too formal and long-winded. The decision to write a CREP is at at their discretion; however, new features MUST always be proposed via email, even if this is just a couple of sentences. If a feature requires a CREP, the proposer should find a seconder for their idea. This sanity check step happens before a CREP is written to ensure at least the possibility of consensus on the CREP. Next the proposer should write a CREP, starting by copying and pasting the [http://wiki.ckan.net/CKAN_Revision_and_Enhancement_Proposals template on the wiki] into a new Trac ticket. This will be with a status of ""new"" and Type of ""CREP"". The proposer should notify the [http://lists.okfn.org/mailman/listinfo/ckan-dev ckan-dev] mailing list, and possibly the [http://lists.okfn.org/mailman/listinfo/ckan-discuss ckan-discuss] list for less technical CREPs. The draft can be discussed via email, verbally, or via the trac ticket. In any case, it is the proposer's responsibility to keep the CREP updated to reflect the current consensus. Once consensus has been reached, the ticket should be marked with the ""accepted"" status and assigned to a CKAN release milestone. When an accepted CREP has been implemented, it should be resolved as ""fixed"". If no consensus can be reached on a draft CREP, or for some reason an accepted CREP doesn't get completed, it should be marked as or ""wontfix"". If a completed CREP becomes obsolete, it should be marked as ""invalid"", with a note pointing to the obsoleting ticket(s) == Why do it this way == Given the distributed nature of the core team plus other volunteers, some kind of written procedure is necessary to ensure a fully documented and discussed proposal. The idea of ""Enhancement Proposals"" which can be semi-formally proposed and discussed prior to implementation is common in the Open Source world ([http://www.python.org/dev/peps/pep-0001/ PEPs], [http://dep.debian.net/ DEPs], [http://plone.org/documentation/glossary/plip PLIPs], to name three). Existing historic proposals exist, called CEPs. The proposed system is called CREP (CKAN revision or enhancement proposal) to disambiguate it from the legacy proposals, and from the delicious fungus ''Boletus Edulis''. Giving a formal structure to the proposal is useful as it gives the community a means to identify a CREP that's not had sufficient thought or discussion. An informal email thread can easily be lost and important questions (such as backwards compatibility) overlooked. The use of the proposed template empowers any community member to ask the proposer to expand on rationale, deliverables, etc. The structure chosen is somewhere between Debian's and Plone's. It aims to give a structure to the debate, a clear start at documentation, and also prompt some thinking about implementation and timescales. All this policy about structure should not be construed as mandatory. In particular, the later fields in the CREP template regarding Implementation Plan may be omitted if the author doesn't find them helpful. Some projects (e.g. Debian) keep their enhancement proposals in a versioning repository; others (e.g. Plone) keep them in an issue tracking system. Trac is proposed for CKAN because we already use it for small feature proposals and for team planning. It seems unlikely that change tracking on an individual CREP will be useful; a CREP that changes sufficiently from its original form should probably be marked ""obselete"" and a new CREP started. Using an issue tracking system also means we can easily track CREPs by state. == Backwards Compatibility == Some [https://bitbucket.org/okfn/ceps/src/76b274888bcf/cep/ legacy enhancement proposals], called CEPs, have previously been started. They are currently all marked as ""active"". Any which require discussion should be altered by the proposer to match the new CREP specification and submitted to trac. The original CEP should be updated with a banner at the top pointing a reader to the new CREP. Any that are now obselete should be clearly marked as such in a banner at the top, pointing a reader to the trac for new CREPs. == Implementation plan == === Deliverables === * This CREP, agreed * Support for proposed statuses in Trac * Canned reports for listing CREPs in Trac === Risks and mitigations === * That this CREP is agreed, but rarely acted on. This risk can be mitigated by nominating a CREP champion in the community or core team, whose job it is to say ""where's the CREP for that?"" and generally own the quality of CREPS === Participants === Seb Bacon: as current Documentation Czar (May 2011), responsible for ensuring CREPs are up to date. === Progress === This document is the entire proposal. " 1304601313000000 1305622850000000 1129 CREP kindly ckan-v1.5 closed fixed CREP0002: Moderated Edits "'''Proposer:''' David Raznick[[BR]] == Abstract. == We are trying to achieve these goals. * To get people involved with making edits to CKAN metadata. * To have an ownership model as to who can moderate and validate these changes * To not put too huge a burden on these owners. In order to achieve this, a feature which lets anyone edit a package but only let the moderator/owner accept it. The moderator should be able to look at a list of changes and accept the ones that This cep is not about 'if' we need such a feature, it is about 'how' we go about implementing it. Another cep may needed for the 'if' case. == The Problem == We need the following to be possible. * Storing revision of objects that are not the current active one. * A way of the user viewing past revisions. * Accessing not only the history of a particular object but also of related objects at that time. i.e If a resource related to a package changes we need a way to see this when looking at the package. * A robust way of doing this in the face of database schema changes. * Make sure database queries are quick. == Solutions. == 1. Store the whole dictization of the package and all its related objects every time you change anything in its dictized representation and only save to the database proper if accepted. Pros * Easy to implement, we already have a preview which makes the dictized form of a package without actually saving it. This will just need to be persisted in some way. * Fast retrieval. * Potential to store a branching revision tree of changes. Cons * No easy way to remake the dictized packages historically or if there is an there a change in the way we represent packages, i.e schema changes. * Will only work for the particular objects we decide to store these changes for. * Stores a lot of repeated information 2. Write specialized queries for every read of the database looking only at the revision tables. This method requires there to be a change in the way we use VDM, so that we manage statefulness ourselves. We will need to add other states such as 'waiting for approval'. Pros * No specialized storage required * Only need to change queries when schema changes * Can be made to work easily for other objects Cons * Slower query time on read, as even looking at the last active package will need to do a fairly complicated query. == Implementation details. == 1. A new table with columns id, user, package_id, timestamp, revision_id, parent_id, dictized_package. revision_id should be null unless it is actually persisted to the database. parent_id is the id that this package_dict was changed from. We could store only the diffs of the dictized_package as long as we assure that everything inside the json is stably sorted, this will make getting the historical data out slower. Getting out the history of the dictized packages is an intensive task, as it will require replaying the whole history of all the changes and creating the dict for each change. This re-caching will need to be redone for every change we make to dictized representation of a package. 2. Every normal packages read needs to look at the revision table to see the last accepted change in the dictized representation of the package. We also need to way to get what the dictized representation of the package was like at any point of its revision history. This querying is non-trivial in sql. === Participants === David Raznick to do it. == Progress. == Decided to go with option 2. However we will change the revisioning system to be like the schema attached. This gets rid of difficult querying problems caused by querying the revision tables by adding an end date, meaning you can do range queries. The better and more normalized version of a revisioning system is outlined [https://docs.google.com/drawings/d/1Y7nMgVsrs081Pame2RdbZHlCAlV33ddTZ8VAsab1j-0/edit?hl=en_GB&authkey=CJfd8vsB]. We will be a step closer to that, with this change, but we will keep the current vdm more or less, intact. " 1304851498000000 1325268100000000 1134 CREP amercader ckan-backlog new CREP0003: Description and Configuration of Harvesters "'''Proposer''': Adrià Mercader == Abstract == The new harvester interface allows to create harvesters for different sources, but right now harvesters don't have many ways to describe and configure themselves. We need a way of allowing them to: * Expose their type and other details so they can be used internally and on the UI. * Define configuration settings for particular harvester instances. == The Problem == === Harvester description === The current UI for adding and editing harvest sources is the same used in ckanext-dgu, and thus the 3 harvester types used in DGU to harvest various GEMINI realted sources are hardcoded in the form. The form will be migrated to a DGU-independent one, so we need the harvesters to provide all the necessary data. There is a current {{{get_type}}} method that returns the harvester type, but for make it compatible with the DGU forms, it returns a machine-readable string (e.g. ""CSW Server""), making it error prone. === Arbitrary configuration === In the current implementation, when the harvest process is started, ckanext-harvest looks for all the available plugins that implement the {{{IHarvester}}} interface and calls the appropiate methods for the current stage ({{{gather_stage}}},{{{fetch_stage}}},{{{import_stage}}}). At these stages, harvesters have no way of applying arbitrary configuration options, so all harvesters of the same type behave on the same way. For instance, the CKAN harvester needs a way to define the API version to use when harvesting remote instances (Right now, the version 2 is hardcoded on the code). == Specification == === Harvester description === Harvesters will need to provide the following information so the UI form can be built: * name: machine-readable name (e.g. ""waf""). This will be the value stored in the database, and the one used by ckanext-harvest to call the appropiate harvester. * title: human-readable name (e.g. ""Web Accessible Folder (WAF)""). This will appear in the form's select box. * description: a description of what the harvester does (e.g. ""A Web Accessible Folder (WAF) displaying a list of GEMINI 2.1 documents""). This will appear on the form as a guidance to the user. The way to provide it will be an {{{info}}} method that all harvesters must implement, which will return a dictionary with the previous elements: {{{ { 'name': 'csw', 'title': 'CSW Server', 'description': 'A server that implements OGC's Catalog Service for the Web (CSW) standard' } }}} === Arbitrary configuration === As different harvesters will have very different needs, we need to provide a way to persist arbitrary configuration flags for each harvest source. The more flexible way given the current architecture in my opinion would be to store the configuration options as a JSON encoded object as a property of the harvest source (There already is an unused DB field called {{{config}}} in the database) (Maybe using JsonType?). This will mean adding an extra field in the harvest source form to allow entering the configuration. This could be just a simple text field where users enter the JSON encoded object or a more clever mechanism (i.e an ""Add a configuration flag"" link that adds two new text fields for the key and value for each flag, and a mechanism to later build the JSON object). In any case, this should probably be hidden in an ""Advance options"" section. == Why do it this way == === Harvester description === The {{{info}}} method would provide a single point to get all the information related to the harvester, and future properties could be added to the dictionary returned without having to modify the interface. === Arbitrary configuration === There is an already existing {{{config}}} field in the database, so we won't need to change the model. Harvesters could access the config object at any of the stages. Of course they could provide default values in their implementations so users don't need to enter them everytime. == Implementation plan == === Deliverables === === Risks and mitigations === The highest risk on the harvesters {{{info}}} method side is that harvester implementation don't offer one of the necessary properties (namely name and title). This could fire a warning when showing the UI form or using the CLI. === Participants === Adrià Mercader to do it. === Progress === None yet." 1305108868000000 1339774554000000 2568 enhancement aron.carroll shevski ckan-sprint-2012-06-25 closed fixed Crop/shorten titles in breadcrumb headings So that they don't interfere with buttons such as here: http://s031.okserver.org:2375/en/dataset/afterfibre/resource/5d19f9fa-c5a4-426c-84de-c624d6e8c3b3 1340039534000000 1340108132000000 1219 defect timmcnamara closed fixed CSS issues on IE7 "As reported on [http://lists.okfn.org/pipermail/ckan-dev/2011-July/001043.html ckan-dev]: > items in the footer of CKAN (""Packages"", ""Groups & Tags"", ""About"", ""Language"", etc.) are shown vertically instead of horizontally in IE7. This works fine in later browsers like IE8, IE9, FF4, and latest Opera and Chrome. > This seems to exist in all recent CKAN versions up to 1.4.1a." 1310423688000000 1310740534000000 1660 defect rgrp lucychambers ckan-sprint-2012-02-06 closed wontfix CSV preview broken - OpenSpending "This CSV resource used to preview but now the format appears to be unsupported: ""We are unable to preview this type of resource: x-osdata-csv"" http://thedatahub.org/dataset/lbhf-spending-2010/resource/9661abbd-2816-4d58-8b20-3cb0eb770c69 This is used as an example by the OpenSpending team all the time." 1326717846000000 1328013627000000 710 task johnbywater johnbywater ckan-v1.2 closed fixed CSW GetRecordById request for given identifier 1287432675000000 1287507854000000 623 task johnbywater johnbywater ckan-v1.2 closed fixed CSW GetRecords request for all identifiers (with CSW authentication) 1284220777000000 1287507837000000 728 requirement amercader johnbywater ckan-backlog assigned CSW Harvesting shall be optimised in respect of reharvesting only records that have changed Hi Will, this is important again because some CSW servers we use have over 300 documents in. Could you take a look at modifying the filter please? 1287675340000000 1310124784000000 205 enhancement rgrp dread closed wontfix Custom package importer "Importer function is exposed to logged in users. Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'. * Basic is as it is currently * Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords. All fields should be in step with the government custom input form. For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text. Estimate: " 1260197819000000 1297068450000000 204 enhancement dread dread v0.11 closed fixed Custom package input form "== As a == admin == I want to == input government data in a constrained way == so that == you get consistency in the db which is easily searched, discovered and automatically linked. == implementation == Form has fields suitable for government data. (related to custom excel importer ticket) In pylons config specify the form schema to use - whether the existing one or the government one. '''On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)''' === Simple fields === * all basic ckan fields * default license to OKD::Crown Copyright * co_id, precision, taxonomy url, agency === Date fields === Date format: machine readable. Can be specific or vague: year or month/year or day/month/year. * date_released * date_updated * temporal_coverage_from, temporal_coverage_to - two date fields combined === Strict selection === Drop down combo * department === Selection or free text === Editable combo * update_frequency - selections: never, weekly, monthly, annually, discontinued * geographic_granularity - selections: national, regional, local authority, ward, point * temporal_granularity - selections: yes, months, weeks, days, hours, point === Specialised fields === * geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global (Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?) * national_statisic - UI is check box, store as 'Yes' or 'No' * resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets. * categories - multi values from strict selection of ONS categories. Stored as comma-separated. === Field data processed === * tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories. === Future - optional === Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views. When you edit a package it detects the schema based on the extra fields that are present. " 1260188965000000 1285082169000000 287 enhancement dread dread v1.0 closed fixed Customise Canadian package edit form "Spec - see attached, which is a copy of the Pirate Pad from Lauren: http://piratepad.net/2C2iwiLDhd" 1270801086000000 1270801210000000 2258 enhancement rgrp ckan-future new Customizable contributor agreement " * Customize text at bottom of forms * Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...)" 1332751549000000 1338204747000000 476 requirement johnbywater ckan-v1.3 closed wontfix Daily database dumps shall be discoverable and downloadable 1282313433000000 1294411741000000 456 story johnbywater dread ckan-v1.2 closed duplicate Daily dump 1282299917000000 1282665858000000 428 requirement dread ckan-v1.2 closed fixed Daily dump of DGU CKAN available End users want to get hold of a JSON & CSV dump of the DGU data. 1281703251000000 1285757316000000 355 defect rgrp rgrp closed fixed Dashes versus underscores in package names Sort out how we deal with dashes versus underscores in package names. 1277221996000000 1311177552000000 2738 enhancement toby shevski closed invalid Data API "Why is datastore not enabled for this resource http://s031.okserver.org:2375/dataset/frere/resource/cdeddb39-b0cb-4762-aa5b-f80fc892d4dd ? Is this going to be a site wide setting? How do new resources get added to datastore? Or are we just using the dataproxy for demo?" 1343123012000000 1343125335000000 1446 enhancement rgrp rgrp ckan-sprint-2012-01-09 closed fixed Data Explorer v2 "Super ticket: #1602 (Data Previewer v2) We already have first pass of Data Explorer that was released as part of #1357. * Merging https://github.com/okfn/dataexplorer with http://github.com/okfn/recline and developing it * https://github.com/okfn/recline/issues Tickets include (* indicates improvement over current explorer) * Core Backbone Models representing Dataset and Tabular data: https://github.com/okfn/recline/issues/10 * New theme: https://github.com/okfn/recline/issues/22 * Read-only mode: https://github.com/okfn/recline/issues/17 * Introduce hash navigation / state support (*): https://github.com/okfn/recline/issues/19 * Re-enable editing in DataTable (*): https://github.com/okfn/recline/issues/13 * [super] DataTable view (in Backbone) (*): https://github.com/okfn/recline/issues/14 * DataExplorer parent view (*): https://github.com/okfn/recline/issues/12 * Simple graph widget using flot: https://github.com/okfn/recline/issues/11 Est: 10-15d (should be broken down -- partly is in recline issues)" 1320665596000000 1326281658000000 412 requirement johnbywater closed fixed Data format for package importing shall be documented "Requirements: * cater naturally for multiple packages with multiple resources * international encoding * large number of packages" 1281430809000000 1289822092000000 438 requirement dread dread ckan-v1.3 closed invalid Data importers use minimal processing e.g. ONS importer 1282224383000000 1294661079000000 433 defect wwaites datapkg-0.8 closed fixed Data package metadata in the Egg "Still not sure if we shouldn't use the existing setuptools machinery to manage this -- there is already a way to get at the metadata. In any event, I've just made an addition to datapkg that makes it possible to put datapkg_sources as a dictionary in your package's setup.py. Afterwards it is possible to pull the metadata out of the egg. Of course this could easily be changed to save the information in whatever form, indeed if you pass it a string instead of a dictionary it will just write whatever you gave it into the datapkg_sources.spec. The point is, I think that the egg is a good place to stuff this information. For non-python users, it is always possible to simply put up the datapkg_sources.spec somewhere on the web so they can directly retrieve the data files. From the docstring:: {{{ This is the implementation for an [egg_info.writers] entrypoint. Datapkg adds an argument to setuptools's setup() function called datapkg_sources. The argument should be a dictionary of the form: .. code-block:: python setup( ..., datapkg_sources = { ""cra2009"" : ""http://www.hm-treasury.gov.uk/d/cra_2009_db.csv"" } ) The result of this is that there will be a file in the egg called datapkg_sources.spec that looks like this:: [sources] cra2009=http://www.hm-treasury.gov.uk/d/cra_2009_db.csv How do you get at this data? Simple:: .. code-block:: python import pkg_resources dist = pkg_resources.get_distribution(""ukgov_treasury_cra"") spec = dist.get_metadata(""datapkg_sources.spec"") and 'spec' will be the contents of the file as a string. }}} " 1282078660000000 1283183647000000 1062 defect johnglover sebbacon ckan-backlog assigned Data preview encoding error "The preview of ""Species Misc Turtle Download"" at http://ckan.net/package/taxonconcept results in the following error: Unable to Preview - Had an error from dataproxy: Data Transformation Error (Data transformation failed. Reason: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte " 1301396143000000 1311773731000000 2674 defect kindly shevski demo phase 5 assigned Data preview not loading on s031 "Not loading for all resources as far as I can tell; e.g. http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360 Fine on demo.ckan.org" 1342435102000000 1344349324000000 2285 enhancement icmurray icmurray ckan-sprint-2012-04-30 closed fixed Data Previewer / Viewer v3 : Embedding! - /resource/../embed "Allow a user to embed the data viewer in their webpage. * [x] Embedding widget [1d] * [x] The embed page [1-2d] * [x] Documentation and Announce [0.5d] Depends upon: * [x] Recline improvements [?d] All the above, plus user stories are fleshed out in [1] [1] http://ckan.okfnpad.org/feature-2283-data-viewer-v3?" 1334230810000000 1335440451000000 907 enhancement rgrp ckan-v1.3-sprint-3 closed fixed Data previews: preview package resource data in the WUI "As a user I want to preview a dataset ('resource') before downloading it. == Implementation == This will be implemented as an [wiki:Extensions Extension] NB: This ticket requires a way to get data out of a remote url (the dataset). We do '''not want to put this data load/streaming into ckan core''' due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file) 1. Need a 'data proxy' - see ticket:698 and ticket:888 2. Preview using javascript in WUI - do this in an extension http://bitbucket.org/okfn/ckanext-dataapi " 1295261173000000 1297072303000000 2203 enhancement kindly rgrp ckan-v1.9 assigned Data Viewer support for PDF " * Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf * propose we use pdf.js - https://github.com/mozilla/pdf.js * We could also try iframing for browsers with native support but this seems more complex" 1330860293000000 1348568961000000 3012 enhancement toby toby new data.gov auth 1352393409000000 1352393409000000 502 requirement dread rgrp ckan-v1.3 closed wontfix Data4nr uploading script shall remove deleted packages "Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed. " 1282659304000000 1292586466000000 2270 enhancement johnglover johnglover ckan-sprint-2012-04-16 closed fixed DataExplorer overview * get up to speed with DataExplorer code (and dependencies like Backbone) 1333374339000000 1334073875000000 2380 enhancement ross ross opendatasuite 1 closed duplicate DataGM Upgrade Provide a new test install of DataGM based on the ODS demo site being completed through June. 1337002144000000 1340627558000000 2529 enhancement rgrp ckan-v1.9 new DataHub (or CKAN) widgets "Simple widgets in pure JS. For example: * Count of datasets in a group (could generalise to a query but not sure how useful that is ...) * Embeddable list of top X (5) datasets for a given query * Embeddable list of *my* datasets Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query). Or: we have this attached to areas of site where relevant. Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page). Cf. http://okfnlabs.org/ckanjs/" 1339750049000000 1340624917000000 2530 enhancement kindly rgrp ckan-v1.9 new DataHub purge fails on some revisions "See http://datahub.io/ckan-admin/trash and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like: Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (IntegrityError) update or delete on table ""revision"" violates foreign key constraint ""group_extra_revision_revision_id_fkey"" on table ""group_extra_revision"" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table ""group_extra_revision"". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'}" 1339750498000000 1341268280000000 306 enhancement rgrp rgrp closed duplicate datapkg build command "Need to be able to build a distribution. Need: * new 'build' command * specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution. " 1272474212000000 1318181194000000 307 defect rgrp rgrp datapkg-0.7 closed fixed datapkg upload command "Support uploading a distribution somewhere. * New cli command: upload * Pluggable uploader framework. Suggest initial support Depends on ticket:306" 1272474224000000 1288027815000000 2888 enhancement kindly new Datapreview in Iframe Try and put datapreview in Iframe. 1345745461000000 1345745461000000 2519 enhancement ross ross ckan-v1.9 closed fixed Dataproxy (converters) enhancement " Whilst the error messages returned from the dataproxy are informative, they aren't particularly human readable. It isn't clear to the user whether they should retry, or whether the format of their data is causing the problem etc... Would be lovely if the error messages returned contained a human readable version of the error." 1339584955000000 1345565215000000 2577 enhancement ross ross ckan-v1.9 closed fixed Dataproxy raises 500 when it can't get CSV "When given a url parameter for a file it cannot fetch, the app raises a 500 instead of something useful. Better error reporting in the JSON would be useful" 1340207124000000 1346669735000000 2513 enhancement ross ckan-backlog assigned Dataproxy should not default to utf8 "Unless explicitly told by the source web server the dataproxy should not assume that the content it has can be encoded as UTF-8. Even though the chars from 128 - 255 overlap an attempt to decode some byte array as utf8 will fail whenever a latin1 char whose bitpattern has the MSB set. This will mean that the UTF8Recoder can be more rigid in its acceptance of data, Postel aside." 1339575820000000 1346669646000000 1425 enhancement zephod dread closed fixed Dataset delete button "From Jonathan gray: At the moment to delete a package you have to go to 'state' and change from 'active' to 'deleted'. For me this was not very obvious, and someone recently wrote to me about this too. Its a small issue - but unless you know what 'state: active' refers to, its not obvious what this means, nor that this is where to go if you want to delete a package. From UI perspective this could be replaced by either a trash can or some text that says 'delete', perhaps with some prompt that explains what happens when you delete a package - explaining that non-admins won't be able to see it, and how you purge, etc. " 1319649860000000 1330085282000000 2735 enhancement toby shevski demo phase 5 assigned Dataset order on user page "I think the datasets on user pages http://s031.okserver.org:2375/user/me should be ordered by latest updated (with most recent at the top) instead of in alphabetical order. What do you think?" 1343062877000000 1344349245000000 2326 enhancement toby toby ckan-future new dataset permalinks "create permalinks eg \dataset\~a5-DQ\nice-name ~XXXXX defines the permalink the nice-name is ignored * packages table needs a new field for this data * existing datasets need updating * new datasets need the identifier creating * routing needs updating * needs to work for all urls like edit Once this is implemented we can decide when we will use permalinks vs the normal urls." 1335371946000000 1338212799000000 1832 enhancement dread ckan-backlog assigned dataset purge API "Purging datasets (deleting them fully, not just changing the state to 'deleted') is important for users testing dataset creation over the API on a test CKAN instance. Without this, they need to resort to more difficult methods such as: * cleaning and reloading the database * setting the test datasets to state 'deleted' and also appending a suffix '_00' and incrementing the number until there is no clash of names. Requested for NHSIC. == Implementation == * This could slot into the Action API. * Of course we would need to ensure the user's had been given the specific right to purge. * I suggest we log the full details of the dataset being purged." 1330077745000000 1339773706000000 2908 enhancement johnmartin demo phase 5 new Dataset related with image should retain aspect ratio At the moment if you have an related item with an image associated to a dataset that appears in the side column it's resized but doesn't keep it's original aspect ratio. 1346931162000000 1346931162000000 1683 defect dread dread ckan-sprint-2012-01-23 closed fixed Dataset search results - last item out of order "On each page of package search results, all the items are neatly sorted apart from the last item of the page. SOLR gets the sorting of the results incorrect. This is a known issue: https://issues.apache.org/jira/browse/SOLR-1777 affecting SOLR 1.4 only (which comes with Ubuntu 10.04) It is highlighted in CKAN test ckan/tests/functional/test_pagination.py:TestPaginationPackage of commit 39096ed54bda86d043521b08b2e14fc5e283a0ff which fails most of the time it is run (passes intermittently)." 1326971864000000 1326976925000000 1745 enhancement rgrp ckan-v1.9 new Dataset search UX improvements as of Jan 2012 "Changes to make search both more exploratory and more satisfying to use * Search query build - #1603 * Ability to add new facet fields ""live"" * That is add fields which then contain faceted options (a bit like data.hri.fi) * (??) Autocomplete / drop down on search (i.e. search while you type) * Dubious about value / cost ratio here Probably would involve to pure JS and HTML implementation. == Implementation == Probably require * API changes to expose solr style API directly #1737 " 1328224941000000 1340033358000000 2490 defect seanh seanh ckan-sprint-2012-06-25 closed fixed Dataset term translation isn't working on user pages, ckanext/multilingual test failing 1338822577000000 1338824047000000 2879 enhancement ross ross ckan 2.0 closed wontfix Datasets (Package) should have a 'public' field "The Package model should have a boolean field added to it describing its visibility beyond that supplied via auth. The related permission for viewing packages where public is set to False would be package.view (show, whatever), and *never* visible for unauthenticated users." 1345466389000000 1350562096000000 2880 enhancement icmurray ross ckan 2.0 closed wontfix Datasets (Package) should have owners "Each dataset should have an explicit owner that is an organisation. This is the organisation that owns the dataset* and will be used for specifying who can move a dataset out of the default organisation. This should probably be a foreign key link from dataset to organisation (or rather group). * Currently this won't necessarily be the organisation that the dataset is a member of, but it is likely that this will be the case initially (at least until someone moves it to another organisation)." 1345466535000000 1350561968000000 2278 defect rgrp rgrp ckan-sprint-2012-04-16 closed fixed DataStore enabled is disabled by default but is needed for DataStorer to run "Major UX gotcha reported by several people is that DataStore is disabled by default and must be enabled for DataStorer to run and import data. Suggest switch to enable datastore by default for file and file.upload types (disable for API). People can always disable if they want." 1333441048000000 1333441177000000 2733 enhancement johnglover johnglover ckan-v1.9 new Datastore logic functions "== Where does the data go? == In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url. The user should have rights to create tables. == Whats the api like? == We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern. == What are the initial logic functions? == * datastore_create * datastore_delete * datastore_show == What is the JSON input format for datastore_create == To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id. {{{ { resource_id: resource_id # the data is going to be stored against. fields: a list of dictionaries of fields/columns and their extra metadata. records: a list of dictionaries of the data eg [{""dob"": ""2005"", ""some_stuff"": ['a', b']}, ..] } }}} * The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions. * rows: can be empty so that you can just set the fields * fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering. eg: [{""id"": ""dob"", ""type"": ""timestamp"" }, {""id"": ""some_stuff"", ""type"": ""text""}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though. * Any error results in total failure!! For now pass back the actual error. * Should be transactional == What json does datastore_delete take? == {{{ { resource_id: resource_id # the data is going to be deleted. filters: dictionary of matching conditions to delete e.g {'key1': 'a. 'key2': 'b'} this will be equivalent to ""delete from table where key1 = 'a' and key2 = 'b' "". No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table. } }}} == What json does datastore_search take? == {{{ { resource_id: resource_id # the data is going to be selected. filters : dictionary of matching conditions to select e.g {'key1': 'a. 'key2': 'b'} this will be equivalent to ""select * from table where key1 = 'a' and key2 = 'b' "" q: full text query limit: limit the amount of rows to size default 100 offset: offset the amount of rows fields: list of fields return in that order, defaults (empty or not present) to all fields in fields order. sort: comma separated field names with ordering e.g ""fieldname1, fieldname2 desc"" } }}} Some free code: https://gist.github.com/3163864 == What json does datastore_search return? == {{{ { fields: same type as datastore_create accepts (i.e. with metadata) offset: The same offset that was supplied in datastore_show limit: The original limit filters: The filters that were applied in data_show total: # total matching records without size or offset records: [same as data_create] # list of matching results } }}} On error will return: {{{ { __error__: … sql error … } }}} == What types are allowed? == Aim to support as many postgres/postgis types that have string representations. http://www.postgresql.org/docs/9.1/static/datatype.html http://www.postgresql.org/docs/9.1/static/sql-createdomain.html == IDs == Each row in a table will be given an _id column which has an id generated by us which you can use in queries. == Other Features == Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc." 1343058886000000 1343656105000000 2291 enhancement kindly rgrp ckan-v1.9 new DataStorer incorrectly attempts to store JSON in DataStore "See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670 This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do). Assigning to David for triage and review." 1334431147000000 1340636737000000 2383 defect icmurray icmurray ckan-sprint-2012-05-29 closed fixed Date on add/edit resource breaks if offset is specified 1337080275000000 1338197024000000 1344 enhancement kindly kindly ckan-sprint-2011-10-10 closed fixed datetime error json conversion on search Json decoding error on search, due to date in resources. 1316020195000000 1317211299000000 1205 defect kindly dread closed fixed db upgrade problem on cz.ckan.net "When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below). cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan. In the meantime I've reverted the db to the old version (pre-upgrade). cz.ckan.net is not working properly until this is fixed. {{{ okfn@eu19:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://:@/ module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects 2011-06-29 09:07:41,294 INFO [migrate.versioning.api] 28 -> 29... /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname ret = fn(self, con, *args, **kw) /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname ret = fn(self, con, *args, **kw) /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname ret = fn(self, con, *args, **kw) /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname ret = fn(self, con, *args, **kw) Traceback (most recent call last): File ""/usr/bin/paster"", line 18, in command.run() File ""/usr/lib/pymodules/python2.6/paste/script/command.py"", line 84, in run invoke(command, command_name, options, args[1:]) File ""/usr/lib/pymodules/python2.6/paste/script/command.py"", line 123, in invoke exit_code = runner.run(args) File ""/usr/lib/pymodules/python2.6/paste/script/command.py"", line 218, in run result = self.command() File ""/usr/lib/pymodules/python2.6/ckan/lib/cli.py"", line 84, in command model.repo.upgrade_db() File ""/usr/lib/pymodules/python2.6/ckan/model/__init__.py"", line 163, in upgrade_db mig.upgrade(self.metadata.bind, self.migrate_repository, version=version) File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py"", line 185, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File """", line 2, in _migrate File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py"", line 160, in with_engine return f(*a, **kw) File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py"", line 364, in _migrate schema.runchange(ver, change, changeset.step) File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py"", line 83, in runchange change.run(self.engine, step) File ""/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py"", line 140, in run script_func(engine) File ""/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py"", line 152, in upgrade migrate_engine.execute(q) File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1788, in execute return connection.execute(statement, *multiparams, **params) File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1191, in execute params) File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1271, in _execute_clauseelement return self.__execute_context(context) File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1302, in __execute_context context.parameters[0], context=context) File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1401, in _cursor_execute context) File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py"", line 1394, in _cursor_execute context) File ""/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py"", line 299, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table ""package_group_revision"" violates foreign key constraint ""package_group_revision_continuity_id_fkey"" DETAIL: Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table ""package_group"". 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'} }}}" 1309339700000000 1310568631000000 2897 defect seanh ckan-v1.8.1 new db_to_form_package_schema() crashes historical dataset page "If there's an IDatasetForm plugin active with a db_to_form_schema() method based on db_to_form_package_schema(), then viewing historical versions of datasets (e.g. http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191) crashes. Reason is that revison_id and revision_timestamp need to be added to db_to_form_package_schema(), something like: schema.update({'revision_id': [validators.ignore_missing, unicode]}) schema.update({'revision_timestamp': [validators.ignore_missing, unicode]}) " 1346330495000000 1346330495000000 2815 defect seanh seanh ckan-v1.8.1 new db_to_form_package_schema() strips tracking summary, isopen If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates. 1344444427000000 1350303821000000 977 defect dread ckan-v1.3 closed fixed db_upgrade errors " * db_upgrade (in ckan/model/__init__) makes call to validate_authorization_setup but forgets about rest of initialisation. * it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?)" 1297420558000000 1297420742000000 909 enhancement pudo ckan-backlog assigned DCat importer for CKAN "Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported: * CKANrdf generated exports * opengov.se RDF (not really DCat) * Sunlight Nationdatacatalog as harvested by dcat-tools. " 1295265958000000 1346669602000000 228 enhancement rgrp rgrp closed duplicate Deal with duplicate packages This needs to be thought out ... 1262085763000000 1290596875000000 1125 enhancement dread nils.toedtmann closed fixed "Debian package ""ckan"" should not depend on ""postgresql""" "The debian package ""ckan"" with the two scripts ""ckan-create-instance"" and ""ckan-instance-maintenance"" depends against ""postgresql"". But ""ckan-create-instance"" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs. Please add a flag ""--without-local-db"" to ""ckan-create-instance"" and remove the postgres dependancy from the debain package." 1304538095000000 1310134813000000 1014 defect sebbacon sebbacon ckan-v1.4-sprint-3 closed fixed Decouple oAuth repoze extension from DataNO, write tests The current oAuth extension tightly couples repoze.who, ckan, and data.no. These components should be decoupled; they also need more tests 1298887493000000 1299245293000000 696 task johnbywater johnbywater ckan-v1.2 closed fixed Decoupled generation of package group choices from validation of package group selection 1287066070000000 1287392206000000 1025 enhancement dread dread ckan-v1.4-sprint-3 closed fixed Default authz can be set in config "Currently the default authz for a package is hard-coded to: {{{ , , }}} This should be configurable in the config, so that you can have a more locked down instance etc. " 1299596110000000 1299751045000000 468 task johnbywater johnbywater ckan-v1.2 closed fixed Default config file has commented out configuration variable 1282310659000000 1283250799000000 1066 enhancement dread dread ckan-v1.4-sprint-5 closed fixed Default reader role too permissive "The definition of the 'reader' role includes creating packages, which is too permissive for some CKAN instances (e.g. DGU). 'Reader' suggests only reading, so I think this role should avoid creating and editing. All projects so far want all roles to be able to create users, so this stays as a Reader action for now, as a convenience. Implementation: * Action.PACKAGE_CREATE removed from reader's default_role_actions * Visitor has a new default role, called 'anon_editor' which can edit packages, but not groups / auth groups - you have to log in for that. * Migration script not needed? * Code comments written, to make clear the suggested policy" 1301645250000000 1301932136000000 1679 enhancement dread ckan-backlog new Default roles problem "The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py). The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different. We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?" 1326823042000000 1339773923000000 1270 defect dread dread ckan-sprint-2011-10-28 closed fixed Default site www.ckan.net is a redirect and loses POSTs "The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters: {{{ In [1]: import ckanclient In [2]: c = ckanclient.CkanClient() In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api') In [4]: c.package_search('', search_options={'groups':'openspending'}) Out[4]: {u'count': 2102, u'results': } In [5]: c2.package_search('', search_options={'groups':'openspending'}) Out[5]: {u'count': 29, u'results': } }}} Thanks to borior for finding this and raising it." 1312911847000000 1312912188000000 1039 enhancement dread dread ckan-v1.4 closed fixed Default user roles read from config (instead of being hard coded) 1300212822000000 1300212856000000 2834 enhancement aron.carroll demo phase 4 new Defer all publish calls until all modules have been initialised "Currently if a module uses sandbox.publish() in its initilaize() function then any modules initialised afterwards will miss the event. We should queue all calls to .publish() in the pubsub module in an array until all modules have loaded then iterate over and execute them." 1344532093000000 1344856692000000 1162 enhancement amercader amercader pdeu-1 closed fixed Define a common extra for storing the geographic extent "We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city). Relevant discussion: http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html " 1306408704000000 1308647187000000 583 task johnbywater ckan-v1.2 closed fixed Define fieldset object for harvest source forms 1284211169000000 1284261424000000 1375 enhancement amercader amercader ckan-sprint-2011-10-10 closed fixed Define new package id before sending it to the IPackageController plugins "When creating a package, the package_create logic function sends the newly created package object to all plugins implementing the IPackageController interface https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64 At this point, the changes have not yet been commited and the id of the package is not set (It has a default value of make_uuid, which is set when commited). For this reason, plugins have no way of knowing the package id. E.g. the spatial query plugin needs to know this id to link the package extent to the actual package. One possible solution would be to set the package id in the data dict received by package_create (if it's not defined): {{{ from ckan.model.types import make_uuid if not 'id' in data or not data['id']: data['id'] = make_uuid() }}}" 1317910441000000 1318260008000000 321 enhancement thejimmyg johnbywater closed duplicate Delegate authentication to Drupal "When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model. The Drupal front-end shall have: 1. Login page - fixed location, can authenticate users, on successful authentication sets auth cookie and redirects to HTTP_REFERER. 2. Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions. 3. Access denied page - fixed location, static resource, gently indicates what has happened, and how to ask for permission. The CKAN slave edit page shall: 1. Try to detect a Drupal session key (passed as cookie or as request param). 2. Redirect to Drupal login page if no session key. 3. Check authorisation if session key is found. 4. Redirect to access denied page if session key not authorised. 5. Present the Package edit page. 6. Reject unauthenticated or unauthorised edit submissions. 7. Snag invalid edit submissions from authenticated and authorised users. 8. Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end. " 1274705234000000 1291831399000000 1733 enhancement amercader amercader ckan-sprint-2012-02-20 closed fixed Delete (inactivate) harvest sources "Right now harvest sources can be inactivated (only via the cli), but we need to make sure: * Expose this on the web interface (only for sysadmins, see #1731) * Inactive sources are not shown in the web interface Also for deleted packages: a) Deleted packages get state=""deleted"" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for ""Delete"" for sysamdins Done in #1425. Not sure if the button is only shown to sysadmins. b) No undelete: If you reharvest a deleted package, it remains deleted * Publisher should create a new record with new identifier and same/different content as applicable. * If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated. * You won't get the state where a package is updated in its deleted state * If they change the ID and reharvest, a new record appears on DGU. c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API. " 1328008287000000 1329754689000000 17 enhancement johnbywater johnbywater closed invalid Delete a file See ticket:16 (update a file) 1152550822000000 1250619180000000 13 enhancement somebody rgrp closed invalid Delete a release 1152550611000000 1218545330000000 2685 enhancement toby toby demo phase 2 closed fixed delete buttons should be form buttons stop bots/browsers etc following links and auto deleting stuff 1342510565000000 1342514443000000 2646 enhancement toby toby demo phase 2 closed fixed delete dataset make it so 1341920466000000 1342014170000000 2661 enhancement aron.carroll toby demo phase 2 closed fixed delete dataset/group javascript confirm "in 2375-demo-theme-development templates/package/confirm_delete.html needs to be made to look nice note some writing in black delete button needs nice home on dataset/edit you can add js support add ?confirm=yes to url on dataset button to force delete Please note delete is not yet implemented - but will be in the next few hours" 1342012395000000 1342716070000000 783 story johnbywater johnbywater ckan-v1.3 closed duplicate Delete harvested documents when remote metadata disappears from harvest source We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer. 1289210029000000 1294409399000000 1544 task dread ckan-backlog new delete old git branches "We have 150 odd branches ({{{git branch -a}}}) - most of them old - we should prune them. At very least, branches that have been merged in should be deleted. Look at old branches that haven't been merged in and wonder why. May be of some use: {{{ git branch --merged git branch --no-merged }}}" 1323702610000000 1323702610000000 11 enhancement johnbywater johnbywater milestone16 closed fixed Delete package "== As a == Authenticated user == I want to == Delete a package == So that == So that the package is no longer displayed for visitors of the site == Notes == * The permissions for this action are the same as for updating the package details (see ticket:10) * When a package is deleted it is put in an inactive state and is invisible to users and vistors of the service. However its name remains reserved and it may be undeleted or purged by the service administrator " 1152550328000000 1157371568000000 579 story johnbywater ckan-v1.2 closed Delete remote metadata harvest job 1284209440000000 1285260356000000 2712 enhancement toby toby demo phase 2 closed fixed delete resource functionality needed we need a delete_resource logic action and enable for resources 1342680517000000 1342690104000000 2992 enhancement johnmartin new Delete resource should send me back to edit dataset page 1350407956000000 1350407956000000 1341 enhancement kindly kindly ckan-backlog reopened Delete spam users from ckan Spam users where added to thedatahub and we need to clean them. 1315995034000000 1320141540000000 2728 defect toby shevski demo phase 4 new deleted group shows on search index - for admins 'test-group', which has been deleted,shows up on main search page under groups - and can be filtered by - see http://s031.okserver.org:2375/dataset?groups=test-group 1342950784000000 1345023944000000 2721 defect toby shevski demo phase 4 new deleted groups should not show on 'Add to Groups' dropdown "Groups previously deleted still show up in the add dataset process in step 3 'Additional info' http://s031.okserver.org:2375/dataset/new_metadata/ff" 1342948632000000 1344544214000000 561 defect pudo ckan-v1.2 closed duplicate Deleted packages are returned in the API Anja is reporting this, severe bug, I think. 1283775578000000 1283775711000000 1283 enhancement dread dread ckan-backlog closed fixed Deleted packages shouldn't be searchable or browsable "A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else. (But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.) Same in the Web interface and API. This ticket ties in with #948 highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can #1076." 1314112481000000 1330083933000000 560 defect kindly rgrp ckan-v1.4 closed fixed Deleted packages showing up in tag lists "E.g. http://ckan.net/tag/lod contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins). Also suggested that these are showing up in the API (courtesy of Anja) Cost: 2h" 1283700001000000 1297084192000000 1382 defect thejimmyg thejimmyg ckan-backlog new Deleted resources are present for harvested package Perhaps the importer deletes them before re-importing. We shouldn't have deleted resources, so let's investigate. 1318337889000000 1318337889000000 1053 defect dread dread ckan-v1.5 closed fixed Deletion in Model API "Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted. There is no way to delete objects at the moment - changes to 'state' are ignored in the API. Do we need an alternative way to purge objects in the API?" 1300790039000000 1310126546000000 2595 enhancement toby aron.carroll demo phase 2 closed fixed Demo add dataset needs to remember which stages have been visited "For example. 1. Start on stage one 2. Add dataset click next 3. Click previous Expect Stage 1 to be ""active"", stage two to be ""complete"" 1. Start on stage one 2. Add dataset click next 3. Click next 4. Click previous Expect Stage 2 to be ""active"", stage one and three to be ""complete"" 1. Start on stage one 2. Add dataset click next 3. Click next 4. Click previous 5. Click previous Expect Stage 1 to be ""active"", stage two and three to be ""complete""" 1340642246000000 1342086346000000 2597 enhancement toby aron.carroll demo phase 1 closed fixed Demo add dataset throws error if tags are empty " 1. Go to new dataset 2. Add a slug 3. Click next Get error. {{{ URL: http://localhost:5000/dataset/new File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond app_iter = self.application(environ, detect_start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__ resp = self.call_func(req, *args, **self.kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, *args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__ return request.get_response(self.app) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__ resp = self.call_func(req, *args, **self.kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, *args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/injector.py', line 52 in __call__ response = request.get_response(self.app) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 152 in __call__ return self.wrap_app(environ, session_start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/base.py', line 258 in __call__ res = WSGIController.__call__(self, environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__ response = self._dispatch_call() File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call return func(**args) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 422 in new return self._save_new(context) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 703 in _save_new return self.new(data_dict, errors, error_summary) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 430 in new data['tag_string'] = ', '.join(h.dict_list_reduce(c.pkg_dict['tags'], 'name')) TypeError: string indices must be integers, not str }}}" 1340642399000000 1340644591000000 2492 enhancement toby aron.carroll ckan-sprint-2012-06-25 closed fixed Demo create related form has broken error output "See: http://s031.okserver.org:2375/dataset/adur_district_spending/related/new Submit the form to see broken formatting. eg. [u'Missing value'] [[Image(ckan-broken-error.png)]]" 1338977874000000 1338982678000000 2907 enhancement johnmartin johnmartin demo phase 5 closed fixed Demo header The user login / avatar with in the header really should be within another topbar, as it goes against convention to have the user information within the normal header. 1346930925000000 1352206307000000 2643 enhancement aron.carroll aron.carroll demo phase 2 closed fixed Demo homepage redesign "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments" 1341848685000000 1342617247000000 2339 enhancement seanh seanh ckan-sprint-2012-05-15 closed fixed Demo install on RHEL/CentOS, and install instructions 1335876250000000 1336492657000000 2557 enhancement aron.carroll aron.carroll demo phase 1 closed fixed Demo issues on the search dataset page " * Crosses have disappeared * Formats should stretch across the page" 1340035222000000 1340185609000000 2765 enhancement aron.carroll aron.carroll demo phase 3 closed fixed Demo login error messages "I'd suggest we add a generic ""failed to login, invalid fields"" in the error_summary block. Then more specific ""this field is required"" when a required field is left empty. So in general, provide an error dict" 1343317842000000 1343817858000000 2491 enhancement toby aron.carroll ckan-sprint-2012-06-25 closed fixed Demo related page is missing breadcrumb "The following pages are missing complete breadcrumbs. Add related page: http://s031.okserver.org:2375/dataset/adur_district_spending/related/new Needs to look like. Dataset > {dataset_title} > Related > Create Related Template here: https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/pages/form_page.html#L6 Same deal with Edit related. http://s031.okserver.org:2375/dataset/adur_district_spending/related/edit/5f9e665b-3e3c-428f-88b1-8554189cf14a Needs to look like. Dataset > {dataset_title} > Related > Edit Related https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/edit.html#L5 " 1338977634000000 1338979614000000 2493 enhancement aron.carroll toby demo phase 5 accepted Demo site 404 needs some love "templates/error_document_template.html I've converted the 404 page to the new style but we probably want to serve this as a 'full page' ie content is 100% page width not sidebar and primary content just go to a non-existant url to see Discusion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63374042/ Implementation: http://s031.okserver.org:2375/404" 1338986081000000 1343135815000000 2558 enhancement aron.carroll aron.carroll demo phase 1 closed fixed Demo site add dataset has missing functionality " * Needs tags * Needs groups * Needs slug * Licenses links to something * Needs tooltips (Ira to write some tooltips)" 1340035280000000 1340615257000000 2494 enhancement toby aron.carroll ckan-sprint-2012-06-25 closed invalid Demo site has server error for a 404 "Example http://s031.okserver.org:2375/not-here Should show the 404 template." 1338988913000000 1338994890000000 2555 enhancement toby aron.carroll demo phase 5 new Demo site needs a breadcrumb helper Something to make building breadcrumbs a bit nicer 1340026983000000 1342618384000000 2510 enhancement aron.carroll aron.carroll ckan-sprint-2012-06-25 closed fixed Demo tag lists should link to the search page "Instead of pointing to /tags/{tagname} all tag links should just point to a dataset search filtered by that tag. Examples are on the homepage and the dataset pages." 1339513217000000 1339670520000000 2560 enhancement toby aron.carroll demo phase 2 closed fixed Demo theme add metadata is missing functionality " * Hide remove button when adding new field * Finish takes you to the newly created dataset" 1340035384000000 1342084988000000 2559 enhancement toby aron.carroll demo phase 2 closed fixed Demo theme add resource has missing functionality " * Show resources in the sidebar * Allow people to go backward and forward * Have a back button on each form * Lose resource type * Add extras as optional metadata * Remove save * Add back/Previous * Add Another" 1340035336000000 1342084940000000 2578 defect toby toby demo phase 4 closed fixed demo theme breaks the tests "The demo branch fails huge numbers of tests. many are going to be template based - these need to be sorted - ideally we need to get them to work with legacy/new templates but see how we go on that one anyhow this is not a priority whilst we are branched but needs resolving before we merge into master" 1340276924000000 1347901892000000 2561 enhancement aron.carroll aron.carroll demo phase 1 closed fixed Demo theme edit dataset is missing functionality "* Add tags * Add groups * JavaScript for custom fields" 1340035457000000 1341497328000000 2609 enhancement toby aron.carroll opendatasuite 1 closed fixed Demo theme errors when creating a dataset without a resource "1. Complete step one of add dataset 2. Click next without adding any resource data. Page errors. {{{ URL: http://localhost:5000/dataset/new_resource/asdasd File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond app_iter = self.application(environ, detect_start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__ resp = self.call_func(req, *args, **self.kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, *args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__ return request.get_response(self.app) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__ resp = self.call_func(req, *args, **self.kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, *args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/injector.py', line 52 in __call__ response = request.get_response(self.app) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 152 in __call__ return self.wrap_app(environ, session_start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/base.py', line 258 in __call__ res = WSGIController.__call__(self, environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__ response = self._dispatch_call() File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call return func(**args) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 495 in new_resource get_action('resource_create')(context, data) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/logic/action/create.py', line 240 in resource_create raise ValidationError(errors, _error_summary(errors)) ValidationError: {u'URL': u'Missing value'} }}} http://localhost:5000/dataset/new_resource/asdasd" 1340810211000000 1340900271000000 2526 enhancement aron.carroll aron.carroll demo phase 5 closed fixed Demo theme new custom form fields should be created when needed "Implementation: http://s031.okserver.org:2375/dataset/new_metadata" 1339680105000000 1343924205000000 2525 enhancement aron.carroll aron.carroll demo phase 1 closed fixed Demo theme should auto-generate slugs on forms 1339679954000000 1341497344000000 2562 enhancement johnmartin aron.carroll demo phase 5 closed invalid Demo theme tag improvements "Tags should be implemented in two sections. * Custom predefined tags using vocabularies * Provide suggestions, auto completion then fallback to free text" 1340035714000000 1352206599000000 2421 enhancement aron.carroll toby closed fixed demo-theme pills break in dataset search if pills cross onto 2 lines they split which they shouldn't to see just add some tags/formats till it breaks 1337865196000000 1343220041000000 2814 enhancement shevski markw assigned Demo: upload file behaves oddly "3. Uploading a file behaves counter-intuitively (I would suggest wrongly). When adding a new resource by uploading a file, I select a file called say create-group.png. I expect the following to happen: * the pathname of that file is filled in the box; * nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc). Instead of this, * the file is immediately and, seemingly, irrevocably uploaded; * the box is filled with a mysterious URL (https://commondatastorage.googleapis.com/ckan-demo ...) * the mysterious URL is so long I can't even see the final element, which is my file name. " 1344420360000000 1346670381000000 681 task dread dread ckan-v1.3 closed fixed Department field bug https://trac.dataco.coi.gov.uk/projects/datagov/ticket/613 1286386685000000 1290507180000000 1117 defect thejimmyg nils.toedtmann closed invalid "Depend deb package ""ckan"" against ubuntu package ""python-pastescript""" ... otherwise the scripts fails. 1304089619000000 1304277240000000 411 requirement pudo pudo ckan-v1.2 closed fixed Deploy and test Varnish on eu3 "* Need to figure out cache invalidation for CKAN * Evaluate against an internal cache (i.e. memcached) " 1281349275000000 1287393033000000 915 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Deploy dcat-tools/rdfsolr to publicdata.eu as a placeholder We should present something there, why not start with a meta-search. 1295267040000000 1296467518000000 1687 enhancement ross ross closed invalid Deploy new storage changes from master to datahub We need to deploy the new storage changes (and webstore) to thedatahub for further testing 1327056976000000 1329338671000000 1582 enhancement johnglover johnglover ckan-sprint-2012-01-09 closed fixed Deploy QA for thedatahub - 0.5d " * deploy celery * deploy QA and archiver tasks * write up a blog post announcing QA on thedatahub" 1324458494000000 1326110801000000 1622 enhancement johnglover johnglover ckan-sprint-2012-01-23 closed fixed Deploy QA on DGU UAT test server - 0.5d " * Update CKAN on DGU UAT to 1.5.1 * Deploy Celery * Deploy QA extension" 1326116380000000 1326127702000000 862 defect wwaites wwaites ckan-v1.3 closed fixed deploy rdf on catalogue.data.gov.uk handle various dgu extras, configure apache for autonegotiation, etc. 1291726237000000 1291726261000000 2238 enhancement seanh seanh ckan-future closed duplicate Deploy some test vocabs and publishers on test.ckan.net, check tutorial-style instructions for these 1332165311000000 1343392681000000 1330 enhancement rgrp ckan-backlog closed invalid Deprecate / Remove test_authz.py "test_authz.py appears to test in great detail some very specific additional authz (related to total site lock-down it seems -- introduced I think for hri project). I think there are simpler ways to get total site lockdown (use external auth!) and this test is slow and delicate (e.g. depends on specific words in templates). Suggest removing. If we don't remove we should at least refactor tests for access to certain pages to use a proper method of testing (e.g. agreed html comments in each page) rather than being depending on the presence of absence of specific wording." 1315899129000000 1327060201000000 1252 enhancement rgrp rgrp ckan-v1.5 closed fixed Deprecate ckan.net in favour of thedatahub.org "Deprecate ckan.net in favour of thedatahub.org * Announce * Create thedatahub.org and have it run same data as ckan.net * Announce closing down of ckan.net with 302 * Implement 302 permanent Also redirect wiki.ckan.net to wiki.ckan.org" 1311965339000000 1318160334000000 2313 enhancement icmurray icmurray ckan-v1.8 closed fixed Deprecate old facet data structures and related functions. "In branch feature-1821-multilingual-extension, a new faceting datastructure was introduced. This makes the old one and related functions obsolete. These have been marked as deprecated for the 1.7 release, and should be removed for the 1.8 release. * helpers.py:facet_items() * facets.html:facet_sidebar() * uses of c.facets (rather than c.new_facets) " 1334677916000000 1343124732000000 1006 enhancement kindly rgrp ckan-v1.4-sprint-4 closed fixed Deprecate stable branch "Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time). Cost: 10m (giving high priority because of low cost) (Assigning to dread as he has been managing the stable branch)." 1298624996000000 1300372286000000 1718 defect aron.carroll dread closed fixed Deprecated JS properties used "I get this warning when viewing the home page: {{{ event.layerX and event.layerY are broken and deprecated in WebKit. They will be removed from the engine in the near future. }}} CKAN: latest master Browser: Chrome Ubuntu Minor, but worth checking?" 1327665842000000 1340703280000000 2649 enhancement ross ross ckan-v1.8 closed fixed Description on resources not rendered as markdown "When a resource doesn't have a description, it is rendered as markdown from the description of the dataset. If the resource does have a description it isn't rendered as markdown .. See http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96" 1341923135000000 1342084280000000 2650 enhancement ross ross ckan-v1.8 closed fixed Description on resources not rendered as markdown "When a resource doesn't have a description, it is rendered as markdown from the description of the dataset. If the resource does have a description it isn't rendered as markdown .. See http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96" 1341923137000000 1341943716000000 374 task johnbywater johnbywater v1.1 closed fixed Design servicization of CKAN API 1280223118000000 1280398501000000 581 task rgrp johnbywater ckan-v1.2 closed fixed Determine package name from metadata document Need to establish procedure for automatically generating package names from metadata documents. 1284210874000000 1284689274000000 1712 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed fixed DGU /data page "CKAN provides the /data page page of data.gov.uk Breakdown of tasks: * [x] Analysis / refinement of spec. * [x] Log-in / register as publisher (Waiting on publisher form) * [x] Population of ""browse by publisher"" list groups ordered by most datasets (Waiting on publisher integration) * [ ] ~~Browse by nation~~ * [ ] ~~Featured datasets~~ Now a possible integration point with drupal * [X] Tag cloud * [X] Developers section" 1327590527000000 1329733846000000 1755 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Alpha paged view of publishers The publisher index page should show an alpha paged list of publishers. 1328530634000000 1328631064000000 1758 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Check user management auth Ensure publisher edit page only allows admins to manage users. #1675 is related. 1328530915000000 1328709897000000 1759 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Create publisher form access Only sysadmin can access create publisher form. 1328530956000000 1329732744000000 1751 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed DGU dataset form: collection of updates from feedback "A collection of improvements to the dataset creation/edition form (feedback form DGU). - [x] validation error formatting - [x] geograohic coverage: add local authority - [x] resource format field - [x] resource format auto-complete - [x] renamed data tab -> files tab - [x] integrate Adris'a INSPIRE changeset, and migration scripts as part of build - [x] remove ""discontinued"" from the update frequency - [x] disable the disabling of save - [x] Remove ability to edit url on the edit-form (and remove ""url not available"")" 1328526612000000 1329733458000000 1713 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed fixed DGU dataset search page "Search results page of DGU. Breakdown: [X] ensure faceting by group (publisher) works [X] allow faceting by UKLP dataset type [X] allow faceting by INSPIRE" 1327590700000000 1327932401000000 1714 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed fixed DGU dataset view page "Dataset read page for DGU Breakdown: - [X] Fix missing title - [X] Links to the social media stuff - ""Share your app"" - ""Share your idea"" - ""Request new data"" - [ ] ~~Presentation of additional information should be broken into sections~~ Content is there. Don't do anything until theming/design work is undertaken. - [ ] ~~""More like this"": links to similar datasets.~~ Dropping - [N] ~~Stars~~ (I think these are being removed) - [ ] ~~Comments~~ Possible integration point with Drupal. Needs it own ticket. - [X] Give feedback to department - [X] Tidy up the QA stars" 1327591090000000 1329734018000000 1760 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Extract current publisher hierarchy " Get access to live Drupal MySQL database to get latest publisher list for migration. Write a one-off script for import of the data." 1328531001000000 1328709560000000 505 enhancement dread rgrp closed fixed DGU fieldset supports v3 of DGU metadata specification 1282660416000000 1298368280000000 1757 enhancement ross ross ckan-sprint-2012-03-05 closed fixed DGU Hide administrators from public view On publisher view page - hide administrators from public view - only logged in admins of group should see them. 1328530775000000 1338205187000000 1761 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU History visibility History on user profile should only be visible to a user who is a member of at least one group. 1328532889000000 1328716686000000 1762 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Join publisher form "A new form for create publisher -> Add publisher access to your account' New wizard Autocomplete publisher name if linked from publisher view. Email address used to publish should be activated/validated on first use. Original email still used to log in. " 1328532992000000 1329729305000000 714 bug johnbywater closed fixed DGU package form shall have a read only field for ??? attribute So that Drupal doesn't need to make any adjustments to the form, and risk data being lost on submission. 1287581408000000 1291733788000000 1756 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Treeview of hierarchical publishers The publisher index page should show a treeview of the publishers within the system. Probably behind a tab. 1328530689000000 1328709846000000 1768 enhancement icmurray icmurray ckan-v1.7 closed fixed DGU: 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1 1328539740000000 1337159873000000 1752 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed DGU: collection of updates from feedback " - [x] hide black stars on the data-view page - [x] Edit on view form (if you have the right permission) - [x] RES_FORMAT ~> Resource Format - [x] Map Based Search -> UK Location Map Based Search - [x] Note at the top right expalining that these are seach filters (if they are) -> this requires ""add a dataset"" and ""other access"" to be moved to a different place (just some whitespace is fine) - [x] Display publisher and provider (UKLP records only). - [x] analysis of filter by OGL / non-OGL - [x] provide /dump on the build server" 1328526750000000 1329733515000000 1046 enhancement kindly thejimmyg ckan-v1.4-sprint-5 closed fixed Dictization and the new logic layer "The stages involved with doing this. * Convert model objects to standard dict format (DONE) * Convert standard dicts to current api formats (DONE) * Make standard dicts savable (DONE) * Validate standard dict format. (DONE) * Authorize actions" 1300364694000000 1302777668000000 1217 defect dread dread ckan-sprint-2011-11-07 closed invalid Diff exception "Exception raised doing this particular diff. http://ckan.net/revision/diff/sudocfr?diff_entity=package&oldid=a18b2520-09e0-44be-80dd-4fea94ffe7d1&diff=3934ff16-a3a7-4481-b390-72944903b945 {{{ ... Module ckan.controllers.revision:142 in diff << if c.diff_entity == 'package': c.pkg = model.Package.by_name(id) diff = c.pkg.diff(c.revision_to, c.revision_from) elif c.diff_entity == 'group': c.group = model.Group.by_name(id) >> diff = c.pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:449 in diff << display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] >> display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' }}} Seems to be to do with this particular extra key: {{{ [] }}} Is it because it was created between the from_revision and to_revision?" 1310375498000000 1320663277000000 173 enhancement rgrp dread v0.11 closed fixed Diffing WUI "Use vdm diff functionality (see ticket:82) to implement diffing of packages between versions in WUI. In package history you should be able to select to revisions and show the diff. Estimated cost: 12h" 1256565412000000 1265293968000000 267 defect johnbywater rgrp vdm-0.7 closed fixed Diffs for object that is created in that revision Currently this raises an exception because there is no package/package_revision in existence at from_revision. 1268041383000000 1272960518000000 1520 task dread ckan-sprint-2011-12-19 closed fixed Disable name changing Because of #1514 we should just disable name changing, until #1514 is done. 1323169663000000 1323280999000000 1386 enhancement rgrp rgrp ckan-sprint-2011-10-24 closed fixed Disallow account creation via openid "Superticket: #1343 Creation of accounts from OpenID causes large number of problems: * No guarantee of other profile info * Poor username (just from openid) We therefore will: 1. Require creation of user account via register (no auto-creation of accounts via openid) * sidebar of register page should point to login page rather than openid signin 2. Permit association of an openid with an account ... * How: paste in your openid url (requires change to edit form to have this) * won't work for google with their weird openid urls * [future] Nicer way is to have login via openid while logged in (which association then happening in background) 3. Migration (for 1+2): * (?) Generate decent usernames for all existing accounts * As this is nontrivial suggest instead we allow editing of usernames (by account owner and sysadmins). This is useful in its own right and is a reasonable 80/20 solution. * This does present the problem of re-associating commits with the new username. Best solution to this would be to switch revisions to point to userid rather than username. See #1534 4. [optional] Change display_name to always be username * Change My Account to Username at top right of all pages (once we know usernames are short ...) * could do this straight away by truncating long usernames (e.g. truncate at 20 chars ...) * Also should we lose the icon? " 1318425493000000 1324036930000000 280 defect dread dread v1.0 closed fixed Disappearing extra fields "When I edit a package on no.ckan.net with lots of extra fields, lots of them completely disappear. Metastable branch." 1270209684000000 1271173769000000 477 story johnbywater ckan-v1.3 closed duplicate Discover location of the daily database dumps 1282313788000000 1294411761000000 444 task dread closed duplicate Discuss package relationships ideas with JF "* Create test data on visible ckan * Discuss with JF" 1282299238000000 1294414008000000 1447 defect kindly dread ckan-backlog assigned disk space leakage "Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it. One problem in the past has been file handles running out when creating lots of tiny files in the data directory. Another problem has been several enourmous backups being created every day - pdeu on eu25." 1320666843000000 1340727330000000 124 enhancement rgrp rgrp v1.0 closed fixed Display Generic Package Attributes in WUI Split out from ticket:43 1253709702000000 1254735558000000 260 enhancement dread rgrp v1.0 closed fixed Display group package list in standard (richer) way "Currently use old ""bullet-point"" style. Should change to now standard style used on e.g. package search results page. Cost: 0.5h" 1267100660000000 1267122137000000 1228 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Display OpenID login errors When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again. 1311086562000000 1311325161000000 1227 enhancement timmcnamara ckan-backlog new Display packages' tags in search results "In when displaying search results, it would be useful to also display the tags of a package. Sometimes it's difficult to infer the scope of what the package does from the title and the first sentence of the description. Tags are quite concise way to display rich information. ENV=datacatalos.org, with CKAN 1.4.2a" 1311034262000000 1339774147000000 2202 enhancement rgrp ckan-future reopened Display page view count on dataset and resource pages Just like we display download counts we should display view counts. 1330765455000000 1338204929000000 1529 enhancement dread dread ckan-sprint-2011-12-19 closed fixed Display user name when logged in "Currently when you log in it says ""logged-in"". Most sites show your user-name and this is helpful when you have more than one account or human using the computer." 1323252086000000 1324318628000000 2311 enhancement toby toby ckan-sprint-2012-04-30 closed fixed disqus - improvements "get working on TDH look at removing stream filters" 1334598853000000 1336046186000000 896 story thejimmyg closed duplicate Distributed Data and Syncing Between CKAN Instances "Based on the ideas in: * DistributingChanges * SyncingInstances Merging the tickets #296, #297, #298, #299 into this one. Related ideas include: * Using dcat to exchange data * Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance This should be considered a wishlist item for the timebeing. " 1294406999000000 1300217375000000 2264 enhancement seanh seanh ckan-sprint-2012-04-16 closed fixed Do an audit of string translations in CKAN core and ckanext-ecportal Check to see that all the strings are getting translated, i.e. that all user-visible strings are being passed to gettext. Fix any strings that are not getting translated. 1332864512000000 1339411578000000 240 defect johnbywater rgrp v1.0 closed fixed Do not allow creation of PackageResource object without a url Currently it is possible (v0.11) to have a PackageResource object without a url. This should be impossible to do (either at creation time or via modification). 1265399747000000 1272383770000000 97 enhancement rgrp rgrp closed duplicate Do not create a distribution on a path is something already exists there "(2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option). Cost: 1h" 1249983557000000 1318181317000000 2582 enhancement rgrp ckan-v1.9 new Do not hide notes / readme on dataset pages "Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?) Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket." 1340312340000000 1340625009000000 2505 enhancement amercader amercader ckan-sprint-2012-06-25 closed fixed Docs improvements for 1.7.1 "There are some areas where the documentation could use some improvement and it would be good to have it available for 1.7.1. * Add an introduction to harvesting support * Add an introduction to CSW support * Better upgrade docs: Some users suggested improvements to the upgrade docs on the list [1]. Also we should mention how the upgrade would work for point point releases * Source install: see #2406 [1] http://lists.okfn.org/pipermail/ckan-dev/2012-May/002178.html" 1339166533000000 1340018770000000 822 enhancement Javier de la Cueva closed fixed Docs: Configuration as site-available instead of site-enabled for apache "The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net " 1290107103000000 1290506354000000 1315 enhancement dread dread ckan-sprint-2011-11-07 closed fixed Document /api/action We need some documentation for the /api/action. Even though it's beta and subject to change it would be good to have a guide. 1315396722000000 1320173768000000 994 defect kindly kindly ckan-v1.4-sprint-3 closed fixed document a new migration methodology It is too easy to make a mistake with the migrations at the moment. A more systematic way of doing them is needed and this needs to be documented. 1298281192000000 1298912842000000 2761 enhancement seanh ckan-v1.8 new Document all the errors you can get when setting up filestore, and how to fix them "Add it to a 'Troubleshooting' section on the filestore page: http://docs.ckan.org/en/ckan-1.7.1/filestore.html For the error messages and their solutions, see various threads on ckan-dev" 1343302566000000 1343302566000000 1400 enhancement johnglover rgrp ckan-sprint-2012-03-05 closed fixed Document Archiver Extension " * Create page on http://wiki.ckan.org/ about setting up a queue (this will merge into core docs when this is stable) - having a queue will now become a generic requirement for a bunch of extensions * README with install instructions in ckanext-archiver * Create an entry in http://wiki.ckan.org/Extensions * Write a blog post" 1318600705000000 1331544816000000 2240 enhancement ross ross ckan-sprint-2012-04-02 closed fixed Document CityData ideas Document the ideas raised on 14/03 for CityData and distribute to team for discussion 1332169761000000 1332408769000000 289 enhancement rgrp dread v1.0 closed fixed Document config options 1271248690000000 1271249368000000 825 enhancement dread dread closed fixed Document fabric tool 1290609160000000 1290624449000000 425 task johnbywater johnbywater ckan-v1.2 closed fixed Document how licenses can be configured in CKAN 1281436551000000 1281605524000000 973 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Document license configuration and license system. "Document how license system works and specifically how licenses can be configured. Cost: 2h" 1297332948000000 1297678851000000 1700 enhancement johnglover kindly ckan-sprint-2012-02-06 closed fixed Document server setup for ckan server. Document deployment on tenforce test server. 1327417952000000 1327425247000000 974 defect dread ckan-v1.3 closed fixed Document site_description Configuration option 'ckan.site_description' isn't documented 1297342254000000 1297342599000000 2205 enhancement johnglover johnglover ckan-sprint-2012-03-19 closed fixed Document Tag Vocab schema/form setup 1330941067000000 1330956840000000 2520 defect seanh seanh ckan-v1.9 assigned Document undocumented config options "There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either: ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled There are also some options that are in the default deployment.ini even though they're deprecated: ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358" 1339588368000000 1340624908000000 928 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Documentation - CKAN credits / contributors page " 1. Remove contributors list from about page (out-of-date and does not belong there) * A ""Powered by CKAN"" link in footer is enough 2. Add contributors / team / credits page to ckan.org * May want to separate credits (e.g. software use), from contributors (also is team different from contributors) * (perhaps with photos!) so that people can see who is part of it and from which organisations. (this is medium-term, not really part of this ticket) 3. ? CREDITS.txt file to the source repository and add a link to it on bitbucket for more detailed credits info. Cost: 1h" 1295872661000000 1297115136000000 2771 enhancement seanh seanh ckan-v1.8 new Documentation and examples for IDatasetForm and IGroupForm "Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests. The IDatasetForm example should use tag vocabularies (two birds with one stone) The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples. Tab Vocabularies docs should reference IDatasetForm example. When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe) There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date." 1343392238000000 1350303564000000 644 task johnbywater ckan-v1.2 closed fixed Documentation for deleting harvest job entity 1284226402000000 1285260346000000 640 task johnbywater ckan-v1.2 closed fixed Documentation for getting harvest job entity from API 1284226036000000 1284491233000000 641 task johnbywater ckan-v1.2 closed fixed Documentation for getting harvest job errors from API 1284226049000000 1285348033000000 587 task johnbywater ckan-v1.2 closed fixed Documentation for getting harvest source create form from API 1284211364000000 1284492381000000 638 task johnbywater ckan-v1.2 closed fixed Documentation for getting harvest source entity from API 1284225805000000 1284491185000000 551 task johnbywater johnbywater ckan-v1.2 closed fixed Documentation for getting package create form from API 1283339845000000 1283354302000000 595 task johnbywater ckan-v1.2 closed fixed Documentation for getting remote metadata edit form from API 1284212572000000 1285198863000000 601 task johnbywater ckan-v1.2 closed fixed Documentation for listing remote metadata entities for a given publisher via API 1284213928000000 1284493045000000 639 task johnbywater ckan-v1.2 closed fixed Documentation for posting harvest job to API 1284226008000000 1284491239000000 590 task johnbywater ckan-v1.2 closed fixed Documentation for submitting harvest source create form submission to API 1284211659000000 1284492397000000 555 task johnbywater johnbywater ckan-v1.2 closed fixed Documentation for submitting package create form submission to API 1283340290000000 1283354313000000 2499 enhancement toby icmurray ckan-v1.8 new Documentation for the internal analytics "There's no docs for setting up the internal analytics, nor what it provides. == Setup == Add: `ckan.tracking_enabled = true` to .ini file Run: `paster tracking -c dev.ini` == Related links == http://trac.ckan.org/ticket/2251" 1339070297000000 1339070297000000 2363 enhancement toby kindly ckan-v1.8 new Documentation of best caching practice. Need better documentation on best practices in making page loads faster for non logged in users. 1335889017000000 1340099794000000 976 defect dread ckan-v1.3 closed fixed Documentation references knowledgeforge.net 1297346649000000 1297346954000000 1580 enhancement johnglover johnglover ckan-sprint-2012-01-09 closed fixed Documenting TaskStatus table and QA changes - 0.5d " * Write up: task status table in http://wiki.ckan.org/Domain_Model * Write up: http://wiki.ckan.org/Data_Quality to reflect current state of play (move all old material to ""Discussion / Talk"" page) * Ping ckan-discuss re new Data Quality page" 1324399664000000 1324478635000000 1430 defect amercader ckan-sprint-2011-11-07 closed fixed Documents get mixed between SOLR cores "On some occasions (apparently random), the documents indexed in a specific SOLR core get mixed with different site_ids. E.g: We look for all documents in the testing.iatiregistry.org core, faceted by site_id. We would expect all documents to have site_id = iati_testing, but some of them have site_id = iatiregistry.org http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=*:*&fq=+state:active&facet=true&facet.field=site_id {{{ 265 255 }}} If we compare one of the records which disappeared from the ""iati_testing"" site_id in both the production and testing SOLR cores of the server, the records are exactly the same, including the indexed_ts property: http://okfn-solr.fry-it.com:8080/solr/iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id Note that the response from the URLs shown may vary, as the testing site could have been reindexed." 1320068076000000 1324033923000000 645 task johnbywater johnbywater ckan-v1.2 closed fixed Domain object and migration script to associate packages and harvester objects 1284227152000000 1287774090000000 604 task johnbywater ckan-v1.2 closed fixed Domain object class for harvest job 1284217689000000 1284326615000000 582 task johnbywater ckan-v1.2 closed fixed Domain object class for harvest source 1284211104000000 1284261402000000 618 task johnbywater ckan-v1.2 closed fixed Domain object class for metadata document 1284219669000000 1284598039000000 2816 enhancement toby toby ckan-v1.8 closed fixed DomainObjectOperation not in model "Over-enthusiastic removal of some model items this breaks qa extension" 1344496621000000 1345040471000000 2487 enhancement johnglover johnglover ckan-sprint-2012-06-25 closed fixed Don't preview text/html Quite a few of the current test dataset resources have their format incorrectly specified as text/html. Attempting to preview these results in the browser trying to automatically download the resource, so for now we will not preview this format. 1338806098000000 1338892521000000 2864 defect toby shevski demo phase 4 closed fixed don't show `create one now` message on other's user pages e.g. I can see 'You haven't created any datasets. Create one now?. ' http://s031.okserver.org:2375/user/kindly when logged in as shevski 1345114552000000 1345471138000000 1393 enhancement johnglover dread ckan-backlog assigned Don't skip search tests Now we don't use postgres search, all the tests involving search now don't need to be skipped when running on sqlite. Should help coders spot earlier if these tests break. 1318505453000000 1320153590000000 1638 enhancement seanh seanh ckan-sprint-2012-01-23 closed wontfix Don't use JsonType in activity streams Dump and load JSON explicitly instead. 1326304935000000 1326305570000000 516 defect dread ckan-v1.2 closed fixed Double-escaped text visible - revision page "At: http://ckan.net/revision you can see: ""Next »"" on pager This doesn't seem to be a problem with earlier (1.0.1) version." 1282829539000000 1288002933000000 88 enhancement rgrp rgrp v0.11 closed fixed Download (link) metadata "Support for download metadata (file size, file formats, version information). See also: ticket:87 (multiple download links)" 1248693385000000 1265891491000000 337 enhancement johnbywater rgrp v1.1 closed fixed Download links for resources should open in new window "target = _blank Cost: 30m" 1275422151000000 1279300972000000 406 requirement pudo rgrp datapkg-0.8 closed fixed Download sub-system "Factor out current downloader into dedicated download subpackage, abstract interface and make it pluggable so we can have downloaders loaded as per type of resource to download. * Also rename InstallCommand -> DownloadCommand And we should support interactive retrieving of resources ... == Implementation Ideas == {{{ for resources in pkg.resources: # datapkg/hg load_storage_handler(resource.format) }}} " 1281346822000000 1296467427000000 1183 defect johnglover nickstenning ckan-sprint-2011-10-28 closed fixed "Downloads ""Preview"" button doesn't preview." "The ""Preview"" button is a nice idea, but it doesn't seem to actually ""preview"" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file. This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of `application/x-some-junk-here` in order to force a download." 1307701669000000 1311771069000000 2828 enhancement toby shevski demo phase 4 new Draft datasets are confusing - tickets need creating "reported as editing datasets incorrect e.g. clicking on edit here http://s031.okserver.org:2375/dataset/ff takes you to create dataset page http://s031.okserver.org:2375/dataset/edit/ff but this is the correct behaviour of a draft dataset We need to show draft datasets correctly proper tickets need making for the different issues after review of issues with them - who can see, where, admins and viewing, orgs too etc" 1344506178000000 1344547324000000 2432 enhancement seanh seanh ckan-sprint-2012-06-25 closed fixed Draft new trac methodology "We want a new way of working with trac tickets and milestones etc: - We will have a single queue of tickets, ordered by priority, that we all work on - Tickets only get into the queue after triaging (which the devs do at regular intervals), so we and others can use new trac tickets as a dumping ground for any and all CKAN bugs and ideas without messing up the priority queue. Therefore the priority queue needs to be a special CKAN milestone or something. The devs need to regularly go through the inbox of untriaged tickets and triage them all, moving them all out of the inbox. - Will we also assign tickets from the priority queue to individual devs in sprint planning meetings like we currently do? " 1337947327000000 1338325120000000 675 task johnbywater johnbywater ckan-v1.3 closed fixed Draft specification of CSW harvesting behaviour with guide for publishing agents 1286214877000000 1288038316000000 2792 enhancement aron.carroll toby demo phase 3 closed fixed dropdown fixes - open correctly "http://s031.okserver.org:2375/ language dropdown expands downwards it should open upwards select2 can do this see http://ivaynberg.github.com/select2/#documentation can you see if we can get this in ckan demo " 1343895296000000 1343901501000000 3015 enhancement johnmartin johnmartin ckan 2.0 closed invalid Dropdowns don't work without JavaScript Essentially the default bootstrap dropdowns don't work properly without JavaScript which is a little bit of an issue with JS not working. 1352805553000000 1353410072000000 1255 enhancement kindly kindly ckan-backlog assigned Drupal consistancy checks. Make a robust way to make sure the drupal database is consistent with the ckan data. 1312219968000000 1313400054000000 457 story dread dread ckan-v1.2 closed fixed Dump database on a daily basis 1282299921000000 1282909051000000 1623 enhancement dread dread ckan-sprint-2012-01-23 closed fixed Dump to exclude deleted objects "The database dump currently contains all Packages and their associated objects, even those that have been set to state=deleted. We should exclude these from the dump now. Dump = paster db simple-dump-csv/json == reasoning == The dumps are designed for end-users to be able to run scripts on the mass of data. Since end-users don't see state=deleted packages then they shouldn't need them in the dump. In fact their presence in the dump probably confuses them. Admins get the full database anyway in the backup pg_dump. We only included them in the user dump because it was designed before use of state=deleted was established. Time estimate: 2h" 1326118987000000 1326892264000000 663 defect kindly dread ckan-v1.4 closed fixed Duplicate sibling relationships displayed "Looking at: http://ckan.net/package/rkb-explorer-acm 'rkb-explorer-resex' gets display multiple times and shouldn't" 1285576369000000 1298913603000000 2239 enhancement ross ross ckan-sprint-2012-04-30 closed fixed EC ODP RDF/N3 templates "Implement the EC/ODP template for RDF to match the requested schemas. Using the previous work on allowing for templatised RDF representations we should implement the proposed schema for EC/ODP. We also agreed to allow upload of RDF as an extra (raised to top level in package schema) which will instead be used for the RDF template. On insertion extra CKAN triples will be added... initially this is just owl:sameAs although we're awaiting EC/ODP agreement on which triples we should add to show the CKAN link. " 1332169703000000 1335516499000000 2269 enhancement johnglover johnglover ckan-sprint-2012-04-30 closed fixed EC ODP review and server setup " * code review with D.R. for ckanext-ecportal * merge and test all multilingual code * fix server issues: routing, Git access to CKAN repository Bugs to fix: * ~~rename groups to publishers~~ * ~~update home page layout (recent updates along side publishers)~~ * ~~don't add 'en' to main menu links if english~~ * ~~resource edit page, fix missing images~~ * ~~change dataset 'settings' to 'edit'~~ * ~~translate 'published by' drop-down list~~ * ~~fix JS on 'add a dataset page'~~ * ~~prepopulate 'published_by' when clicking 'add a dataset' from group~~ * ~~chosen broken (dataset edit) if in different language~~ " 1333374177000000 1335779180000000 2346 enhancement johnglover johnglover ckan-sprint-2012-05-15 closed fixed EC ODP Theme updates "Changes: * tidy up activity feed on home page or replace with search results (by modified date) * ~~move ckan menu to a left sidebar (see Sam's template)~~ (not doing this now) * update templates with changes from master * use local image in language dropdown (ecportal.js) * update dataset read template To discuss: * should we rename 'organization' to 'publisher'? Requires updating all templates and the url mapping." 1335876838000000 1338205553000000 2586 enhancement toby toby demo phase 1 closed fixed edit dataset - tags broken tags is list not string 1340361630000000 1340380561000000 2940 enhancement johnmartin johnmartin ckan 2.0 closed fixed Edit different resources from edit dataset pag We need a nice way to be able to get to editing the different resources that are associated to a dataset. 1348580812000000 1350560926000000 125 enhancement dread rgrp v1.0 closed duplicate Edit Generic Package Attributes in WUI Split out from ticket:43 1253709712000000 1258377621000000 36 enhancement rgrp rgrp v0.4 closed fixed Edit package page should support previews When editing a package one would like to be able to preview one's changes before saving. 1177007112000000 1185470035000000 2584 enhancement aron.carroll toby demo phase 1 closed fixed edit profile needs new theme " http://localhost:5000/user/edit/" 1340359925000000 1340620576000000 979 enhancement kindly dread ckan-backlog assigned Edit Resource extras in the API Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API. 1297429777000000 1315222244000000 978 enhancement zephod dread ckan-sprint-2012-03-05 closed fixed Edit Resource including Extras in Web UI "Part of super-ticket #1506. Follows on from #826. We can configure some extra fields in resources and can edit them in the Web UI, but we can't create new columns in the Web UI. Update the WUI to handle tis. " 1297429619000000 1330547181000000 2667 enhancement shevski toby demo phase 2 closed fixed edit resource needs form "I thought I'd made this ticket before but can't find it. When we have a fully created dataset we need to be able to edit the resources. template: package/resource_edit.html when you've made a form reassign to me and I'll get it working " 1342170103000000 1343058578000000 1423 enhancement markbrough ckan-backlog new Edit resources suggestions " * Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: http://thedatahub.org/dataset/edit/iati-registry * Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in." 1319641906000000 1338203678000000 1215 defect dread closed fixed Edit Resources table - removing all rows prevents adding them again "I'm not sure if this was reported earlier, but I have found a minor bug in the ""Add new package"" form. As far as I can see this bug exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). This is a problem with adding new resources in ""Resources"" section. To reproduce the bug do the following: 1. Go to ""Add new package"", for instance http://ckan.net/package/new 2. Scroll down to ""Resources"" section. 3. Click on ""Remove this row"" link (red circle). 4. Say ""Yes"" to ""Are you sure you wish to remove this row?"" 5. Now there should be no rows. Only the header ""URL* Format Description"" 6. Click on ""Add row to table"" 7. All you get now is another row with table header ""URL* Format Description"". You will not be able to add any resources now. Solution is to reload the page. The simplest solution to this bug will be to hide the ""Remove this row"" link if there is only one row." 1310133224000000 1310385485000000 2845 defect seanh seanh ckan-v1.8 closed fixed Editing an organization removes all datasets from the organization Run CKAN with the publisher profile and organizations plugins enabled. Make an organization. Make a dataset and add it to the organization. Edit the organization and change e.g. its description. After saving the organization, all its datasets have been removed from the organization and now belong to no organization. 1344859286000000 1347271110000000 1015 defect kindly dread ckan-v1.4-sprint-3 closed fixed Editing group gives exception "Editing a group properties on ckan.net gives an exception when you submit the form. (Is this related to db migration issues?) Example page: http://ckan.net/group/edit/civil-society Exception: {{{ WebApp Error: : (IntegrityError) duplicate key value violates unique constraint ""group_revision_pkey"" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'description': u""A group for open data related to civil society supported by members of the interest group at: http://okfn.org/groups/civil-society\r\n\r\nCivil society is composed of the totality of voluntary civic and social organizations and institutions that form the basis of a functioning society, as distinct from the force-backed structures of a state (regardless of that state's political system) and commercial institutions of the market. \r\n\r\nhttp://en.wikipedia.org/wiki/Civil_society"", 'created': datetime.datetime(2011, 2, 24, 14, 30, 53, 334842), 'title': u'Civil Society', 'state': u'active', 'continuity_id': u'f4f9f09 }}}" 1298898588000000 1299788821000000 1005 defect dread dread closed duplicate Editing in ckan missing publishers Package edit form at hmg.ckan.net has no publishers any more. 1298561348000000 1300100085000000 1340 defect dread dread ckan-sprint-2011-10-28 closed fixed Editing non-existent group causes Exception "e.g. http://ckan.net/group/edit/history {{{ Module ckan.controllers.group:135 in edit << try: old_data = get_action('group_show')(context, data_dict) c.grouptitle = old_data.get('title') c.groupname = old_data.get('name') >> old_data = get_action('group_show')(context, data_dict) Module ckan.logic.action.get:347 in group_show << if group is None: raise NotFound check_access('group_show',context, data_dict) >> raise NotFound }}}" 1315991374000000 1315991534000000 1595 defect kindly rgrp ckan-v1.7 closed wontfix Editing of a Group leads to recording a change against all associated datasets "Or at least it appears that way from revision log which displays all Group datasets against the changelog, see http://thedatahub.org/revision/4cdeeb42-3281-4f53-a29d-c694e1fd9217 -- not apparent from that page but see my user page: http://thedatahub.org/user/rufuspollock This may turn out to be a UX bug rather than a real bug ... Assigning to kindly for review and comment." 1324504875000000 1338203959000000 2675 defect aron.carroll shevski demo phase 2 closed fixed editing related item uses 'create' button instead of save "see http://s031.okserver.org:2375/dataset/gold-prices/related/edit/46ba1174-3666-486e-b822-b5b45ffbb703 when editing a record, button at bottom right should change to save instead of create" 1342435576000000 1342461534000000 1635 enhancement seanh seanh ckan-backlog new Email notifications (e.g. for activity streams) "CKAN should be able to send email notifications to users. Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users. One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well. Need to implement (at least some of) #1634 before this can be implemented, in order to have something to send notifications about. Analysis here: http://ckan.okfnpad.org/27" 1326304587000000 1355141157000000 2894 defect seanh ckan-v1.8.1 new Empty lists get stripped from package dicst in package_show() "If there's an IDatasetForm plugin active with a form_to_db_schema(), then packages with e.g. no extras get the 'extras' key entirely stripped by the validation in package_show, which causes some tests to crash, e.g. in ckan/tests/functional/api/test_activity.py (if the tests are run with a suitable IDatasetForm plugin active). If a package has no extras the dict returned by package_show should still have an 'extras' key with an empty list as the value. Suspect this also effects packages with no tags, no groups, etc." 1346157634000000 1346157634000000 920 defect thejimmyg hellmann@… closed fixed empty tags are kind of confusing "http://ckan.net/tag/linguistsic doesn't have a single entry, but is still kept and displayed. It should be purged !!!" 1295355280000000 1300319140000000 1775 enhancement seanh seanh ckan-sprint-2012-03-05 closed fixed Enable adding many tags at once to controlled vocabularies via API "We already have API calls for adding one tag or removing one tag at a time to/from a vocab. Want more convenient ways to add or remove many in one call, or just pass in a list of tags to completely replace a vocab's current list. - Add tags_create() function that calls the existing tag_create() many times? - Also tags_delete()? - Or just let tag_create() and tag_delete() accept a list? - Add tags argument to vocabulary_update() to pass in a list of tags to replace the vocab's current tags? (But it should still be possible to simply rename a vocab without changing its tags and without having to pass in the tag list.)" 1328541056000000 1329754459000000 648 requirement pudo ckan-v1.3 closed fixed Enable lock-down of package creation " - copy exisiting tests, modify authz in setUp, adapt and extend tests * Problem: default_role_actions is read by init_db * Solution: nuke db after monkey-patching * role = model.Role('Reader'), del role.actions[...] self.PRE_MAUTHZ_RULES = copy(mauthz.default_role_actions) mauthz.default_role_actions.remove((Role.READER, Action.CREATE)) #raise Exception(mauthz.default_role_actions) model.Session.remove() model.repo.rebuild_db() 3. Start from the functional * Both the api controllers and wui controllers to do * tests/function/test_authz.py extended to check CREATE in lockedDown mode. fails nicely. http://bitbucket.org/pudo/ckan-authz2 is_authorized(user, Action.Create, model.Package) -> Doing this will put 'Package' in the context field of the user_object_role table. This will trigger SQLAlchemy to attempt a join towards PackageRole in all queries. Since for class-level role assignments there never is a PackageRole join table entry, this will never return any results. * cf branch ""classes"": http://bitbucket.org/pudo/ckan-authz2/src/6fd0475e0c66 * No possibility of including ""instance"" column in inheritance decision. * Even if this can be cirumvented, it must happen on a per-query level and would require a major re-modelling. * have a ckan install that would not allow visitors to either list packages or list groups * two paths: create the listing, but for each group/pkg decide that you cannot show this * lock down the whole page (/package/list) - this is class-based, not object-based is_authorized(user, Action.Package_Create, model.System()) is_authorized(user, Action.Group_Create, model.System()) [Separating Package and Group roles may be useful going forward: PackageEditor, GroupEditor etc] 4. Find a standard way to lock down classes * possibly add default rows in user_object_role * introduce lock-down into controllers: group new, package new, REST equivalents, * confirm tests" 1284889519000000 1284889656000000 2686 defect shevski ckan-backlog assigned enabling datastore & data API breaks recline "First I noticed that the gold prices dataset preview was not displaying & has data API enabled Secondly I tried enabling datastore for http://datahub.io/dataset/adur_district_spending/resource/281dffa6-ea9b-4446-be41-05dced06591f and after I saved the preview no longer worked. Unticking the datastore & data api checkbox brought it back Is this a known issue?" 1342516011000000 1346663300000000 2485 defect icmurray nils.toedtmann ckan-v1.9 new Encourage leak containment by limiting the number of requests a CKAN process serves "CKAN has [ticket:1345 memory leaks]. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced. One should ... * research ways to limit requests-per-process with the different web servers (see below for a start); * explain these safeguards in the CKAN documentation and encourage users to apply them; * consider the helper script '''[https://github.com/okfn/ckan/blob/master/ckan_deb/usr/bin/ckan-create-instance ckan-create-instance]''' to generate Apache configs with '''WSGIDaemonProcess ... maximum-requests=XY''' being active instead of [https://github.com/okfn/ckan/blob/master/ckan_deb/usr/lib/ckan/common.sh#L262 commented out]. Or at least let it warn the user to use '''!MaxRequestsPerChild'''; * Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them. ---- How to limit requests-per-process * Apache: * Use '''WSGIDaemonProcess''' with '''maximum-requests=50''' or whatever limit is appropriate. We did this sucessfully on [http://trac.okfn.org/ticket/904 datahub.io] and the [http://trac.okfn.org/ticket/1245 ckan farm] * (This need to be verified) Without WSGIDaemonProcess, '''!MaxRequestsPerChild 50''' should achieve the same. * nginx/supervisord: to be researched. " 1338551229000000 1340636028000000 784 task thejimmyg johnbywater ckan-v1.4 closed wontfix End to end testing for UKLP 1289210251000000 1304936251000000 1262 enhancement pudo ckan-backlog new "Enforce ""create-user"" permission" This does not seem to have any implications at the moment, it should lock down registration and remove all related links. 1312375296000000 1323090112000000 2820 defect danieljohnlewis demo phase 5 new English Language: Visualization -> Visualisation "Problem: In the English version (which has a UK flag, indicating British English), the word ""Visualization"" is used. For an example see the ""Filter by type"" drop down on the /apps page. Expected: This should be ""Visualisation"" in British English. Any instances of ""Visualize"" should be changed to ""Visualise"" too. " 1344504455000000 1344504455000000 1765 enhancement seanh seanh ckan-sprint-2012-02-20 closed fixed Enhance Tag and Package models with vocabularies Add a vocabulary column to the tags database table, change tags to have unique (tag_name, vocabulary_id) instrad of unique tag name, update methods in the Tag and Package classes to deal with the fact that tags may belong to vocabularies and that tag names are no longer unique. 1328537061000000 1329131124000000 119 enhancement dread dread v0.10 closed fixed Ensure non-active packages don't show up "Ensure pending packages don't show up in search or browse cost: 4h" 1253529414000000 1253791147000000 279 enhancement johnbywater dread v1.0 closed fixed Ensure package's license is an option when it's edited "With forms configurable to show a subset of licenses and ckan instances moving packages from server to server, there is a chance that a user will edit a package and the existing package's license is not displayed in the form. If the user submits this form without touching the license dropdown, then the value is lost. So for this case we should add the package's license to the list displayed." 1270205931000000 1272451384000000 2787 enhancement toby aron.carroll demo phase 5 closed fixed Ensure templates_legacy directory is loaded for extensions Currently the stats plugin has a template and templates_legacy directory. We need to ensure that the templates in templates_legacy are loaded. 1343827782000000 1344543473000000 139 defect rgrp rgrp v0.11 closed worksforme Ensure you cannot add same tag twice to a package "Seems some package have ended up with same tag twice: http://www.ckan.net/package/fred2dot0 Check whether this is still possible and if it is fix it ... Cost: 2h" 1254768552000000 1255188974000000 1805 enhancement toby closed fixed Error pages do not translate "eg 404 page create a 404 error via a bad url - translation links go to /document/error not the actual bad url" 1329489216000000 1329760795000000 1022 enhancement pudo pudo ckan-v1.4-sprint-3 closed fixed Error reporting in CKAN worker API Report on errors by any worker daemons, send them out via E-Mail 1299493047000000 1299512991000000 1285 enhancement dread ckan-future assigned Errors cause emails "Currently a sysadmin gets an email when an exception is not caught. But there are occasions when we DO want to catch an exception so we can fail nicely for the user, but the sysadmin STILL gets an email to know to fix something. e.g. if there is an exception when search indexes a package. You want to catch the exception and still run any other notify calls. " 1314116944000000 1338206151000000 2406 defect kindly markw ckan-v1.9 new Errors in source install doc "There are a few errors or unclarities in the source install doc - see http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html " 1337595093000000 1340636804000000 2585 enhancement seanh ckan-v1.9 new Escape solr control characters in search queries, add advanced search screen "Suggestion from David Read: We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for ""Spend over £25,000 - NHS Leeds"" then it would not come up with the dataset with that exact name. It was excluding datasets with the word ""NHS"" due to the dash/minus sign. It works fine if you escape the minus sign: ""Spend over £25,000 \- NHS Leeds"". So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine: http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/ Perhaps you would consider providing this in CKAN core in future? I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?" 1340360773000000 1340625078000000 823 defect dread dread closed fixed Etags cache doesn't update when only non-core package fields changed. "1. Create a package. 2. Edit package and add a resource. 3. View package in browser - no sign of the resource - ERROR. 4. View package in API - resource is there. Reason: hash doesn't change because package.revision only takes account of core fields." 1290505078000000 1290506116000000 1388 defect dread ckan-backlog closed fixed etags caching on home page "Needs to update on: 1. language - It's broken on thedatahub.org if you change to German and then click on the CKAN icon top-left to go to thedatahub.org again. AND 2. whether signed in or not - #1373 BUT not latest revisions (which is what it was) Or get rid of etag caching on this page and others? " 1318429443000000 1320174229000000 1008 defect rgrp pudo ckan-v1.4-sprint-2 closed fixed eval() of user display name in template head "We're currently setting the user preferences links via a javascript snippet that also evals the name. This should be removed as we're not displaying the user name any longer. We should also have page fragment caching in Genshi, which is not currently implmented. cf. http://api.rubyonrails.org/classes/ActionController/Caching/Fragments.html" 1298632686000000 1298821826000000 325 enhancement dread dread v1.1 closed fixed Event push notification "=== As a === CKAN client program === I want to === be notified when changes to the CKAN metadata occur. === Examples of use === 1. An external search engine needing to (re)index a package. (interest: Package) 2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333. 3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: !PackageResource) 4. Do some processing on resource (e.g. extract sample data for display) (interest: !PackageResource) === Context === The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling. === Design === Split-off into two tickets: * Notification message - ticket:323 * Which events to notify on * Message format * Interface for Notifier Service - ticket:322 === Testing === To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine. === Related === * Run CKAN search indexing using this system - ticket:324 * Run SOLR indexing using this system - ticket:353 * Web hooks for notifications - ticket:327 " 1274723512000000 1278599979000000 688 task johnbywater johnbywater ckan-v1.2 closed Example GeoNetworks service for CSW development 1286786893000000 1286980827000000 178 enhancement rgrp dread v0.11 closed fixed Excel package importer "/tool New 'tools' section of the WUI with 'Excel package import' page. /tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste. /tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process. File format: First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes: * license - name of license * tags - space separated * groups - space separated * anything - for extra key-values When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete. User has to log in to use importer. Auth is checked for putting into groups." 1257244034000000 1265305558000000 820 defect dread closed fixed Exception browsing /user/apikey when not logged in "Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in. {{{ WebApp Error: : 'NoneType' object has no attribute 'apikey' URL: http://test-hmg.ckan.net/user/apikey }}}" 1290006079000000 1311325226000000 1347 defect dread dread ckan-sprint-2011-10-28 closed fixed Exception caused by New Group form POST with bad formed parameter "Exception caused by bots POSTing 'new group' form, but with bad (old-style) parameters. {{{ : invalid literal for int() with base 10: 'cache' ... Module ckan.controllers.group:102 in new << if context['save'] and not data: return self._save_new(context) data = data or {} >> return self._save_new(context) Module ckan.controllers.group:152 in _save_new << try: data_dict = clean_dict(unflatten( tuplize_dict(parse_params(request.params)))) context['message'] = data_dict.get('log_message', '') group = create.group_create(data_dict, context) >> tuplize_dict(parse_params(request.params)))) Module ckan.logic:57 in tuplize_dict << for num, key in enumerate(key_list): if num % 2 == 1: key_list[num] = int(key) tuplized_dict[tuple(key_list)] = value return tuplized_dict >> key_list[num] = int(key) ValueError: invalid literal for int() with base 10: 'cache' }}}" 1316094466000000 1316094608000000 500 defect dread ckan-v1.2 closed duplicate Exception from diff "Investigate exception occured occasionally in last couple of days on ckan.net: {{{ WebApp Error: : 'NoneType' object has no attribute 'key' URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:73 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.revision:119 in diff << c.revision_to = model.Session.query(model.Revision).get( request.params.getone('diff')) diff = pkg.diff(c.revision_to, c.revision_from) c.diff = diff.items() c.diff.sort() >> diff = pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:340 in diff << display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] >> display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'zh-cn,zh-tw' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Baiduspider+(+http://www.baidu.com/search/spider.htm)' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' REMOTE_ADDR '123.125.66.32' REMOTE_PORT '63767' REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '
    Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (, '') paste.parsed_querystring ([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'} repoze.who.logger repoze.who.plugins {'openid': , 'auth_tkt': } routes.route routes.url webob._parsed_query_vars (GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgiorg.routing_args (, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}) }}}" 1282553033000000 1287747652000000 1338 defect amercader amercader ckan-sprint-2011-10-28 closed fixed Exception when trying to use a custom form schema from an extension "Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators. {{{ File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit old_data, errors = validate(old_data, schema) File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate converted_data, errors = _validate(flattened, schema, context) File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate convert(converter, key, converted_data, errors, context) File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert converter(key, converted_data, errors, context) File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator model = context['model'] KeyError: 'model' }}}" 1315925217000000 1317117738000000 1126 enhancement dread dread ckan-v1.4-sprint-7 closed fixed Exceptions arising from error page "I'm not completely clear what the use case is for loading the error page in this way, but somehow original_request is None and that creates an unnecessary exception with the logic refactor. {{{ http://ckan.net/error/document?__cache=39020485 ... Module ckan.controllers.error:29 in document << original_response = request.environ.get('pylons.original_response') # Bypass error template for API operations. if original_request.path.startswith('/api'): return original_response.body # Otherwise, decorate original response with error template. >> if original_request.path.startswith('/api'): AttributeError: 'NoneType' object has no attribute 'path' }}}" 1304586683000000 1304587078000000 1249 enhancement pudo rgrp ckan-v1.5 closed fixed Exclude script tag from extraction for i18n "Currently have this script section put in for i18n. It shouldn't be. {{{