{22} Trac tickets (2647 matches)

Results (701 - 800 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#876 enhancement sebbacon rgrp ckan-v1.3-sprint-2 closed fixed Support sqlite as a database backend for CKAN

Among other things this will allow the tests to run much quicker.

1292858711000000 1294753889000000
#466 requirement dread johnbywater ckan-v1.3 closed wontfix Name of HTTP header for API key shall be configurable 1282308995000000 1294835610000000
#870 defect memespring ckan-v1.3-sprint-1 closed fixed Menu items not selecting correctly

Since the main menu items have changed (part of the UI redesign) the selected items are not functioning correctly.

Specifically if Home / About or Add Package / Search are selected both menu items are highlighted.

This is because the code assumes only one top-level menu item per controller.

1292243583000000 1294914243000000
#897 defect dread dread closed duplicate "New package" highlighted in Navigator when searching

On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted.

1294648815000000 1294914333000000
#219 enhancement nickstenning dread closed wontfix UI Review - Package
  • Propose three-column package listing: two lists of highest-scorers on some metric and last column is an invitation to register a new package, with a little copy explaining what's involved and why you'd want to.
  • Add in search widget
1260879947000000 1294914901000000
#463 defect dread ckan-v1.3 closed worksforme package diff exception

Investigate exception:

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
AUTH_TYPE	'cookie'
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'en-US,en;q=0.8'
HTTP_CACHE_CONTROL	'max-age=0'
HTTP_COOKIE	'__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173'
HTTP_HOST	'ckan.net'
HTTP_REFERER	'http://ckan.net/package/history/dbtune-audioscrobbler'
HTTP_USER_AGENT	'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'92.226.62.168'
REMOTE_PORT	'55102'
REMOTE_USER	'http://pudo.org/'
REMOTE_USER_TOKENS	['']
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0xbb528cc>
beaker.cache	<beaker.cache.CacheManager object at 0xbb528ec>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0xbb5286c>>
beaker.session	{'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: __utma='27730403.1381387877.1281962773.1282248244.1282250173.11' __utmb='27730403.6.10.1282250173' __utmc='27730403' __utmz='27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan' auth_tkt='e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!'>, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0xf702aec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xf6eaf0c>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xf6eaf0c>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0xf6eaa8c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.identity	<repoze.who identity (hidden, dict-like) at 280815340>
repoze.who.logger	<logging.Logger instance at 0xbcdfccc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 196612236>, 'auth_tkt': <AuthTktCookiePlugin 198048140>}
routes.route	<routes.route.Route object at 0xbbc44cc>
routes.url	<routes.util.URLGenerator object at 0xf5e55ec>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0xfe0b0b0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xf5e55ec>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})
1282306037000000 1294916148000000
#886 defect wwaites wwaites ckan-v1.3 closed duplicate move uklii forms api to dgu module

doesn't belong in ckan core

1293278145000000 1294916538000000
#900 defect dread closed fixed Intermittent error with new revision not having id

There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision. We add the uuid earlier to fix this. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e.

From David Raznick

1294659696000000 1294916587000000
#899 enhancement rgrp dread closed fixed Optimise check_real_change

There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e

From David Raznick

1294659583000000 1294916615000000
#474 task dread dread closed wontfix Questions related to fi.ckan.net

Several questions to answer (and see file in email). To respond to the Finnish team and copy all Q&A to the ckan-discuss list.

1 ) I have played with two different datasets to get the metadata from those imported to CKAN. The other dataset is in PCAXIS format and other one is in Excel format. Please find the scripts + sample files attached in a zip file that contains instructions too to run the scripts. I looked at import scripts written in object oriented manner and getdata samples in CKAN but it was easier for me too write these from scratch (copied the unicrap function from CKAN samples). If I run the PCAXIS import script I get Unicode decode error (see errors file). When I use the unicrap function then the error does not appear in logs. I assume this is some Python environment setting issue. Of course I would like to run the script without unicrap function to get scandic letter imported correctly to ckan(and displayed in WUI). In Excel case there is no scandic letter issue which is kind of strange but this probably relates how the Excel library handles encoding.

To be honest the whole unicode/encoding/decoding issue has been quite difficult to deal with and debug. I hope I would get proper environment configuration to handle scandic letters.

2 ) It seem in CKAN by default the localisation file contains 462 strings. The transifex link Rufus sent contains 400 strings. Is transifex the one we should use as localisation file ? Where are the rest 62 strings then fetched from or are they needed at all. I have found that as I have been using the .po file coming in CKAN source (which has API translations too) this also causes similar unicode problems when an error in API occurs (the API error messages containing scandic letters provide the similar UnicodeDecoding? errors as mentioned above). So, it seems the unicode issues appear in various cases. Maybe a proper environment settings will be able to prevent also these problems... If transifex is the one to use I can prepare it and maybe then you could upload it to fi.ckan.net as there will be also other people checking the translations. To run the test and see the API translation errors especially in pcaxis case I would recommend to install attached FI translations to your environment and include lang = fi configuration. It looks like the API errors does not come when running default lang = en.

3 ) Is it possible to change the license selection/options available in license field drop-down ? Where are these located and can these be modified locally ?

4) It looks like the values stored in package extra fields are not searchable. Is this intentional ? It looks more or less that all other fields can be searched using free-text search in CKAN.

5) How package dependencies are defined and managed ? For instance we want to import some packages in a manner that the other package is a child package of another. Does API support this and/or is this possible to do in WUI ?

Best Regards, Henri

1282312255000000 1294916760000000
#798 task dread dread ckan-v1.3 closed fixed cospread importer using API

Move existing getdata/cospread.py into dgu repo and use the importer/loader mechanism.

1289475362000000 1294916973000000
#512 story dread closed duplicate User creates package via API with incorrect core fields specified 1282754750000000 1294917121000000
#541 enhancement wwaites rgrp closed fixed Full-page caching even for logged in users

The simplest way to improve performance on specific pages is to implement full page caching.

This, however, has challenges on pages with user-specific content. For example, on CKAN when a user is logged at the top of the page we have "You are logged in as: {username}". Similarly we change the visibility of some material depending on the status of users:

  • Visibility of Edit or Authz link on packages and groups
1283243139000000 1294924872000000
#838 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed UI improvements December 2010

This is a meta ticket for a bunch of UI changes:

  • #839 Package page redesign
  • #846 Merge css files
  • #850 Show welcome message
  • #863 Prompt users to enter missing info
  • #864 Search results changes
  • #866 Add download formats to search results
1291296811000000 1295259773000000
#865 defect pudo pudo ckan-v1.3-sprint-1 closed fixed Support external strings in JS
  • Provide and include a "language.js" file.
1291801938000000 1295259773000000
#847 enhancement pudo rgrp ckan-v1.3-sprint-2 closed fixed CKAN search using SOLR backend

This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010).

3 key aspects of this:

  • Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN)
    • Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable
    • Asynchroously: via worker with queue
      • #324 Search indexing using notifications
      • #669: Refactor Solr (indexer) to become a generic worker
      • #874: Extract Solr search backend into an extension
  • In core support using SOLR backend as well as postgres for queries
    • #317: Make search pluggable (?)
  • Integrate into WUI (if changes needed)
    • #672 - Facets in CKAN search results

Extras:

  • Solr index testing tool #431
1291639273000000 1295259902000000
#775 task pudo dread closed fixed Search warning

We're getting this warning a great deal on live servers. Is these really a sign of the system not operating correctly or can we reduce the level to an INFO?

e.g. on hmg.ckan.net:

2010-10-29 17:12:08,262 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,333 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,375 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,406 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,480 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,613 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
1288372692000000 1295260144000000
#431 task pudo pudo closed wontfix Solr index testing tool

There seem to be a few conditions under which either queue processing or the indexer fail in their current state. To get a more systematic picture of these failures, we should have a small testing tool to compare the index to the database of a live CKAN instance.

1281963532000000 1295260262000000
#917 enhancement kindly kindly closed fixed Turning off Autoflush in SQLAlchemy

David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior.

I have a patch in https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default.

This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet).

Things to know about using flush:

  • Most of the time nothing will change. A commit implies flush.
  • You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query.
  • A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated.
  • You can use them as a limited form of nested transaction.

It increased the performance on the tests by about 5%. There are less flushes altogether so is faster.

I do not see any downsides.

1295279694000000 1295280232000000
#691 requirement thejimmyg johnbywater ckan-backlog closed duplicate Package Relationships 1286822735000000 1295610145000000
#676 requirement thejimmyg johnbywater ckan-v1.3 closed wontfix The system shall support browsing to correct version of the CKAN documentation for a given service 1286276136000000 1295869107000000
#832 enhancement rgrp rgrp ckan-v1.3 closed fixed Convert stats to a plugin

Convert stats section to a ckan plugin.

Stats need not be in core and making it a plugin would make it easier to extend and expand.

1290765915000000 1296334980000000
#881 defect thejimmyg fccoelho closed invalid http authentication requirement when installing

I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code.

1293021889000000 1296335072000000
#181 enhancement rgrp jwyg closed invalid New 'stats' box on right hand sidebar

With up to date statistics such as:

  • total packages
  • total tags
  • total users
  • most active users
  • last update
  • etc.

Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See ticket:184

1257534385000000 1296339510000000
#844 defect wwaites wwaites closed fixed bad mail config ckan.net/ckan.org

ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example:

B30E6D9F       2873 Sat Dec  4 14:05:40  MAILER-DAEMON
(delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out)
                                         [email protected]

both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null

1291556753000000 1296340486000000
#871 defect nils.toedtmann closed invalid Check whether localhost-only exim installtions need upgrading too

The infamous exim bug only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers.

We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not.

BTW did we already run a rootkit checker like Rootkit hunter on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark? has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could.

1292264117000000 1296340558000000
#186 enhancement rgrp rgrp closed duplicate Automated upload to archive.org s3

(Follows on from ticket:107). We want to provide facility for users to automatically upload material.

1257803430000000 1296341182000000
#75 enhancement dread rgrp closed duplicate Record and display package "usage" information
  • Number of package page visits on ckan (can we get this straight from google analytics)
  • Number of times url or download url is used - now ticket:937 (Record download stats for resoures)

How do we do this?

  • Google analytics will miss a lot of this usage (and how do we get that data out anyway)
  • Could use javascript but again misses usage.
  • One option is to redirect link but that is kind of nasty (but may be only option ...)
1247828785000000 1296341223000000
#107 enhancement rgrp rgrp closed invalid Provide information about uploading material

Often people who are registering resources want to make the associated material available. We should:

  • Provide a dedicated upload page giving instructions as to upload process
    1. Explain we don't store the data itself on CKAN. Suggest uploading to a store somewhere and then linking using download_url (that is its purpose)
    2. Upload to
      • Talis CC for RDF
      • archive.org
      • grid.okfn.org
  • link to this page from next to download_url item on new package page

In long run (now ticket:186) we may automate this by providing a ckan upload facility which caches the data and then reuploads it to relevant service (disadvantage is "we" are the owner of the data on that service ...)

1251466274000000 1296341644000000
#327 defect pudo pudo closed wontfix Create a web hook worker for CKAN

This will be useful as an example consumer of the new queue notifications (#325).

  • A user can register any URL to be notified upon an event.
  • Possible extensions: GET with ID only vs. POST with serialized object
  • E-Mail notifications are also hooks, essentially.

We need to have a UI module that allows for the hook CRUD and some feedback (e.g. non 200 status codes)

1274807361000000 1296467361000000
#911 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Add pager to dcat-tools/rdfsolr 1295266211000000 1296467375000000
#406 requirement pudo rgrp datapkg-0.8 closed fixed Download sub-system

Factor out current downloader into dedicated download subpackage, abstract interface and make it pluggable so we can have downloaders loaded as per type of resource to download.

And we should support interactive retrieving of resources ...

Implementation Ideas

for resources in pkg.resources:
    # datapkg/hg 
    load_storage_handler(resource.format)
1281346822000000 1296467427000000
#915 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Deploy dcat-tools/rdfsolr to publicdata.eu as a placeholder

We should present something there, why not start with a meta-search.

1295267040000000 1296467518000000
#402 task pudo pudo ckan-v1.3 closed duplicate Archiving worker to back up package resources from a CKAN instance

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

  • Naming scheme?
    • Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
      • What happens if names change
    • File: filename? hash?
  • Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages.

Extra points for:

  • Properly checking for source file modification (Last-modified, Etag)
  • Using PIP VCS Backends for retrieval
  • OFS/S3 Storage
1281018912000000 1296467635000000
#262 enhancement rgrp rgrp closed fixed Mark package as duplicate of another package

Need a way to mark packages as duplicate.

Ideas/Questions?:

  • Could work off a tag (meta-duplicate or somesuch)
  • What happens if we want to delete the package (to we leave a place-holder, do we merge content?)
1267139682000000 1296468392000000
#346 defect dread johnbywater ckan-v1.3 closed wontfix Revision search API (response data format and documentation issue)

Whilst going through the API docs for the revision search API, it was noticed that the "Gdu" SoS doc doesn't match up. It returns revision IDs (perhaps this is useful to note in the spec?) so the format is probably not 'limitedstring'. Also, they appear to be ordered youngest first, not oldest as stated.

And in the revision model, it refers to 'simplestring' which it doesn't define - I guess the names should be 'limitedstring'?

Could this be checked out?

1276523517000000 1296477510000000
#252 enhancement dread johnbywater closed invalid Change revision object so that it has parent(s) attribute 1266519767000000 1296477560000000
#699 defect pudo TimDavies closed fixed Paging search results re-sets Open Only and Downloadable Only flags

When getting a search results page (e.g. http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1) and then using the on-page links to page through results, the next link / numbered page links go to http://ckan.net/package/search?q=development&page=2 rather than http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1&page=2 meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings.

1287143698000000 1296499061000000
#903 defect pudo pudo ckan-v1.3 closed fixed Proper handling of deleted packages in Solr serps

Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr.

1294831054000000 1296588070000000
#884 enhancement thejimmyg wwaites ckan-v1.3 closed fixed harvest job delete operation

implement paster subcommand to delete harvest jobs

1293277800000000 1296592140000000
#563 requirement thejimmyg johnbywater closed duplicate Support a minimal CSW server interface or export to GeoNetwork 1284033576000000 1296592472000000
#711 requirement thejimmyg johnbywater ckan-v1.3 closed fixed Packages shall support both UK government and location information with a common set of attributes

Previously, a package type attribute was intended to distinguish between package data that came from government depts. and publishers location information (GEMINI). Now there is an intention use a common set of attributes to support data from either source.

Hence, there currently a v.3 of the government attributes, a document about extracting the attributes from Gemini document using XPath, and a comparison documents mapping between governemtn attributes and Gemini attributes.

That means there are potentially four different list of attributes. I'm not sure how to go about reviewing the lists and converging any differences.

But we do want to end up with a list of names, a description of their purpose, and then perhaps the XPath which would reveal the value from a Gemini doc?

1287580866000000 1296592589000000
#739 requirement thejimmyg johnbywater ckan-v1.3 closed fixed The system shall support retrieving a copy of the harvested metadata for a given package

...obviously, not all packages will have a harvested metadata document.

1287779710000000 1296592842000000
#740 requirement thejimmyg johnbywater closed duplicate Get copy of harvested metadata for a given package 1287779799000000 1296592889000000
#750 enhancement thejimmyg johnbywater closed duplicate Get CSW records modified since given time 1288014402000000 1296592940000000
#499 story thejimmyg johnbywater closed wontfix Registration and management of UKLII related code lists, controlled vocabularies and other forms of reference/static data 1282432571000000 1296593038000000
#789 requirement wwaites johnbywater ckan-v1.3 closed duplicate The harvest source form shall validate location field by positive identification of source type 1289212106000000 1296593257000000
#795 story thejimmyg johnbywater ckan-v1.3 closed duplicate "Match up" GEMINI2 "metadata point of contact" with registered entities

TM wants to know how this will be "matched up". I don't know whether this always the data provider, or sometimes the publisher.

1289228025000000 1296593361000000
#757 task thejimmyg johnbywater ckan-v1.3 closed duplicate Create migration script to add harvest source filter attribute to existing tables 1288040584000000 1296593448000000
#766 requirement wwaites johnbywater ckan-v1.3 closed duplicate The system shall support harvesting metadata from ArcGIS

Perhaps 80% of the users of the UKLII may be using ArcGIS.

1288177933000000 1296593519000000
#770 defect thejimmyg johnbywater ckan-v1.3 closed wontfix Improve the paster harvester help command

Issue 1. Help command without arguments fails:

$paster help --plugin ckan --config development.ini

gives:

EITHER

Command help not recognized

OR

paster help: error: no such option: --plugin

Issue 2: Harvester command with arguments fails.

$ paster harvester Traceback (most recent call last): ...

cmd = self.args[0]

IndexError?: list index out of range

1288190220000000 1296593925000000
#908 defect pudo rgrp ckan-v1.4-sprint-1 closed fixed Issues deploying extensions with modwsgi / uwsgi

Some extensions import pylons features (such as request) at root.

Have encountered following issues:

  1. You need to import ckanext before any ckan stuff in your wsgi script (extension stats and dataapi)
  1. Running under wsgi daemon mode (normal mode is fine!) the front page (front page only) will not work (does not occur with disqus or synchronous_search but does with stats and dataapi extensions). You get errors like this in the log:
[Mon Jan 17 09:01:19 2011] [error] Exception KeyError: KeyError(-1218594160,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Mon Jan 17 09:01:20 2011] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
1295264624000000 1296730578000000
#953 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Add tagline/description to template and set in configuration

Have a tagline / description are under main title and set if from ckan.site_description config variable.

1296809808000000 1296809834000000
#946 defect pudo dread ckan-v1.3 closed fixed No default system rights

CKAN currently comes out of the box being uneditable. We'd like to encourage people to have open CKANs and therefore by default init the db with this right:

visitor -> is editor on -> System system
1296499922000000 1296833383000000
#930 defect wwaites dread closed fixed call_timing files created for every request

On a production server we produce literally millions of little files saying how long every single request took to process.

They are here:

hmg.ckan.net.2/pylonsdata/call_timing

They were added by ckan/lib/base.py in cset:da438a9085d3.

I don't believe anyone uses these stats.

1296061721000000 1297066292000000
#245 enhancement rgrp rgrp closed duplicate Support for composite primary keys

Problem here is that foreign key then becomes "complicated" (composite).

  • Could also deprecate continuity_id field in favour of the basic foreign key on ie
1265882630000000 1297066620000000
#246 enhancement rgrp rgrp closed duplicate Support for primary key not named id

At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys)

1265882862000000 1297066757000000
#776 enhancement rgrp dread closed duplicate Avoid generating vdm warnings

We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided?

2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object
2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object
2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object
2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object
2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object
2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object
1288372797000000 1297066840000000
#916 enhancement rgrp pudo vdm-0.8 closed duplicate Port new vdm to Mongodb

We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG.

1295277011000000 1297066902000000
#205 enhancement rgrp dread closed wontfix Custom package importer

Importer function is exposed to logged in users.

Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'.

  • Basic is as it is currently
  • Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords.

All fields should be in step with the government custom input form.

For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text.

Estimate:

1260197819000000 1297068450000000
#667 defect thejimmyg dread ckan-v1.3 closed wontfix API is slow

See message and script: http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html

1500 requests shouldn't take hours.

1285885228000000 1297069121000000
#914 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Handle formats in dcat-tools
  • Create a list of valid predicates for format information (accessURL et al)
  • Execute #235 via Google Spreadsheets and include in DCat enrichment
1295266569000000 1297069958000000
#940 defect pudo rgrp ckan-v1.3 closed wontfix OpenID login via google for same user generates 2 different openid urls

This has been noted twice in the last couple of weeks.

Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See:

To fix this we should set the openid.realm (not sure how atm).

If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache).

Cost: 1h

1296401521000000 1297071748000000
#907 enhancement rgrp ckan-v1.3-sprint-3 closed fixed Data previews: preview package resource data in the WUI

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

  1. Need a 'data proxy' - see ticket:698 and ticket:888
  2. Preview using javascript in WUI - do this in an extension http://bitbucket.org/okfn/ckanext-dataapi
1295261173000000 1297072303000000
#404 requirement rgrp pudo datapkg-0.8 closed fixed Make registries pluggable

Allow using different indexes, including ckan, file, dcat, ...

1281346690000000 1297072955000000
#817 requirement cygri ckan-v1.3 closed worksforme Proposed redesign of Resources table on package pages

There are some problems with the Resources table on package pages:

  1. The column labelled “URL” doesn't contain URLs, but just the word “Download”.
  2. The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links).

Attached is a screenshot for a proposed redesign. The changes are:

  1. Move the Format column to the right of the Description column (description is more important)
  2. Make the description a clickable link
  3. Drop the “URL/Download” column because it's now redundant
  4. Rename “Description” to “Resource” (not that important)
1289997384000000 1297073724000000
#409 task rgrp rgrp datapkg-0.8 closed fixed Refactor cli to be more 'pluggable'
  • (Load cli commands from setuptools entry point name datapkg.cli)
  • Document how other can write new commands
  • Refactor base Command class to support this pluggability (if necessary)
1281348659000000 1297074197000000
#942 defect pudo pudo ckan-v1.4-sprint-1 closed fixed Fix SERP pager argument passing

It only passes "q", but the whole query must be serialized.

1296469490000000 1297074807000000
#949 enhancement pudo pudo ckan-v1.4-sprint-1 closed fixed Generic username/password login

based on repoze.who-friendlyForms

1296658126000000 1297074827000000
#557 defect pudo ckan-v1.3 closed invalid Remove siblings from link_to/from relationship display

Displaying siblings is not needed for link_to/from.

1283542595000000 1297075159000000
#511 requirement dread ckan-v1.3 closed worksforme Better warnings and errors when using API 1282754677000000 1297075354000000
#807 enhancement rgrp ckan-v1.3 closed wontfix Handle robot traffic better

We get a lot of robot traffic. Notice that robots are visiting package edit pages. Suggest:

  • Add rel=nofollow to various links
  • Throttle some robots (not all seem to respect robots.txt)
1289899826000000 1297075372000000
#809 requirement pudo ckan-v1.3 closed wontfix Link to the Wiki pages in package form

This would allow for a community-driven definition of field values, conventions, etc.

Plus: create wiki.ckan.net

1289992231000000 1297075561000000
#943 enhancement pudo pudo ckan-v1.4-sprint-1 closed fixed Create CKAN Wiki

Set up a Mediawiki to contain info on CKAN.net (community) conventions etc.

1296469547000000 1297076498000000
#932 enhancement dread dread ckan-v1.3 closed fixed Move to SQLAlchemy-Migrate 0.6

Basic steps from kindly:

  • changing the setup.py dependencies for sqlalchemy-migrate and sqlalchemy to their latest versions.
  • running the migrate script on the repository (or changing the repository scripts manually).
  • checking in both these changes at once.
1296156660000000 1297076710000000
#938 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Message flashing / notification in WUI

Good user interfaces provide feedback to users. We should provide more feedback about the success / failure of actions using message flashing (see http://flask.pocoo.org/docs/patterns/flashing/).

We already do some of this (slightly poorly), e.g. on adding an authorization the app says: "X authorization added".

Implementation

  • Standardize this process and add section to html for this purpose.
  • Already done in other OKFN apps e.g. yourtopia.net, wheredoesmymoneygo and commentonit
  • Testing: suggest keeping this very simple

Estimate

  • Cost: 0.5h (html etc) + 1h to do some integration
1296398920000000 1297077023000000
#48 enhancement rgrp johnbywater ckan-v1.4-sprint-1 closed fixed Create favicon

Because CKAN's worth it!

1201202789000000 1297077492000000
#902 defect rgrp dread ckan-v1.4-sprint-1 closed fixed Package buttons not clickable / Improve submenu tabs

On a package page there are three buttons to the right - View, Edit, History. Each of these buttons contain an icon and the text, but you can only activate the button by clicking on the icon. When you click on the part of the button where the text is (about 50% of the button) then nothing happens.

Using: Chrome, Firefox

1294756898000000 1297078885000000
#950 enhancement dread dread ckan-v1.3 closed fixed Move importer controller to ckanext-importer

ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN.

In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext.

1296666038000000 1297079743000000
#99 enhancement rgrp rgrp closed wontfix Test DbIndex with sqla 0.5

Test DbIndex? with sqla 0.5 (and get it working if it does not work).

Cost: ??

Priority: low because DbIndex? is not currently that important to us (also wonder whether we should try using CKAN code here to stop us reinventing the wheel).

1249985537000000 1297081088000000
#560 defect kindly rgrp ckan-v1.4 closed fixed Deleted packages showing up in tag lists

E.g. http://ckan.net/tag/lod contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins).

Also suggested that these are showing up in the API (courtesy of Anja)

Cost: 2h

1283700001000000 1297084192000000
#875 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Search backend supports solr interface and query API mimics solr

Consolidate search API interface (and backend) on solr (solrpy) type interface.

  • Support for standard query structure
  • Support for facet options

Do not need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

  1. Disable (happens automatically)
  2. Implement - suggest using group by etc

Extras

  • Front-page tag cloud: change this to use facets
    • Accept this means that if facets not functional in backend we have no tag cloud
1292844957000000 1297085261000000
#928 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Documentation - CKAN credits / contributors page
  1. Remove contributors list from about page (out-of-date and does not belong there)
    • A "Powered by CKAN" link in footer is enough
  2. Add contributors / team / credits page to ckan.org
    • May want to separate credits (e.g. software use), from contributors (also is team different from contributors)
    • (perhaps with photos!) so that people can see who is part of it and from which organisations. (this is medium-term, not really part of this ticket)
  3. ? CREDITS.txt file to the source repository and add a link to it on bitbucket for more detailed credits info.

Cost: 1h

1295872661000000 1297115136000000
#960 defect pudo rgrp ckan-v1.4-sprint-1 closed fixed Support non-ascii character in internal error messages

Just saw an error like the following which looks like it is to do with having a user object with non-ascii characters in it. Either get rid of 'cast' to str type here or do it in a unicode aware way.

Should do this not just here but everywhere we can find in the code base.

Module ckan.controllers.package:302 in edit
<<          am_authz = self.authorizer.am_authorized(c, model.Action.EDIT, pkg)
               if not am_authz:
                   abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
       
               auth_for_change_state = self.authorizer.am_authorized(c, model.Action.CHANGE_STATE, pkg)
>>  abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Estimate

Cost: 1h

1297069721000000 1297169056000000
#98 enhancement rgrp rgrp datapkg-0.7 closed fixed Replace use of pastescript templates in PythonDistribution with of something simpler

Replace use of pastescript templates in PythonDistribution? (write) with of something simpler (allowing us to remove dependency on pastescript).

1249983810000000 1297210774000000
#171 enhancement rgrp rgrp datapkg-0.7 closed fixed Create consolidated config for CLI

Create consolidated set of config by merging config from .datapkgrc and command line options and use that instead of mixture of command line options and other stuff ... (likely to be less buggy!)

1256488011000000 1297210925000000
#964 enhancement rgrp rgrp datapkg-0.8 closed fixed Create Resource Downloader and make it pluggable

In downloading packages we need to download resources and we need a ResourceDownloader? object for this. these should be pluggable so that we can add support for different types of resources.

1297211004000000 1297211237000000
#405 task rgrp rgrp datapkg-0.8 closed fixed Retrieval options for package resources

Download Command (was install command) should support the following modes:

  • Download only the first listed resource (current behaviour, slightly arbitrary)
  • Download resources by interactive selection
  • Download resources by MIME type (cf #235)
  • Download all resources.
1281346806000000 1297214833000000
#232 enhancement rgrp rgrp closed invalid Add strapline to CKAN logo

Strapline (?): Sharing, discovery and reuse of data and content

Need to consult on this.

Cost: 1h

1262975291000000 1297246297000000
#665 requirement johnbywater johnbywater ckan-v1.3 closed duplicate The system shall support withdrawing a harvested dataset or service from publication

Discussion between John and Peter:

Given we can identify a document, does the disappearance of a document from a registered source imply the disappearance of the metadata (such that we delete packages once the documents disappear from the registered source)?

I would expect a more explicit 'delete'. The UKLP Use Case Library describes this as "withdraw a dataset or service from publication" (part of UCD03 Maintain resources).

1285588250000000 1297268097000000
#955 defect dread dread ckan-v1.3 closed fixed Tidy up tests
  • Removal of init_db from every setup etc.
  • Get them working
1296819300000000 1297342534000000
#974 defect dread ckan-v1.3 closed fixed Document site_description

Configuration option 'ckan.site_description' isn't documented

1297342254000000 1297342599000000
#975 defect rgrp rgrp ckan-v1.4-sprint-1 closed fixed Placeholder attribute not displayed in some browsers leading to poor UX

Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE).

This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search.

We should fix this by finding a way to display placeholder attribute in all browsers.

1297343755000000 1297344448000000
#92 enhancement rgrp rgrp closed wontfix Add RDFa to package pages 1249049927000000 1297344859000000
#976 defect dread ckan-v1.3 closed fixed Documentation references knowledgeforge.net 1297346649000000 1297346954000000
#569 requirement thejimmyg johnbywater ckan-v1.4 closed fixed Harvested GEMINI document metadata validation 1284040221000000 1297347214000000
#933 enhancement kindly ckan-v1.4 closed fixed get rid of self when it should be cls in classmethods

use pylint

1296174526000000 1297348975000000
#682 enhancement dread dread ckan-v1.4-sprint-1 closed fixed Add ckanclient tests to buildbot

Some work done on this to make tests run independently.

1286386769000000 1297358266000000
#977 defect dread ckan-v1.3 closed fixed db_upgrade errors
  • db_upgrade (in ckan/model/init) makes call to validate_authorization_setup but forgets about rest of initialisation.
  • it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?)
1297420558000000 1297420742000000
#980 defect pudo pudo ckan-v1.4-sprint-1 closed fixed Fixes to HR Infoshare

We've fixed a few things in CKAN that were discovered and tracked by HRI in their own issue tacker. This includes:

  • add authz checks to package edit links (cset:0752316cd2fe)
  • replace gettext with _ in controllers, to support unicode error msgs (cset:822340e6077e)
  • handle broken html in notes field without crashing on package read (cset:4b6be037dda0)
  • update i18n (cset:37d57dc3c492, cset:ea03173f5e77, cset:f16f4ee40fe7)
1297430918000000 1297503177000000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Note: See TracReports for help on using and creating reports.