{22} Trac tickets (2647 matches)

Results (201 - 300 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#674 task johnbywater closed Extract metadata documents from CSW GetRecordById 1286214780000000 1286787032000000
#680 task dread dread closed fixed Prepare for DGU tech meeting 1286386074000000 1287141809000000
#683 story dread dread closed fixed metastable, stable & ultrastable smoketests run by buildbot 1286388134000000 1287142289000000
#684 task dread dread closed fixed metastable tested on buildbot 1286388181000000 1286452710000000
#685 task dread dread closed fixed stable tested on buildbot 1286388206000000 1287997008000000
#686 task dread dread closed wontfix ultrastable tested on buildbot 1286388225000000 1287997047000000
#694 bug wwaites dread closed fixed No postgres tools for current version

Database for all CKAN instances upgraded to Postgres 8.4, but none of the eu machines were upgraded with the tools necessary to administer them.

1286977052000000 1287087916000000
#695 bug pudo dread closed fixed Search indexing broken on ckan.net

e.g. searching for 'buddhist' or 'sanskrit', you don't get this newly created package: http://ckan.net/package/digitalsanskritbuddhistcanon

1286991201000000 1287766973000000
#699 defect pudo TimDavies closed fixed Paging search results re-sets Open Only and Downloadable Only flags

When getting a search results page (e.g. http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1) and then using the on-page links to page through results, the next link / numbered page links go to http://ckan.net/package/search?q=development&page=2 rather than http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1&page=2 meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings.

1287143698000000 1296499061000000
#714 bug johnbywater closed fixed DGU package form shall have a read only field for ??? attribute

So that Drupal doesn't need to make any adjustments to the form, and risk data being lost on submission.

1287581408000000 1291733788000000
#717 bug pudo pudo closed fixed Fix and validate "setup_default_user_roles" in IATI 1287583892000000 1290005099000000
#729 bug pudo closed fixed Assets to be loaded from assets.okfn.org, not m.okfn.org

Move from hetzner box to s3.

1287736519000000 1288012854000000
#730 task rgrp closed duplicate Back up package data from all CKAN packages to storage.ckan.net

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net.

  • Naming scheme?
    • Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
      • What happens if names change
    • File: filename? hash?
  • Store hash back on ckan instance?
1287737109000000 1291139609000000
#735 task dread dread closed fixed Investigate 'deduping' (ONS import)

ww 22/9/10:

(discussion of ticket:509) This leads us to deduping which should be the next order of business oin this thread...

ww 20/10/10:

if a dataset is missing a department and the ons script makes a new version of it with a department we now have two reccords for the same dataset. these should be manually merged - need a merge tool where you can tell it that two datasets are the same.

1287748786000000 1292957248000000
#740 requirement thejimmyg johnbywater closed duplicate Get copy of harvested metadata for a given package 1287779799000000 1296592889000000
#743 story johnbywater closed wontfix Create harvest source registration via Web UI 1288012860000000 1294412807000000
#744 story johnbywater closed wontfix Update harvest source registration via Web UI 1288012960000000 1294412813000000
#746 story johnbywater closed wontfix Request new harvesting job via Web UI 1288013198000000 1294412827000000
#747 story johnbywater closed wontfix Track harvesting jobs via Web UI 1288013217000000 1294412834000000
#748 story johnbywater closed duplicate Link new sample package to previous sample package of continuous series 1288013849000000 1294412976000000
#749 story johnbywater closed duplicate Fold up continuous series in search results behind newest sample package 1288014002000000 1294412986000000
#750 enhancement thejimmyg johnbywater closed duplicate Get CSW records modified since given time 1288014402000000 1296592940000000
#751 story johnbywater closed duplicate Get harvested document for a given package 1288014518000000 1288014616000000
#761 story dread closed invalid Read-only CKAN for maintenance

Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done.

1288090334000000 1288091977000000
#762 story dread closed invalid Permanently Read-Only CKAN instance

A CKAN is used just for distributing metadata. Updates may still arrive through direct db manipulation, e.g.:

  • another (but writable) CKAN instance is connected to the same db
  • restoring database dumps from another CKAN db
1288090578000000 1288091982000000
#775 task pudo dread closed fixed Search warning

We're getting this warning a great deal on live servers. Is these really a sign of the system not operating correctly or can we reduce the level to an INFO?

e.g. on hmg.ckan.net:

2010-10-29 17:12:08,262 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,333 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,375 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,406 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,480 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
2010-10-29 17:12:08,613 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position
1288372692000000 1295260144000000
#776 enhancement rgrp dread closed duplicate Avoid generating vdm warnings

We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided?

2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object
2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object
2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object
2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object
2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object
2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object
1288372797000000 1297066840000000
#799 requirement johnbywater johnbywater closed duplicate WAF harvesting shall be optimised in respect of reharvesting only records that have changed 1289483264000000 1294232675000000
#802 story johnbywater johnbywater closed duplicate Save last harvested time on source 1289484226000000 1294233294000000
#803 story thejimmyg johnbywater closed invalid Auto-generate next migration version unit and test stub files

As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually.

1289486678000000 1314031451000000
#820 defect dread closed fixed Exception browsing /user/apikey when not logged in

Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in.

WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'apikey' 						

URL: http://test-hmg.ckan.net/user/apikey
1290006079000000 1311325226000000
#822 enhancement Javier de la Cueva <jdelacueva@…> closed fixed Docs: Configuration as site-available instead of site-enabled for apache

The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net

1290107103000000 1290506354000000
#823 defect dread dread closed fixed Etags cache doesn't update when only non-core package fields changed.
  1. Create a package.
  2. Edit package and add a resource.
  3. View package in browser - no sign of the resource - ERROR.
  4. View package in API - resource is there.

Reason: hash doesn't change because package.revision only takes account of core fields.

1290505078000000 1290506116000000
#824 enhancement dread dread closed fixed 'Getdata' moved out of CKAN

Data importers should use API instead of object model directly:

  • saves load on server
  • much better error checking and authentication
  • eating our own dogfood (tests the API well)
1290595507000000 1290595559000000
#825 enhancement dread dread closed fixed Document fabric tool 1290609160000000 1290624449000000
#827 defect rgrp dread closed fixed metadata_created_and_modified error

Intermittent failure on ckan default branch. Looks like brittle test.

======================================================================
FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py", line 250, in test_02_metadata_created_and_modified
    assert out == exp, (out, exp)
AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938))

http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio

1290678624000000 1290766239000000
#830 enhancement rgrp closed fixed Themes for CKAN

Already have the basic capacity using *_extras paths for templates and public directory (see http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: http://packages.python.org/Flask-Themes/

1290760612000000 1311180263000000
#843 enhancement wwaites closed fixed user list ui for group curation
  • perhaps we should show the openid as well to distinguish between users with the same name.
  • perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name.
  • also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it.
1291484245000000 1319721601000000
#844 defect wwaites wwaites closed fixed bad mail config ckan.net/ckan.org

ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example:

B30E6D9F       2873 Sat Dec  4 14:05:40  MAILER-DAEMON
(delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out)
                                         [email protected]

both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null

1291556753000000 1296340486000000
#849 defect pudo closed duplicate Import extra CSS from main template

The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import.

1291714923000000 1291715179000000
#855 defect pudo dread closed invalid Authenticator tests

The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests.

1291723967000000 1311176988000000
#856 defect wwaites dread closed invalid Tests for caching

Test coverage for ckan.lib.cache is only 40%. Can that be usefully improved?

1291724064000000 1311177085000000
#857 defect dread closed invalid Tests for CLI

Test coverage for ckan.lib.cli is 34%. Need to improve that.

1291724131000000 1311182581000000
#858 defect dread dread closed fixed Tests for diff

Test coverage for ckan.lib.diff is 24%. Need to improve

1291724171000000 1291734297000000
#859 defect pudo dread closed invalid Tests for User model

Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement?

1291724348000000 1311177461000000
#861 defect wwaites wwaites closed invalid unicode errors creating rdf output

http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d

1291726125000000 1311168845000000
#863 enhancement memespring memespring closed wontfix prompt users for missing package information

e.g. license, owner etc

1291729787000000 1338206455000000
#868 enhancement sebbacon thejimmyg closed fixed Test improvements

The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach.

We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!)

Some suggestions for achieving this include:

  • Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database
  • Not setting up and tearing down the database so frequently.
1291987811000000 1294753596000000
#871 defect nils.toedtmann closed invalid Check whether localhost-only exim installtions need upgrading too

The infamous exim bug only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers.

We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not.

BTW did we already run a rootkit checker like Rootkit hunter on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark? has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could.

1292264117000000 1296340558000000
#872 enhancement johnbywater dread closed fixed ONS loader assigns packages to same package even when dept name changes

Solutions: ResourceSeriesLoader? allows synonyms for values.

1292424112000000 1292957110000000
#873 defect dread dread closed fixed ONS loader assigns packages to same package even when dept name added manually

e.g. national_child_measurement_programme had no department but "Information Centre for Health and Social Care" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_.

1292437253000000 1292939274000000
#881 defect thejimmyg fccoelho closed invalid http authentication requirement when installing

I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code.

1293021889000000 1296335072000000
#882 defect pudo pudo closed wontfix Update i18n user docs and backport genshi i18n:domain fixes from WDMMG 1293192044000000 1340626094000000
#896 story thejimmyg closed duplicate Distributed Data and Syncing Between CKAN Instances

Based on the ideas in:

Merging the tickets #296, #297, #298, #299 into this one.

Related ideas include:

  • Using dcat to exchange data
  • Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance

This should be considered a wishlist item for the timebeing.

1294406999000000 1300217375000000
#897 defect dread dread closed duplicate "New package" highlighted in Navigator when searching

On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted.

1294648815000000 1294914333000000
#898 defect rgrp dread closed fixed Changes stored indefinitely

Every change to every object is being stored in memory, which could add up to quite a lot of memory.

This fixes it by making sure the objects are in a weakref. https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f

1294659490000000 1294662408000000
#899 enhancement rgrp dread closed fixed Optimise check_real_change

There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e

From David Raznick

1294659583000000 1294916615000000
#900 defect dread closed fixed Intermittent error with new revision not having id

There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision. We add the uuid earlier to fix this. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e.

From David Raznick

1294659696000000 1294916587000000
#901 enhancement rgrp dread closed fixed vdm support for sqla 0.6 and sqlite 1294660144000000 1294662466000000
#917 enhancement kindly kindly closed fixed Turning off Autoflush in SQLAlchemy

David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior.

I have a patch in https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default.

This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet).

Things to know about using flush:

  • Most of the time nothing will change. A commit implies flush.
  • You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query.
  • A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated.
  • You can use them as a limited form of nested transaction.

It increased the performance on the tests by about 5%. There are less flushes altogether so is faster.

I do not see any downsides.

1295279694000000 1295280232000000
#918 defect memespring dread closed invalid Package preview doesn't show tags or groups
  1. Edit any package
  2. Hit 'preview'

Notice that the preview at the bottom doesn't show what tags or groups the package is in.

This is because the new design of the package view only has this info in the sidebar, which is not displayed in the preview.

One might argue that this is ok because the tag is not a central part of the metadata, only used for searching, so it is ok not to preview it. But I wanted to check that this change was a conscious decision.

1295346496000000 1315911507000000
#920 defect thejimmyg hellmann@… closed fixed empty tags are kind of confusing

http://ckan.net/tag/linguistsic doesn't have a single entry, but is still kept and displayed.

It should be purged !!!

1295355280000000 1300319140000000
#921 enhancement sebbacon thejimmyg closed fixed Refactor DGU Harvesting

Refactor harvesting code to simplify.

1295389237000000 1300197629000000
#922 enhancement kindly kindly closed duplicate Make Resources first class entity.

Why?

  • Will make it easier to give resources their own form, independent of packages.
  • Makes uploading, harvesting and previewing data cleaner as its separate from the packages.
  • Gives the option of sharing a resource between packages.
  • Leads the way to supporting storing richer semantic information against them.
1295713858000000 1320664187000000
#923 defect rgrp dread closed worksforme Search box doesn't work in leaderboard page
  1. Go to: http://ckan.net/stats/leaderboard#content
  2. In the far top-right of the browser, select the search box in 'water'.
  3. Press enter to search. Nothing happens.

Tried in: chrome, firefox

1295867328000000 1340632144000000
#930 defect wwaites dread closed fixed call_timing files created for every request

On a production server we produce literally millions of little files saying how long every single request took to process.

They are here:

hmg.ckan.net.2/pylonsdata/call_timing

They were added by ckan/lib/base.py in cset:da438a9085d3.

I don't believe anyone uses these stats.

1296061721000000 1297066292000000
#934 enhancement kindly closed invalid Key value store for extensions to store data

We need to allow plugins to store information.

Each plug-in should have its own name space and allow any data to be set.

There are two options.

  • make a key value table in ckan that will hold random information.
  • use redis (or other key value store)

If in sql suggest schema of form:

  • (namespace, obj_id, key, value [, value_type])
  • value is json
1296228758000000 1323171047000000
#935 defect pudo pudo closed fixed "groups" field bug in IATI

An internal server error was reported when trying to create a new package (not when editing):

WebApp? Error: <class 'sqlalchemy.exceptions.IntegrityError?'>: (IntegrityError?) null value in column "name" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'}

The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the "groups" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server.

Possible causes include:

  • Stale pyc file
  • Threading issue
  • Dependency issue

As the problem is gone for now, it cannot be further investigated atm.

1296302915000000 1306778682000000
#939 enhancement rgrp closed fixed Notification bar at top of site shown to new users pointing them to simple instructions

On http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions.

Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better.

Implementation

  • Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing)
  • FAQ/instructions page (use the new wiki?)

Depends

Estimate

  • Cost: 1h
1296399008000000 1323171158000000
#947 enhancement rgrp closed duplicate Move the assignment of roles to a package on package creation to an extension

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

1296557478000000 1314031310000000
#951 defect adrian.pohl@… closed invalid Can't add a package to group

I can't add a package (e.g. http://ckan.net/package/ub-konstanz) to a group (e.g. http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is "history") nor on the group page.

1296726886000000 1314031006000000
#957 enhancement kindly kindly closed wontfix new "kind " field added to resources

The new field was outlined in

http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

it says

Can be "data", "metadata", "service", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.

1296821398000000 1314218701000000
#971 enhancement sebbacon sebbacon closed duplicate Support middleware as a plugin

I need to completely munge the rendered data coming from the app. There's not currently a plugin hook for this, where I get access to the entire environment. The most generic form of plugin would be WSGI Middleware. I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system.

1297277473000000 1299245064000000
#987 defect pudo pudo closed duplicate Common harvesting framework

We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include:

  • CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
  • Catalogue scraping for LOD2 experiments (yields RDF graphs)
  • Atom/DCat for LOD2 production (yields RDF graphs)
  • OAI-PMH for http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios.

In general, such a system would have the following stages:

  • Source selection: find what to download/scrape/harvest/parse
  • Index retrieval (i.e. package index)
  • Item retrieval (i.e. package entity)
  • (Optional: Serialization)
  • Normalisation
  • Loading/Merging? into CKAN

Exisiting harvesters are at:

1297684756000000 1311177705000000
#990 defect rene.kapusta closed fixed umlauts in title / text

It's not possible to use umlauts (like "CKAN Österreich") in the title / text.

1297801036000000 1311180850000000
#995 defect kindly kindly closed wontfix consolidate caching

Try and minimise the amount of different caching methodologies, especially for api calls.

1298283499000000 1311179009000000
#997 defect kindly kindly closed duplicate change create on cli to upgrade

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

1298287981000000 1298288665000000
#998 defect kindly kindly closed fixed change create on cli to upgrade

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

1298287981000000 1300364423000000
#1005 defect dread dread closed duplicate Editing in ckan missing publishers

Package edit form at hmg.ckan.net has no publishers any more.

1298561348000000 1300100085000000
#1007 defect pudo pudo closed wontfix Backport WDMMG translations for extensions i18n

Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class.

cf. https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16

1298630621000000 1340626203000000
#1017 defect pudo sebbacon closed fixed Problem assigning users to authz groups through web interface

Against ckan-1.3.1, when I create an authz group called "administrators" and visit /authorizationgroup/edit/administrators, I am unable to add more than one user to it.

Each time I add additional users, the existing user on the list is replaced with the new one.

1299071127000000 1299668555000000
#1020 defect kindly closed fixed harvesting doc revision table

add revisioning to harvested document table

1299205012000000 1300196215000000
#1023 defect pudo pudo closed wontfix Re-queueing of worker requests with errors

Schedule worker calls to be ran several times, with certain intervals.

1299493179000000 1340626231000000
#1030 defect amercader thejimmyg closed fixed Move harvesting out of the rest API 1299776418000000 1303117978000000
#1033 defect dread dread closed fixed Register user with blank password causes 500

Go to http://ckan.net/user/register and fill in all the fields apart from password. On submit you get 500 error.

1299796274000000 1308310446000000
#1034 defect dread closed duplicate Flash message cached
  1. Login (shows ckan home page with flash message "Welcome back xyz")
  2. Click away, to "Add package" say
  3. Click back to ckan "Home". It shows "Welcome back xyz" again - it shouldn't, I assume?

1299845308000000 1320174353000000
#1035 enhancement thejimmyg dread closed wontfix Form impressions given an ID

To counter Cross Site Request Forgery attacks, each form generated is assigned a random number in its url, which must be passed when you submit the form.

(Something to consider for the new form mechanism)

1299857978000000 1338206251000000
#1042 task dread dread closed fixed 'Ckanext' split-up

It's not good to have ckanext doing lots of different things with different dependencies. Split it off into:

  • ckanext-importlib

etc.

And then deprecate the ckanext repo itself.

1300293907000000 1300969865000000
#1043 defect kindly closed fixed stop sqlalchemy message saying at least one scoped seession already present

see summary.

1300310326000000 1300321033000000
#1044 defect pudo dread closed fixed Sysadmins locked-out of API without Right: (visitor, SITE_READ, System)

The problem is that in ckan/controllers/rest.py the BaseApiController? has this method:

    def __before__(self, action, **env):
        BaseController.__before__(self, action, **env)
        if not self.authorizer.am_authorized(c, model.Action.SITE_READ, model.System):
            abort(401, _('Not authorized to see this page'))

which works on the basis of your c.user, rather than your apikey. All API users are treated as visitors (since API users don't get a login cookie) and even a sysadmin's apikey is blocked unless there is a right for a Visitor to SITE_READ.

Also needs tests.

(Also, why is this restriction only on the API, package search, group index and tags and agroup index? I'm guessing SITE_READ is only for places where other authz don't apply, but maybe it should not be called 'SITE_READ' but 'OTHER_READ' or something?)

1300358919000000 1302096155000000
#1049 defect dread closed invalid Fix database errors on offener.datenkatalog.at etc.

We're getting these sorts of exceptions from http://offener.datenkatalog.at/ like we had from ckan.net, which was due to small errors in migration scripts. Is it worth fixing this database, and in fact all the ckans on eu3 at once?

WebApp Error: <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "group_revision_pkey" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, ...

Thoughts James and Rufus?

1300444912000000 1323169424000000
#1050 enhancement thejimmyg johnlawrenceaspden closed invalid Authz lib improvement and refactor of ckan/lib/authztool.py

Refactor ckan/lib/authztool.py so that the relevant methods are independent of the command line interface.

The extracted methods should live in a new file ckan/authz.py. authztool.py should probably move into cli.py and will just do command line parsing and printing and use ckan/authz.py. The updated web gui for authz will also use this code.

Tests should be made. There's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests.

all to go on a branch feature-1050-refactor-authtoolz

Optional extras

  • Rename ckan/authz.py to ckan/lib/authz.py or even ckan/logic/authz.py
1300451937000000 1315394117000000
#1057 defect dread closed fixed JSONP parameter isn't escaped
$ curl "http://127.0.0.1:5000/api/rest/package/annakarenina?callback=<script>jsoncallback"

gives:

<script>jsoncallback({"id": "c10ebd31-5b45-4f6f-885d-dca9b18caec4", "name": "annakarenina", "title": "A Novel By Tolstoy",

which could run script code in the client who made the call.

One idea for filtering: http://tav.espians.com/sanitising-jsonp-callback-identifiers-for-security.html Maybe just better to have a restricted whitelist of characters to be even more sure.

Same as: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/906

1301078389000000 1329150236000000
#1063 defect sebbacon sebbacon closed fixed Groups listing widget on package screen shouldn't show group name by default

I've been asked if we can do something about the overflow of the Group name in the right hand column on this page:

http://register.data.overheid.nl/package/europese-aanbestedingen

The reason is that the list display for groups is in the form "group_tltie (group_name)", and of course group_name can't have spaces and so can't wrap nicely.

I was wondering if there's a good reason why we don't only display group_title (if it exists) and group_name only when there's not a title?

1301408459000000 1302514033000000
#1064 defect amercader closed duplicate Remove Workers from ckanext-queue

The current implementation of Workers in ckanext-queue is broken. Basically the various consume / callback functions expect three arguments (routing_key, operation, payload) when they are in fact receiving only two of them (message_data, message). This is fairly easy to fix, but the question is if Workers add an extra complexity to use the messaging library directly.

1301417891000000 1323169787000000
#1069 enhancement tobes rgrp assigned Stub datasets (request for datasets)

Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki.

We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ...

(Just as we have datasets listed that exist but aren't available ...)

Alternative would be to have a request for datasets subsystem.

I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists.

Implementation

  • Agree a new dedicated tag. e.g. todo.does-not-exist
1301666919000000 1340632215000000
#1081 defect johnlawrenceaspden johnlawrenceaspden closed fixed can't remove user from authz group

I've found that if I make an authorization group I sometimes can't remove myself from it. I've no idea why. I can add and remove other users. I'll investigate, just making a note of it here.

1302541056000000 1303489474000000
#1082 defect johnlawrenceaspden closed fixed language changes behave strangely

Set language to Greek, flash message says 'Language set to: English', but page is now about half in Greek.

Set language back to English causes server error:

AttributeError?: 'NoneType?' object has no attribute 'path'

Module ckan.controllers.error:29 in document view

if original_request.path.startswith('/api'):

However going to a new page reveals that it's back to English

1302541989000000 1315917217000000
#1085 defect dread johnlawrenceaspden closed fixed local development copy of ckan depends on existence of ckan.net

ckan.net appears to have either gone down or be running ultra slowly.

this means that ckan copies running locally on my machine run very slowly indeed.

is this behaviour desirable?

This command finds lots of http://~~~ckan.net references in python, html and javascript files:

find ~/pyenv/src \( -name "*.py" -or -name "*.html" -or -name "*.js" \) -print0 | xargs -0 -e grep --color -nH -e "http://.*ckan.net"

output for reference:


/home/okfn/pyenv/src/ckan/ckan/init__.py:5:Network (CKAN) site: http://www.ckan.net. /home/okfn/pyenv/src/ckan/ckan/lib/create_test_data.py:346:<http://ckan.net/> /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:3:DOMAIN = 'http://ckan.net' /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:4:CKAN_NAMESPACE = 'http://ckan.net/#' /home/okfn/pyenv/src/ckan/ckan/lib/talis.py:60: 'ckan':'http://ckan.net/ns#', /home/okfn/pyenv/src/ckan/ckan/public/scripts/bookmarklet.js:2: f='http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&title='+encodeURIComponent(document.title); /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:16: addtockan.src='http://ckan.net/scripts/bookmarklet.js'; /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:27: <p><strong>Proper bookmarklet (compressed -- need to escape &amp;):</strong> <a href="javascript:(function(){f='http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;title='+encodeURIComponent(document.title);if((n=document.getElementsByName('description')[0])&amp;&amp;(d=n.content)){f+='&amp;notes='+encodeURIComponent(d);}a=function(){if(!window.open(f)){location.href=f;}};if(/Firefox/.test(navigator.userAgent)){setTimeout(a,0)}else{a()}})()">Add to CKAN</a> /home/okfn/pyenv/src/ckan/ckan/templates/home/license.html:31: For convenience, all material - including all package, tag and revision information - is available in bulk, in the form of a full dump of the CKAN database. This (gzipped) dump file is updated daily and can be downloaded from <a href="http://www.ckan.net/dump/">http://www.ckan.net/dump/</a>. /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:71: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:137: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:447: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:458: 'notes': u'Some test notes\n\n### A 3rd level heading\n\nSome bolded text.\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<http://ckan.net/>\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/functional/api/base.py:178: assert '"ckan_url": "http://test.ckan.net/package/annakarenina"' in msg, msg /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:116: api e.g. http://ckan.net/api rather than http://ckan.net/api/rest) /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:261: :param base_location: default *http://www.ckan.net/api* /home/okfn/pyenv/src/ckanclient/ckanclient/init__.py:267: base_location = 'http://www.ckan.net/api'


1302620434000000 1302625314000000
#1086 defect thejimmyg johnlawrenceaspden closed wontfix no way to delete authorization groups from web interface

as title.

1302625333000000 1323346552000000
1 2 3 4 5 6 7 8 9 10 11 12 13
Note: See TracReports for help on using and creating reports.