{22} Trac tickets (2647 matches)

Results (1201 - 1300 of 2647)

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#1121 enhancement dread closed wontfix JSON extras appear in package edit form mangled

It is possible to use the CKAN API to insert JSON format data into package extra values, but this data is displayed in the package edit form.

Example: Package http://ckan.net/package/hbz_unioncatalog in the API the extra value as a list:

"extras": {"publishingInstitution": ["http://lobid.org/organisation/DE-605", "http://lobid.org/organisation/DE-290"...

yet when you edit the package it loses all the quotes and brackets: http://ckan.net/package/edit/hbz_unioncatalog {{{http://lobid.org/organisation/DE-605http://lobid.org/organisation/DE-290... }}} so when you save the package, the list is mangled into a bad string.

1304367504000000 1307358426000000
#1122 enhancement pudo dread closed wontfix JSON Extra data not searchable

It is possible to use the CKAN API to insert JSON format data into package extra values, but this data is not found on searching.

Full text from Pascal:

we encountered a Problem concerning accessing Arrays/Lists.

curl -XGET 'http://ckan.net/api/rest/package/hbz_unioncatalog'

will get you amongst others:

 "extras": {"publishingInstitution":
"[u'http://lobid.org/organisation/DE-605',
u'http://lobid.org/organisation/DE-290',
u'http://lobid.org/organisation/DE-38M',
u'http://lobid.org/organisation/DE-98',
u'http://lobid.org/organisation/DE-38',
u'http://lobid.org/organisation/DE-Kn41',
u'http://lobid.org/organisation/DE-82',
u'http://lobid.org/organisation/DE-107',
u'http://lobid.org/organisation/DE-929',
u'http://lobid.org/organisation/DE-Zw1',
u'http://lobid.org/organisation/DE-832']"}

but if I try to query this:

wget
'http://ckan.net/api/search/package?q=lobid&publishingInstitution="http://lobid.org/organisation/DE-605"'

I get only two packages, among the package "hbz_unioncatalog" is
missing. (These two packages have only one value for
"publishingInstitution").

The "extra/publishingInstitution"-Array was uploaded through a "curl
-XPUT ...
 "extras": {
   "publishingInstitution":[
     "http://lobid.org/organisation/DE-605",
     "http://lobid.org/organisation/DE-290",
     "http://lobid.org/organisation/DE-38M",
     "http://lobid.org/organisation/DE-98",
     "http://lobid.org/organisation/DE-38",
     "http://lobid.org/organisation/DE-Kn41",
     "http://lobid.org/organisation/DE-82",
     "http://lobid.org/organisation/DE-107",
     "http://lobid.org/organisation/DE-929",
     "http://lobid.org/organisation/DE-Zw1",
     "http://lobid.org/organisation/DE-832"
   ]
   },
...
1304367510000000 1306747714000000
#1123 requirement dread nils.toedtmann closed fixed Please re-package CKAN packages as "noarch"

... or, if the CKAN packages do contain architecture-specific binary code, build packages for i386 too.

Currently, http://apt-alpha.ckan.org/debian only offers packages for amd64, but e.g. "m1.small" EC2 instances are i386.

We would need this in order to migrate the community instances to a packaged based CKAN.

Rufus, pls prioritise.

1304530050000000 1311863806000000
#1124 enhancement thejimmyg nils.toedtmann ckan-sprint-2011-12-05 closed fixed push apt package python-ckanext-solr into our debian repository

python-ckanext-solr is already available in http://apt-alpha.ckan.org/datanl-dev, but not yet in http://apt-alpha.ckan.org/debian (that is why we had to [pip-install it for DataGM). Please push into main repo.

1304537793000000 1323168156000000
#1125 enhancement dread nils.toedtmann closed fixed Debian package "ckan" should not depend on "postgresql"

The debian package "ckan" with the two scripts "ckan-create-instance" and "ckan-instance-maintenance" depends against "postgresql". But "ckan-create-instance" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs.

Please add a flag "--without-local-db" to "ckan-create-instance" and remove the postgres dependancy from the debain package.

1304538095000000 1310134813000000
#1126 enhancement dread dread ckan-v1.4-sprint-7 closed fixed Exceptions arising from error page

I'm not completely clear what the use case is for loading the error page in this way, but somehow original_request is None and that creates an unnecessary exception with the logic refactor.

http://ckan.net/error/document?__cache=39020485
...
Module ckan.controllers.error:29 in document
<<          original_response = request.environ.get('pylons.original_response')
               # Bypass error template for API operations.
               if original_request.path.startswith('/api'):
                   return original_response.body
               # Otherwise, decorate original response with error template.
>>  if original_request.path.startswith('/api'):
AttributeError: 'NoneType' object has no attribute 'path'
1304586683000000 1304587078000000
#1127 CREP sebbacon closed fixed CREP0001: Formalise new feature discussion and definition using CREPs

Proposer: Seb Bacon
Seconder: Rufus Pollock

Abstract

When adding major new features to CKAN, a longer, more formal discussion will improve software design quality and documentation, better engage the wider community, and ensure the core team are up to date with latest developments.

I propose a formal process (CREP -- CKAN Revision and Enhancement Proposal) for making this happen.

The Problem

The current workflow for introducing major new features into CKAN is very informal, typically based around one person's great idea, which they've discussed with one or two other people in the team. The originator of the idea is typically the only person with access to all the input they've had through such discussions. Often, the only location of this information is in that person's head.

However, there is a lot of experience embodied in the CKAN community which should be drawn on before making large design decisions. This will lead to better software. Additionally, building consensus in the community around a proposal before implementation ensures positive community engagement and buy-in to new features, making them more likely to be a success.

We aren't great at documenting new features. Documentation after coding is complete is an unrewarding experience for most programmers. Requiring skeleton documentation before code is written is a good discipline that can form the basis of better documentation in the future (e.g. by a writer rather than a programmer).

Specification

Minor features don't require a CREP, and can just be entered in the issue tracking system as a bug or feature. As a rule of thumb, a feature is major if it will take more than a day to implement, or is likely to involve matters of opinion in its design.

A developer may decide that a CREP is too formal and long-winded. The decision to write a CREP is at at their discretion; however, new features MUST always be proposed via email, even if this is just a couple of sentences.

If a feature requires a CREP, the proposer should find a seconder for their idea. This sanity check step happens before a CREP is written to ensure at least the possibility of consensus on the CREP.

Next the proposer should write a CREP, starting by copying and pasting the template on the wiki into a new Trac ticket. This will be with a status of "new" and Type of "CREP". The proposer should notify the ckan-dev mailing list, and possibly the ckan-discuss list for less technical CREPs.

The draft can be discussed via email, verbally, or via the trac ticket. In any case, it is the proposer's responsibility to keep the CREP updated to reflect the current consensus.

Once consensus has been reached, the ticket should be marked with the "accepted" status and assigned to a CKAN release milestone.

When an accepted CREP has been implemented, it should be resolved as "fixed".

If no consensus can be reached on a draft CREP, or for some reason an accepted CREP doesn't get completed, it should be marked as or "wontfix".

If a completed CREP becomes obsolete, it should be marked as "invalid", with a note pointing to the obsoleting ticket(s)

Why do it this way

Given the distributed nature of the core team plus other volunteers, some kind of written procedure is necessary to ensure a fully documented and discussed proposal.

The idea of "Enhancement Proposals" which can be semi-formally proposed and discussed prior to implementation is common in the Open Source world (PEPs, DEPs, PLIPs, to name three).

Existing historic proposals exist, called CEPs. The proposed system is called CREP (CKAN revision or enhancement proposal) to disambiguate it from the legacy proposals, and from the delicious fungus Boletus Edulis.

Giving a formal structure to the proposal is useful as it gives the community a means to identify a CREP that's not had sufficient thought or discussion. An informal email thread can easily be lost and important questions (such as backwards compatibility) overlooked. The use of the proposed template empowers any community member to ask the proposer to expand on rationale, deliverables, etc.

The structure chosen is somewhere between Debian's and Plone's. It aims to give a structure to the debate, a clear start at documentation, and also prompt some thinking about implementation and timescales.

All this policy about structure should not be construed as mandatory. In particular, the later fields in the CREP template regarding Implementation Plan may be omitted if the author doesn't find them helpful.

Some projects (e.g. Debian) keep their enhancement proposals in a versioning repository; others (e.g. Plone) keep them in an issue tracking system. Trac is proposed for CKAN because we already use it for small feature proposals and for team planning. It seems unlikely that change tracking on an individual CREP will be useful; a CREP that changes sufficiently from its original form should probably be marked "obselete" and a new CREP started. Using an issue tracking system also means we can easily track CREPs by state.

Backwards Compatibility

Some [https://bitbucket.org/okfn/ceps/src/76b274888bcf/cep/ legacy enhancement proposals], called CEPs, have previously been started.

They are currently all marked as "active". Any which require discussion should be altered by the proposer to match the new CREP specification and submitted to trac. The original CEP should be updated with a banner at the top pointing a reader to the new CREP.

Any that are now obselete should be clearly marked as such in a banner at the top, pointing a reader to the trac for new CREPs.

Implementation plan

Deliverables

  • This CREP, agreed
  • Support for proposed statuses in Trac
  • Canned reports for listing CREPs in Trac

Risks and mitigations

  • That this CREP is agreed, but rarely acted on. This risk can be mitigated by nominating a CREP champion in the community or core team, whose job it is to say "where's the CREP for that?" and generally own the quality of CREPS

Participants

Seb Bacon: as current Documentation Czar (May 2011), responsible for ensuring CREPs are up to date.

Progress

This document is the entire proposal.

1304601313000000 1305622850000000
#1128 task dread dread ckan-sprint-2011-10-28 closed fixed Upload Scotland gov data

Upload to ckan.net: https://sites.google.com/site/scotlandsdata/

1304712293000000 1310568028000000
#1129 CREP kindly ckan-v1.5 closed fixed CREP0002: Moderated Edits

Proposer: David Raznick

Abstract.

We are trying to achieve these goals.

  • To get people involved with making edits to CKAN metadata.
  • To have an ownership model as to who can moderate and validate these changes
  • To not put too huge a burden on these owners.

In order to achieve this, a feature which lets anyone edit a package but only let the moderator/owner accept it. The moderator should be able to look at a list of changes and accept the ones that

This cep is not about 'if' we need such a feature, it is about 'how' we go about implementing it. Another cep may needed for the 'if' case.

The Problem

We need the following to be possible.

  • Storing revision of objects that are not the current active one.
  • A way of the user viewing past revisions.
  • Accessing not only the history of a particular object but also of related objects at that time. i.e If a resource related to a package changes we need a way to see this when looking at the package.
  • A robust way of doing this in the face of database schema changes.
  • Make sure database queries are quick.

Solutions.

  1. Store the whole dictization of the package and all its related objects every time you change anything in its dictized representation and only save to the database proper if accepted.

Pros

  • Easy to implement, we already have a preview which makes the dictized form of a package without actually saving it. This will just need to be persisted in some way.
  • Fast retrieval.
  • Potential to store a branching revision tree of changes.

Cons

  • No easy way to remake the dictized packages historically or if there is an there a change in the way we represent packages, i.e schema changes.
  • Will only work for the particular objects we decide to store these changes for.
  • Stores a lot of repeated information
  1. Write specialized queries for every read of the database looking only at the revision tables.

This method requires there to be a change in the way we use VDM, so that we manage statefulness ourselves. We will need to add other states such as 'waiting for approval'.

Pros

  • No specialized storage required
  • Only need to change queries when schema changes
  • Can be made to work easily for other objects

Cons

  • Slower query time on read, as even looking at the last active package will need to do a fairly complicated query.

Implementation details.

1.

A new table with columns id, user, package_id, timestamp, revision_id, parent_id, dictized_package. revision_id should be null unless it is actually persisted to the database. parent_id is the id that this package_dict was changed from.

We could store only the diffs of the dictized_package as long as we assure that everything inside the json is stably sorted, this will make getting the historical data out slower.

Getting out the history of the dictized packages is an intensive task, as it will require replaying the whole history of all the changes and creating the dict for each change. This re-caching will need to be redone for every change we make to dictized representation of a package.

2.

Every normal packages read needs to look at the revision table to see the last accepted change in the dictized representation of the package. We also need to way to get what the dictized representation of the package was like at any point of its revision history. This querying is non-trivial in sql.

Participants

David Raznick to do it.

Progress.

Decided to go with option 2. However we will change the revisioning system to be like the schema attached. This gets rid of difficult querying problems caused by querying the revision tables by adding an end date, meaning you can do range queries.

The better and more normalized version of a revisioning system is outlined https://docs.google.com/drawings/d/1Y7nMgVsrs081Pame2RdbZHlCAlV33ddTZ8VAsab1j-0/edit?hl=en_GB&authkey=CJfd8vsB. We will be a step closer to that, with this change, but we will keep the current vdm more or less, intact.

1304851498000000 1325268100000000
#1131 defect dread dread ckan-v1.4-sprint-7 closed fixed Search param validation exception not caught

Example request:

http://nl.ckan.net/api/2/search/package?q=delft&order_by=&offset=&limit=&tags=

Gives 500 error:

<type 'exceptions.ValueError'>: invalid literal for int() with base 10: ''
1304942023000000 1305537897000000
#1132 defect johnlawrenceaspden ckan-backlog closed invalid test_authz doesn't run

Trying to run the tests in test_authz.py with

$ nosetests --ckan ckan/tests/functional/test_authz.py results in no tests being run:


Ran 0 tests in 0.840s

OK (SKIP=3)

1304966923000000 1307352675000000
#1133 defect johnlawrenceaspden closed worksforme command line rights manipulation doesn't work

It appears that the command

$ paster rights add russianfan admin warandpeace

has no effect, even though

$ paster rights remove russianfan admin warandpeace

works fine. This may be specific to something I've done, could someone confirm?

If it's the case more generally, then I'm assuming this behaviour is untested? Tests should probably be added.

1305054948000000 1324057072000000
#1138 enhancement johnlawrenceaspden closed invalid minor navigations behave inconsistently

For Authorization Groups, if you have admin privileges you see view, edit and authz tabs, and if you don't have the necessary privileges you only see the view tab.

For Packages, you see all tabs whatever your permissions, so there's a link you can click on which will redirect you to the login page.

1305279888000000 1316965357000000
#1139 enhancement lucychambers lucychambers ckan-backlog closed fixed Create CKAN Theme Gallery

Take screenshots of existing ckan instances esp those mentioned <http://wiki.ckan.net/Theming> and put on flickr in ckan or ckan-theme group so we can create a gallery ... (both to illustrate theming but also to show ckan instances that are around -- could add to http://wiki.ckan.net/Instances)

1305645859000000 1306941356000000
#1140 defect dread ckan-v1.5 closed fixed Adding the package to the group is not search indexed

To reproduce:

  1. paster db init
  2. paster create-test-data
  3. In Web UI: create new group 'tilo', which includes package 'annakarenina'
  4. curl http://localhost:5000/api/search/package?groups=tilo results in 0 results (WRONG)
  5. paster search-index rebuild
  6. curl http://localhost:5000/api/search/package?groups=tilo results in 1 result
1305718290000000 1312537257000000
#1141 CREP johnglover ckan-backlog closed fixed [super] Moderated Edits User Interface

Proposer: John Glover
Seconder: James Gardner

Abstract

We are trying to achieve these goals:

  • To get people involved with making edits to CKAN metadata.
  • To have an ownership model as to who can moderate and validate these changes
  • To not put too huge a burden on these owners.

This feature allows anyone to edit a package and create a new revision, but requires an owner/moderator to approve a revision before it is are made "official".

There have been a lot of discussions around the revisioning system side of this ticket (CREP 0002) and I think these are now largely resolved. We now want to discuss the user interface.

The Problem

We require the following functionality:

  • Allow a group of changes to be stored as a new revision.
  • Allow a linear stack of "community" revisions.
  • Provide a way for the editor and moderator to compare previous revisions to the current one.
  • When a moderator approves a change it creates a new revision flagged "moderated" (this is analogous to a merge commit)
  • Provide a way for the editor and moderator comment on revisions if necessary.

Extra features:

  • Need a way to summarise the changes (as part of the preview perhaps)
  • Sysadmin needs to purge a revision completely

Specification

UI/UX

UI Mockup:

Revisions:

  • Revisions are per package rather than per field.
  • Internally CKAN has separate revisions for resources, extras and package metadata. From a user's point of view this could be confusing to expose, so everything that they see on a package form when they hit save is a single revision.

On the Edit page:

  • We have a panel on the right, listing all the revisions with the current moderated one selected. Moderated revisions are highligted in some way (red and bold?).
  • The values displayed in the form are by default populated from the latest revision (whether community or moderated)
  • Under each field is a "shadow", showing the value of the field in the revision selected in the panel, if it is different from the value in the field. By default the shadow values are populated from the latest moderated revision which is the one selected in the revision panel by default too.
  • When you change the value of a field, a shadow may appear or disappear accordingly. If they disappear a box saying that they are the same replaces it
  • If you want to edit values from a previous revision, you first select that revision to get the shadows populated. There is a button named "Replace fields with values from this revision" under the revision list. You click this, a warning pops up and then you say "Yes". You then select the moderated revision again.
  • We also allow package comments the same way as the todo extension works at the moment. Additionally, we need to be able to differentiate between what the moderator wrote and what a community member wrote, and so we may need to make a small change to the todo extension to facilitate this.
  • In addition to package comments, each revision will have a revision log (analogous to a commit message).

Technical Details

  • This CREP will result in a new CKAN extension.
  • It depends heavily on the new revisioning system (CREP0002), some of the details of which are yet to be finalised.
  • This CREP therefore requires working closely with David Raznick to come up with an API that the UI AJAX calls can use.
  • We will then use suitable test data to mimic these API calls until CREP0002 is ready.

Why do it this way

This hopefully provides a clear and consistent mechanism allowing both a community member to make new revisions and a moderator to view and approve revisions, with largely the same UI/UX.

Implementation plan

Deliverables

A new CKAN extension, consisting of:

  • Code: Python, HTML, CSS, Javascript
  • Unit tests
  • Localization
  • Documentation

Participants

John Glover to do it.

Progress

John has implemented the bulk of this UI. Just some things to tidy up before it is complete:

  • Genshi stream filters to be updated with CKAN 1.5 / 1.5.1 templates
  • history_ajax / read_ajax to be replaced with calls to Action API (or Util REST API)

I've split these two off into a new ticket #1604.

Related Progress

The Todo extension is written and available at: https://bitbucket.org/johnglover/ckanext-todo.

In the section 'The Problem', under extra features, we mention a need for the sysadmin to be able to purge a revision already. This is already done.

See also

#1129 Backend work

1305721003000000 1325352507000000
#1142 enhancement annapowellsmith rgrp ckan-v1.5 closed fixed [super] Major Overhaul and Extension of CKAN Documentation

Child tickets:

  • #1041 Start Using the CKAN Wiki for Tutorial-style documentation
  • #1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927

  • CKAN 1-page overview (for enterprise and for data hackers)
  • Administrator's Guide (including install)
  • Extensions Guide
  • Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: http://wiki.ckan.net/Documentation_Plans

Minor Items

1305727452000000 1313775665000000
#1143 enhancement dread dread ckan-v1.5-sprint-2 closed fixed Improve stats page
  • Ensure we don't include deleted packages in the stats
  • Some visual improvements:
    • Number of packages:
      • fix x axis to start at first revision
      • fix y axis to start at zero
    • Fix problem with legend to 'Revisions to packages' graph.
    • Hide sidebar in the template, so it isn't right-cropped in the DGU theme
  • Add some testing of the stats lib - results of basic stats
1305734836000000 1307024681000000
#1146 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed make logic layer control its own state

Logic layer should not use any vdm defined state and should manage it itself.

1305829117000000 1307957527000000
#1147 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed Add expired_id to all revision tables.

Revision tables need expired_id to make querying history AND pending changes more efficient.

This involves making a session extension and a large table migration.

1305839833000000 1307957556000000
#1148 refactor kindly kindly ckan-v1.5-sprint-1 closed fixed test speed improvements and cleanup

The tests have been running slower recently and need fixing. They also could do with a bit more consistency to them.

1305969223000000 1305969925000000
#1149 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed Change domain object modification plugin to use Session extension.

This should make it more efficient as it currently does a lot of repeating work. i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1.

1305969863000000 1306090663000000
#1150 defect johnglover timmcnamara ckan-sprint-2011-10-28 closed fixed Non-ASCII chars prevent data preview

Characters outside of ASCII range are not supported within data previews.

Steps to reproduce:

  1. Visit http://ckan.net/package/kivele2010
  2. Click on [preview] for any of the resources
1306019914000000 1311774141000000
#1151 enhancement rgrp timmcnamara ckan-sprint-2012-03-05 closed wontfix Preview for geographic data should be a map

Data viewer / previewer for a resource that has a KML file, or others, such as GeoRSS and GeoJSON, we should provide a map.

  • Support for KML files: #1458

This is non-trivial for external files as we need a way to jsonify. For files stored locally this is more of a recline issue (and will require a bit of work to either guess columns or allow user to specify them).

1306043217000000 1330908188000000
#1153 enhancement lucychambers lucychambers ckan-backlog closed fixed Update CKAN wiki front page

Update CKAN wiki front page - a la OpenSpending?: http://wiki.openspending.org/Main_Page

Sections should relate to different types of people using the site:

Developers, Users etc..

1306155211000000 1306941386000000
#1154 enhancement johnglover nils.toedtmann ckan-sprint-2011-10-28 closed fixed Make ckan robust against solr failure

According to pudo, a ckan with activated solr extension throws a 5xx when solr is unreachable. Instead, it should behave more like a ckan without ckanext-solr when this happens.

1306254472000000 1314287519000000
#1155 enhancement pudo pudo pdeu-1 closed fixed Harvester for data.london.gov.uk

Write a harvester for data.london.gov.uk to import catalogue metadata into PDEU. API (or at least documentation) is available at: http://sourceforge.net/projects/londondatastore/files/

1306337318000000 1306773174000000
#1156 enhancement pudo pudo pdeu-1 closed fixed Scraping harvesters for Paris and Vienna Catalogues

Import metadata from both sources into PDEU via the Harvesting framework but by scraping their respective catalogue pages.

1306337428000000 1306855111000000
#1157 enhancement pudo pudo pdeu-1 closed fixed A simple theme for publicdata.eu, CKAN

Change the visual style of CKAN to be more like these sites:

This does not include major UX work.

1306337573000000 1309804029000000
#1158 enhancement pudo pudo pdeu-1 closed fixed Put PDEU into read-only mode and remove unneeded functions

PDEU should be in a mode where:

  • Users cannot edit or create packages
  • Users cannot sign up on their own
  • AuthorizationGroups? and RevisionHistory? is completely gone from the UI
  • Groups have been repurposed as Subjects
1306337786000000 1309804060000000
#1159 enhancement pudo pudo pdeu-1 closed fixed Add RDFa to CKAN package pages

RDFa can be used as a simple way to expose linked data or at least sameAs the API version of the data but we need to make sure we do not expose anything different from the "official" representation.

1306337907000000 1307615133000000
#1160 enhancement amercader amercader pdeu-1 closed wontfix Havest CSW servers on PDEU

It could be interesting for the publicdata.eu project to add some CSW datasources. Thanks to the INSPIRE directive, there are quite a few national catalogs of geodata that could be worth harvesting. Work on this should include:

  • Compose Endpoint List (1d)
  • Harvest on staging system and evaluate which sources have useful material (2d)

First point is being performed in this GDoc: https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&authkey=CNu4hsEB&hl=en_US#gid=0

1306407271000000 1309804038000000
#1161 requirement pudo amercader pdeu-1 closed duplicate Build a simplified theme for PDEU

We need to offer a strongly simplified version, read-only of CKAN under publicdata.eu, with a focus on its role as search engine instead of a data catalogue.

This ticket relates to work on the PDEU theme only!

1306407835000000 1306408026000000
#1162 enhancement amercader amercader pdeu-1 closed fixed Define a common extra for storing the geographic extent

We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city).

Relevant discussion: http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html

1306408704000000 1308647187000000
#1164 enhancement amercader amercader pdeu-1 closed fixed Cloropleth Map of European Data Availability for PDEU

A nice map in the homepage showing the availability of data across Europe

1306408824000000 1308647224000000
#1166 enhancement amercader amercader ckan-sprint-2011-10-10 closed fixed Less stringent filters for displaying the WMS preview

Right now, for the "View available WMS layers" link to be displayed, the package needs to have an "INSPIRE=True" extra, a "resource-type=service" extra and a resource of type "WMS".

This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough.

(The repository for this is ckanext-spatial, but it's not on the list)

1306422646000000 1317381932000000
#1167 task nils.toedtmann closed fixed Create a standard CKAN system image for Amazon EC2 (AMI)

We should create a public AMI with CKAN pre-installed and configured such that users can easily create their own EC2 machine with a running CKAN to play with.

There are three phases:

  1. [nils] Deploy an empty EC2 instance to become the CKAN image master instance
  2. Install a CKAN and give it a standard configuration.
  3. [nils] Create a AMI from the CKAN image master instance and publish it.

I am happy to do first and last. Who is installing and configuring CKAN?

Unfortunately AMIs are specific to region, architecture and storage type. We cannot maintain too many images, so a number of choices have to be made:

  • Which distribution/version? Ubuntu 10.04 LTS
  • Which architecture/instance-type? I suggest 64-bit/t1.micro
  • Which region? I suggest us-east-1 and maybe eu-west-1
  • Which storage type? EBS (way easier to make an AMI from than instance-store)
  • Install CKAN from deb packages via mercurial/virtualenv? I assume the latter because the AMI is targeted to developers?

1 2

1306423016000000 1311178516000000
#1169 enhancement dread dread ckan-v1.5-sprint-2 closed fixed Link to csv/json dumps on ckan.net 1306749768000000 1306754833000000
#1170 defect dread dread ckan-v1.5-sprint-2 closed fixed Production deployment docs out of date

Need to cover pip-requirements.txt on different branches in the doc/deployment.rst.

Also could do with tidying up upgrade.txt into this document too.

1306864171000000 1306864422000000
#1172 defect dread thejimmyg ckan-sprint-2011-10-28 closed fixed Remove all try: except: blocks that don't re-raise the original exception

The current codebase has one or two try: except; blocks that don't catch specific exceptions. Under no circumstances should any broad try: except: blocks be allowed unless the exceptions they catch are immediately re-raised. Uncaught exceptions are wasting us quite a lot of time when trying to track down problems.

1306926049000000 1314118883000000
#1173 enhancement amercader amercader pdeu-1 closed fixed Offer a DCat representation for packages in the API

/api/rest/package/foo.rdf should return a DCat representation of the package. To create it, we will use the functions in ckanext-rdf.

1306939639000000 1307615271000000
#1174 enhancement pudo pudo pdeu-1 closed wontfix API Representation Registry

As CKAN grows, more and more optional representations of packages will become used. Besides RDF (which is the motivation for this ticket), support for DSPL, home-grown XML, or CSV listings is imaginable.

To properly support this CKAN should have an extensible representation registry that can be extended when new output types become available (and without changing the API in the process). This needs to integrate in two places:

  • We need to add support for file format extensions to the package handler and REST API.
  • We need to add HTTP content type negotiation (Accept headers).

To support this we should have a registry with two registers:

  • to map format extensions to mime types (e.g. "json" -> "application/json")
  • to map mime types and entity types to converter functions (e.g. 8"application/json", ckan.model.Package) -> func(obj, mime_type) )

This should be set up on load_environment so that IConfigurer plugins can feed into it.

The registry should then first be added to _finish in the REST API (which needs to be refactored to be passed the {format} part of the URL if one is given. Based on the format part and HTTP headers, an appropriate representation can be generated by the registry and then be returned to the user.

Forwarding of requests to the regular WUI controllers with Accept headers set or a format specified can be implemented in a separate effort.

1307050992000000 1307615200000000
#1175 defect dread [email protected] closed invalid Stats extension not working

Hi, I get a 500, Internal server error when I enable ckanext-stats. Flavio

1307350823000000 1325355170000000
#1176 defect thejimmyg dread ckan-sprint-2011-10-28 closed fixed unicode_body exception

We seem to occasionally get this exception on ckan.net.

User agents causing this include:

  • Alexa ia_archiver (3/6/11)
  • Maxthon browser (17/5/11)
Module ckan.controllers.error:32 in document
<<              return original_response.body
               # Otherwise, decorate original response with error template.
               c.content = literal(original_response.unicode_body) or cgi.escape(request.GET.get('message', ''))
               c.prefix=request.environ.get('SCRIPT_NAME', ''),
               c.code=cgi.escape(request.GET.get('code', str(original_response.status_int))),
>>  c.content = literal(original_response.unicode_body) or cgi.escape(request.GET.get('message', ''))
AttributeError: 'NoneType' object has no attribute 'unicode_body'
1307351562000000 1314273012000000
#1177 defect fccoelho closed fixed routes.util.GenerationException

Hello,

I have done a recent update to mu ckan install, and I am now getting this error for all my packages:

URL: http://ckan.emap.fgv.br/package/dengue-net
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
          
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
              
               response = self.app(environ, start_response)
              
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<         
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
              
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
          
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']   
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
              
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                  
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
          
           def _inspect_call(self, func):
>>  return func(**args)
Module ?:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__("logging").getLogger("proxy_cache")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
      
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.package:208 in read
<<          #render the package
               PackageSaver().render_package(c.pkg)
               return render('package/read.html')
      
           def comments(self, id):
>>  return render('package/read.html')
Module ckan.lib.base:74 in render
<<     
           return cached_template(template_name, render_template, cache_key=cache_key,
                                  cache_type=cache_type, cache_expire=cache_expire)
                                  #, ns_options=('method'), method=method)
>>  cache_type=cache_type, cache_expire=cache_expire)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:63 in render_template
<<              stream = item.filter(stream)
              
               return literal(stream.render(method=method, encoding=None))
          
           if 'Pragma' in response.headers:
>>  return literal(stream.render(method=method, encoding=None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
      
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out=out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:339 in __call__
<<          for filter_ in self.filters:
                   stream = filter_(stream)
               for kind, data, pos in stream:
                   cached = cache_get((kind, data))
                   if cached is not None:
>>  for kind, data, pos in stream:
Module genshi.output:670 in __call__
<<          _gen_prefix = _gen_prefix().next
      
               for kind, data, pos in stream:
                   output = cache_get((kind, data))
                   if output is not None:
>>  for kind, data, pos in stream:
Module genshi.output:771 in __call__
<<          push_text = textbuf.append
               pop_text = textbuf.pop
               for kind, data, pos in chain(stream, [(None, None, None)]):
      
                   if kind is TEXT:
>>  for kind, data, pos in chain(stream, [(None, None, None)]):
Module genshi.output:586 in __call__
<<      def __call__(self, stream):
               prev = (None, None, None)
               for ev in stream:
                   if prev[0] is START:
                       if ev[0] is END:
>>  for ev in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
      
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.markup:378 in _match
<<                      for event in self._match(self._flatten(template, ctxt,
                                                                  **vars),
                                                    ctxt, start=idx + 1, **vars):
                               yield event
>>  ctxt, start=idx + 1, **vars):
Module genshi.template.markup:327 in _match
<<                      break
      
               for event in stream:
      
                   # We (currently) only care about start and end events for matching
>>  for event in stream:
Module genshi.template.base:565 in _flatten
<<                  elif kind is EXPR:
                           result = _eval_expr(data, ctxt, vars)
                           if result is not None:
                               # First check for a string, otherwise the iterable test
>>  result = _eval_expr(data, ctxt, vars)
Module genshi.template.base:277 in _eval_expr
<<      if vars:
               ctxt.push(vars)
           retval = expr.evaluate(ctxt)
           if vars:
               ctxt.pop()
>>  retval = expr.evaluate(ctxt)
Module genshi.template.eval:178 in evaluate
<<          __traceback_hide__ = 'before_and_this'
               _globals = self._globals(data)
               return eval(self.code, _globals, {'__data__': data})
>>  return eval(self.code, _globals, {'__data__': data})
Module ?:16 in <Expression u"h.subnav_link(c, h.icon('comments') + _('Comments & Questions'), controller='package', action='comments', id=c.pkg.name)">
<<        </li>
             <li py:if="g.has_commenting">
                 ${h.subnav_link(c, h.icon('comments') + _('Comments &amp; Questions'), controller='package', action='comments', id=c.pkg.name)}
             </li>
             <li>${h.subnav_link(c, h.icon('page_white_stack') + _('History'), controller='package', action='history', id=c.pkg.name)}</li>
>>  ${h.subnav_link(c, h.icon('comments') + _('Comments &amp; Questions'), controller='package', action='comments', id=c.pkg.name)}
Module ckan.lib.helpers:126 in subnav_link
<<      return link_to(
               text,
               url_for(action=action, **kwargs),
               class_=('active' if c.action == action else '')
           )
>>  url_for(action=action, **kwargs),
Module routes.util:280 in url_for
<<          raise GenerationException(
                   "url_for could not generate URL. Called with args: %s %s" % \
                   (args, kargs))
           return url
>>  (args, kargs))
GenerationException: url_for could not generate URL. Called with args: () {'action': 'comments', 'controller': 'package', 'id': 'dengue-net'}
CGI Variables
AUTH_TYPE 	'cookie'
DOCUMENT_ROOT 	'/etc/apache2/htdocs'
GATEWAY_INTERFACE 	'CGI/1.1'
HTTP_ACCEPT 	'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
HTTP_ACCEPT_CHARSET 	'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING 	'gzip, deflate'
HTTP_ACCEPT_LANGUAGE 	'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3'
HTTP_CONNECTION 	'keep-alive'
HTTP_COOKIE 	'__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; __utmc=4669863'
HTTP_HOST 	'ckan.emap.fgv.br'
HTTP_KEEP_ALIVE 	'115'
HTTP_REFERER 	'http://ckan.emap.fgv.br/'
HTTP_USER_AGENT 	'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1'
PATH_INFO 	'/package/dengue-net'
PATH_TRANSLATED 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/package/dengue-net'
REMOTE_ADDR 	'10.250.48.110'
REMOTE_PORT 	'47624'
REMOTE_USER 	u'flavio'
REMOTE_USER_DATA 	'userid_type:unicode'
REMOTE_USER_TOKENS 	['']
REQUEST_METHOD 	'GET'
REQUEST_URI 	'/package/dengue-net'
SCRIPT_FILENAME 	'/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py'
SERVER_ADDR 	'10.252.2.60'
SERVER_ADMIN 	'[no address given]'
SERVER_NAME 	'ckan.emap.fgv.br'
SERVER_PORT 	'80'
SERVER_PROTOCOL 	'HTTP/1.1'
SERVER_SIGNATURE 	'<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n'
SERVER_SOFTWARE 	'Apache/2.2.16 (Debian)'
WSGI Variables
application 	<beaker.middleware.CacheMiddleware object at 0x7f051c802690>
beaker.cache 	<beaker.cache.CacheManager object at 0x7f051c802750>
beaker.get_session 	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f051c8026d0>>
beaker.session 	{'locale': u'pt_BR', '_accessed_time': 1307358102.6266389, '_creation_time': 1302605470.0245121}
mod_wsgi.application_group 	'dck093|'
mod_wsgi.callable_object 	'application'
mod_wsgi.handler_script 	''
mod_wsgi.input_chunked 	'0'
mod_wsgi.listener_host 	''
mod_wsgi.listener_port 	'80'
mod_wsgi.process_group 	'ckan'
mod_wsgi.request_handler 	'wsgi-script'
mod_wsgi.script_reloading 	'1'
mod_wsgi.version 	(3, 3)
paste.cookies 	(<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio'>, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode";... __utmc=4669863')
paste.registry 	<paste.registry.Registry object at 0x7f051d7e5710>
paste.throw_errors 	True
pylons.action_method 	<bound method PackageController.read of <ckan.controllers.package.PackageController object at 0x7f051d7e5b10>>
pylons.controller 	<ckan.controllers.package.PackageController object at 0x7f051d7e5b10>
pylons.environ_config 	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons 	<pylons.util.PylonsContext object at 0x7f051d7e5a50>
pylons.routes_dict 	{'action': u'read', 'controller': u'package', 'id': u'dengue-net'}
repoze.who.identity 	<repoze.who identity (hidden, dict-like) at 139659946193344>
repoze.who.logger 	<logging.Logger instance at 0x7f051c801128>
repoze.who.plugins 	{'openid': <OpenIdIdentificationPlugin 139659931643152>, 'friendlyform': <FriendlyFormPlugin 139659929725584>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f051c9e7610>, 'auth_tkt': <AuthTktCookiePlugin 139659931643280>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f051c9dcfd0>}
routes.route 	<routes.route.Route object at 0x7f051c69fd50>
routes.url 	<routes.util.URLGenerator object at 0x7f051d7e57d0>
webob._parsed_cookies 	({'ckan': 'd4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06', '__utmz': '4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'auth_tkt': 'c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode', 'ckan_display_name': 'Fl\xc3\xa1vio Code\xc3\xa7o Coelho', 'ckan_apikey': 'c139718d-918f-4a2b-b219-e33cb05cbe23', 'ckan_user': 'flavio', '__utma': '4669863.1260802706.1295364409.1306935693.1306960483.86', '__utmc': '4669863', 'SESS61faf7f7d2406929b2f9eb4cbfff17d2': 'a3306f5d1194cf0d8a51dbf7b42c1259'}, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"< wbr>; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!... __utmc=4669863')
webob._parsed_query_vars 	(GET([]), '')
webob.adhoc_attrs 	{'language': 'en-us'}
wsgi process 	'Multi process AND threads (?)'
wsgi.file_wrapper 	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f051cc1ad50>
wsgi.version 	(1, 1)
wsgiorg.routing_args 	(<routes.util.URLGenerator object at 0x7f051d7e57d0>, {'action': u'read', 'controller': u'package', 'id': u'dengue-net'})
1307358586000000 1307374897000000
#1178 defect fccoelho closed fixed Local storage issue with ckanext-storage

I have followed the instruction for installation a configuration of ckanext-storage, but I am getting this error when I try to access the /storage/upload url:

URL: http://ckan.emap.fgv.br/storage/upload Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:152 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:130 in call << environSCRIPT_NAME? = environSCRIPT_NAME?[:-1]

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:124 in call << return % paste.registry.restorer.get_request_id(environ)?

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

controller = self.resolve(environ, start_response)

Module pylons.wsgiapp:263 in resolve << if self.log_debug:

log.debug("Resolved URL to controller: %r", controller)

return self.find_controller(controller)

def find_controller(self, controller):

return self.find_controller(controller)

Module ckan.config.environment:39 in find_controller << # Check to see if its a dotted name

if '.' in controller or ':' in controller:

mycontroller = pkg_resources.EntryPoint?.parse('x=%s' % controller).load(False) self.controller_classes[controller] = mycontroller return mycontroller

mycontroller = pkg_resources.EntryPoint?.parse('x=%s' % controller).load(False)

Module pkg_resources:1954 in load << def load(self, require=True, env=None, installer=None):

if require: self.require(env, installer) entry = import(self.module_name, globals(),globals(), __name__?) for attr in self.attrs:

try:

entry = import(self.module_name, globals(),globals(), __name__?)

Module ?:87 in <module> << class StorageAPIController(BaseController?):

ofs = get_ofs()

class StorageAPIController(BaseController?):

Module ?:88 in StorageAPIController << class StorageAPIController(BaseController?):

ofs = get_ofs()

@jsonpify

ofs = get_ofs()

Module ?:69 in get_ofs << continue

kw[k[4:]] = v

ofs = get_impl(storage_backend)(kw) return ofs

ofs = get_impl(storage_backend)(kw)

Module ofs.local.pairtreestore:26 in init << self.hashing_type = hashing_type

self.shorty_length = shorty_length self._open_store()

def _open_store(self):

self._open_store()

Module ofs.local.pairtreestore:30 in _open_store << def _open_store(self):

if self.hashing_type:

self._store = PairtreeStorageClient?(self.uri_base, self.storage_dir, shorty_length=self.shorty_length, hashing_type=self.hashing_type)

else:

self._store = PairtreeStorageClient?(self.uri_base, self.storage_dir, shorty_length=shorty_length)

self._store = PairtreeStorageClient?(self.uri_base, self.storage_dir, shorty_length=self.shorty_length, hashing_type=self.hashing_type)

Module pairtree.pairtree_client:91 in init << self._decode = re.compile(r"\(..)", re.U)

self._init_store()

def char2hex(self, m):

self._init_store()

Module pairtree.pairtree_client:264 in _init_store << self.uri_base = prefix

else:

raise NotAPairtreeStoreException

if not os.path.isdir(self.store_dir):

raise NotAPairtreeStoreException

NotAPairtreeStoreException: CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'utma=4669863.1260802706.1295364409.1306935693.1306960483.86; utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; utmc=4669863; hide_welcome_message=1' HTTP_HOST 'ckan.emap.fgv.br' HTTP_KEEP_ALIVE '115' HTTP_USER_AGENT 'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1' PATH_INFO '/storage/upload' PATH_TRANSLATED '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/storage/upload' REMOTE_ADDR '10.250.48.110' REMOTE_PORT '57426' REMOTE_USER u'flavio' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [] REQUEST_METHOD 'GET' REQUEST_URI '/storage/upload' SCRIPT_FILENAME '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py' SERVER_ADDR '10.252.2.60' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.emap.fgv.br' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.16 (Debian)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7fc2999a6cd0> beaker.cache <beaker.cache.CacheManager? object at 0x7fc2999a6d90> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7fc2999a6d10>> beaker.session {'locale': u'pt_BR', '_accessed_time': 1307373458.4790759, '_creation_time': 1302605470.0245121} mod_wsgi.application_group 'dck093|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio' hide_welcome_message='1'>, 'utma=4669863.1260802706.1295364409.1306935693.1306960483.86; utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid _type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221fla...lcome_message=1') paste.registry <paste.registry.Registry object at 0x7fc299de4f50> paste.throw_errors True pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7fc299e27d50> pylons.routes_dict {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'} repoze.who.identity <repoze.who identity (hidden, dict-like) at 140473773131872> repoze.who.logger <logging.Logger instance at 0x7fc2999a3b48> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 140473779397008>, 'friendlyform': <FriendlyFormPlugin? 140473777417040>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7fc299b93a50>, 'auth_tkt': <AuthTktCookiePlugin? 140473779397136>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7fc299b934d0>} routes.route <routes.route.Route object at 0x7fc299864090> routes.url <routes.util.URLGenerator object at 0x7fc299e27ed0> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fc299dae4e0> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7fc299e27ed0>, {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'}) Reply Reply to all Forward Reply to all [email protected] to fccoelho show details 10:17 (2 hours ago)

  • Show quoted text -

CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'utma=4669863.1260802706.1295364409.1306935693.1306960483.86; utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; utmc=4669863; hide_welcome_message=1' HTTP_HOST 'ckan.emap.fgv.br' HTTP_KEEP_ALIVE '115' HTTP_USER_AGENT 'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1' PATH_INFO '/storage/upload' PATH_TRANSLATED '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/storage/upload' REMOTE_ADDR '10.250.48.110' REMOTE_PORT '57429' REMOTE_USER u'flavio' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [] REQUEST_METHOD 'GET' REQUEST_URI '/storage/upload' SCRIPT_FILENAME '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py' SERVER_ADDR '10.252.2.60' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.emap.fgv.br' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '<address>Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.16 (Debian)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7fc2999a6cd0> beaker.cache <beaker.cache.CacheManager? object at 0x7fc2999a6d90> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7fc2999a6d10>> beaker.session {'locale': u'pt_BR', '_accessed_time': 1307373466.085686, '_creation_time': 1302605470.0245121} mod_wsgi.application_group 'dck093|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio' hide_welcome_message='1'>, 'utma=4669863.1260802706.1295364409.1306935693.1306960483.86; utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid _type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221fla...lcome_message=1') paste.registry <paste.registry.Registry object at 0x7fc29a9da850> paste.throw_errors True pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7fc29a9ef950> pylons.routes_dict {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'} repoze.who.identity <repoze.who identity (hidden, dict-like) at 140473790084352> repoze.who.logger <logging.Logger instance at 0x7fc2999a3b48> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 140473779397008>, 'friendlyform': <FriendlyFormPlugin? 140473777417040>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7fc299b93a50>, 'auth_tkt': <AuthTktCookiePlugin? 140473779397136>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7fc299b934d0>} routes.route <routes.route.Route object at 0x7fc299864090> routes.url <routes.util.URLGenerator object at 0x7fc29a9efcd0> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fc29a9d7d50> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7fc29a9efcd0>, {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'})

1307373810000000 1308045351000000
#1180 defect dread dread ckan-v1.5-sprint-3 closed fixed Links in markdown can be badly formed

User can insert bad anchor tags into the User-About and Package-Notes fields and when you view them (web interface) it causes a 500 error.

Need to improve filtering for anchors in markdown.

<a href="http://xxxsex.com>nasty/website

Also check this related exception:

Module ckan.controllers.user:59 in read
<<          c.is_myself = user.name == c.user
               c.api_key = user.apikey
               c.about_formatted = self._format_about(user.about)
               revisions_q = model.Session.query(model.Revision
                       ).filter_by(author=user.name)
>>  c.about_formatted = self._format_about(user.about)
Module ckan.controllers.user:167 in _format_about
<<      def _format_about(self, about):
               about_formatted = ckan.misc.MarkdownFormat().to_html(about)
               return genshi.HTML(about_formatted) 
       
           def _get_form_password(self):
>>  return genshi.HTML(about_formatted)
WebApp Error: <class 'genshi.input.ParseError'>: junk characters in start tag: u'\u201dhttp://www.settingu': line 1, column 3
1307469037000000 1307544223000000
#1181 defect dread dread closed fixed Link spam vulnerability in Notes and User-About fields

When viewing a user and a package, the about/notes fields contain Markdown, which may have links. These should have rel="nofollow" to discourage link spam.

1307523160000000 1307532816000000
#1183 defect johnglover nickstenning ckan-sprint-2011-10-28 closed fixed Downloads "Preview" button doesn't preview.

The "Preview" button is a nice idea, but it doesn't seem to actually "preview" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file.

This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of application/x-some-junk-here in order to force a download.

1307701669000000 1311771069000000
#1186 enhancement dread closed fixed Password reset facility

You can register a user with password and (optional) email address. But if you forget the password you can't then log in again. We need a password reset facility that sends and email with a new password.

1308142766000000 1310556519000000
#1187 enhancement dread dread ckan-v1.5-sprint-3 closed fixed Stop spam users with links in the 'about' field

When you edit a user, you should be stopped from putting links in the 'about' field, because this is a common tactic by spammers.

1308142999000000 1308143605000000
#1189 enhancement timmcnamara closed invalid Spideroak support in CKAN Storage

Spideroak is a bulk storage platform by a company that releases quite a lot of free software, has an extensive developer API with pricing of $10/100GB/month.

The data store is optimised for bulk data storage and retrieval. This is the kind of use case that CKAN packages require. We don't need low latency. We are after low cost, high reliability solution.

1308187104000000 1323173227000000
#1190 enhancement rgrp rgrp ckan-v1.5 closed fixed [super] CREP 0004 Data API and Data Processing System

For some time (e.g. 1y+!) we have known that we want to integrate some kind of datastore / data processing system with CKAN. We've had a CREP in progress on this for some months (may copy that here at some point):

http://wiki.ckan.org/CEP0004

We can distinguish 3 modules that are needed:

  1. "Webstore": A datastore with dataapi - #1208

Suggestion is this would be sqlite based with a simple sql based API. http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query}

  1. Automated conversion of suitable resources into datastore upon resource creation so that e.g. they are accessible via the API. #1398
  1. A data processing system which utilizes this datastore. One could

get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) - #1432

More info

1308227611000000 1323270522000000
#1191 defect dread dread ckan-v1.5-sprint-3 closed fixed Unicode in user password gives 500 error on user registration

Create/register a user with a unicode character in the password. It creates the user, but the redirect to the user page doesn't work and results in 500 error.

1308312895000000 1308650930000000
#1192 task annapowellsmith annapowellsmith ckan-sprint-2011-10-28 closed fixed Convert CKAN Sphinx docs into admin/reference manual

As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

  • An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install.
  • A Reference Manual which is the primary source of reference for CKAN software - this includes API docs.

The current chapters of the Sphinx docs should be moved as follows:

  • index.rst - copy some stuff from README.txt, keep short
  • README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst
  • [NEW] install.rst - new file on installation
  • [NEW] theming.rst - move over from wiki
  • api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
  • i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
  • design.rst - REMOVE (can copy some over if you can be bothered to http://wiki.ckan.net/Vision (all philosophical stuff should go!))
  • loaders.rst: move to wiki.ckan.net/Using_Loaders
  • feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
  • importer.rst: remove (don't even copy)
  • forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) http://wiki.ckan.net/Customizing_Forms
  • forms-integration.rst: remove
  • model.rst: leave but move to reference (only for core devs)
  • load-testing.rst -> move to http://wiki.ckan.net/Load_Testing
  • distributed.rst -> remove
  • admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
  • deb.html -> go into reference (query james gardner on list about moving this to wiki?)
1308324593000000 1312191462000000
#1193 enhancement kindly kindly closed fixed bug with new formalchemy causing all resouces to be deleted

The new formalchemy stores dict in a new format.

1308525655000000 1309768960000000
#1194 defect dread closed fixed "Welcome back" message for newly registered user
  1. Register a new user (/user/register)
  2. It redirects you to user page and displays "Welcome back" even though you are a new user.
1308563392000000 1317053688000000
#1195 defect rgrp dread ckan-sprint-2011-10-28 closed fixed Storage extension to handle incorrect routes better

We've seen exceptions caused by people getting the file storage link wrong, adding a slash to the end.

e.g. http://ckan.net/storage/f/file/e696d857-e997-41c8-be08-408697168ca8/

We should aim to handle all requests better than causing exception. 404 (or even 302 in this case) is better than 500.

1308576769000000 1311964374000000
#1196 defect dread dread ckan-v1.5-sprint-3 closed fixed Markdown logging ignores config settings

Markdown produces Debug logging, even when the pylons logging config is set to level of Info or Warning. This clogs up the logfile.

1308743476000000 1308743857000000
#1197 enhancement timmcnamara ckan-backlog closed wontfix Add JavaScript guide for CKAN

A new library, guiders.js, has been open sourced that seems to be great at unobtrusively introducing new users to features of websites.

The library drives Optimizely's website. The GitHub repo is here: https://github.com/jeff-optimizely/Guiders-JS

Some examples:

http://posterous.com/getfile/files.posterous.com/temp-2011-06-22/BnlgHDvhjpFfqbwvprsIaklqrpdgJyebfuhxajbDhmvnFgzdzzCJdvluJFki/Screen_shot_2011-06-22_at_6.42.21_PM.png.scaled500.png

http://s3.amazonaws.com/files.posterous.com/temp-2011-06-22/CpvlcinkrxhtnAHfGtEHhxyGkraqlFmCjcnztFuugagBGxmBjCdwuquHpvEB/promo_gmail.png

1308801032000000 1340020357000000
#1199 enhancement pudo pudo closed fixed Implement an email function for users.

We need an email function in CKAN to accept messages sent to users. The basic signature will be:

  • mail_user(user_obj, subject, body, mime_type='text/plain', headers={})

This has a number of use cases:

  • Retrieval of lost passwords
  • E-Mail confirmation

Finally, the mail function should be exposed in the API for sysadmin clients. This way we can have scripts traverse CKAN for 404s, invalid data or missing fields and ping users about that automatically (requires traversal by revision, not package, to get the associated users).

Implementation

Note we have already written code like this (*and* tested it) in isitopen:

1308821986000000 1310555589000000
#1200 enhancement pudo closed duplicate Port PDEU theme to CKAN.net

PDEU has a newer layout which could be adapted to also be used on ckan.net. For this, things should be cleaned up, copied to the ckanext-ckan.net repo as needed and re-colored the core CKAN color scheme.

Maybe we could have this coincide with a CKAN rename?

1308824017000000 1311180218000000
#1202 enhancement rgrp rolf closed invalid Links to datapkg utility don't lead to info about it

In the presentation of data packages, the link in "The information on this page and the downloads / resources are also available using the datapkg command line utility." (http://knowledgeforge.net/ckan/doc/datapkg/install.html) ends up at the home page of ckan.org

Perhaps link it to https://bitbucket.org/okfn/datapkg/overview for now, to save some searching for people who want to find out more?

1309263613000000 1315821804000000
#1204 enhancement timmcnamara closed invalid renaming packages seems to result in a server error

When renaming a package, and doing other changes at the same time an error is reported. However, the changes seem to go through fine.

ENV:

  • datacatalogs.org as of 29 June 2011
1309303816000000 1311179980000000
#1205 defect kindly dread closed fixed db upgrade problem on cz.ckan.net

When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below).

cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan.

In the meantime I've reverted the db to the old version (pre-upgrade).

cz.ckan.net is not working properly until this is fixed.

[email protected]:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://<user>:<pass>@<host>/<dbname>
  module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects
2011-06-29 09:07:41,294 INFO  [migrate.versioning.api] 28 -> 29... 
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname
  ret = fn(self, con, *args, **kw)
Traceback (most recent call last):
  File "/usr/bin/paster", line 18, in <module>
    command.run()
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 218, in run
    result = self.command()
  File "/usr/lib/pymodules/python2.6/ckan/lib/cli.py", line 84, in command
    model.repo.upgrade_db()
  File "/usr/lib/pymodules/python2.6/ckan/model/__init__.py", line 163, in upgrade_db
    mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 185, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 364, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py", line 83, in runchange
    change.run(self.engine, step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py", line 140, in run
    script_func(engine)
  File "/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py", line 152, in upgrade
    migrate_engine.execute(q)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1788, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1191, in execute
    params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
    return self.__execute_context(context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1302, in __execute_context
    context.parameters[0], context=context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py", line 299, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "package_group_revision" violates foreign key constraint "package_group_revision_continuity_id_fkey"
DETAIL:  Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table "package_group".
 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'}
1309339700000000 1310568631000000
#1206 defect wwaites dread closed fixed "Content-Type json" header scuppers package POST

Compare these two requests to create a package:

curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'Content-Type: application/json' -H 'X-CKAN-API-KEY: tester'
curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'X-CKAN-API-KEY: tester'

The second one gets the payload through (ckan log):

Retrieving request params: UnicodeMultiDict([('{name:"test"}', u'')])

But the first one causes a ServerError? because the payload (name:"test") doesn't make it to request.POST or request.params:

Retrieving request params: UnicodeMultiDict([])

The only difference is the "ContentType?: application/json" header, which seems a reasonable thing to include. Javascript lib backbone.js (for example) inserts this automatically.

So why does this header cause the payload to not get through to the request object?

1309344348000000 1309450216000000
#1207 enhancement dread dread closed fixed ckanclient.package_entity_get should raise more specific exception

When package does not exist in ckan catalogue, ckanclient.package_entity_get should raise more specific exception, such as CkanNotFoundError? instead of generic CkanApiError?.

1309515582000000 1311325343000000
#1208 enhancement rgrp rgrp ckan-sprint-2011-10-28 closed fixed Webstore: a datastore with web API
  • SQLite based
  • Use Cyclone for async http server on top with auth
  • Designed to not be specific to CKAN or anything else

Current code is here: https://github.com/okfn/webstore.

API Spec

Read

Two basic ways to query::

    GET: /{owner}/{db-name}/?sql=...
    GET: /{owner}/{db-name}/?table=...&attr=value&attr=value&limit=...

Returns::

  {
      u'keys': [u'id', u'name'],
      u'data': [
          [1, u'jones'],
          [u'aaa', u'jones']
          ]
  }

Write

POST to::

/{owner/{database}/{table}

Payload is json data structured as follows::

{

unique_keys: [list of key attributes] data: {dict of values}

}

Authentication and Authorization

Authentication: use basic auth header.

Authorization:

  • Default: all read, owner can write
  • Restricted: owner can read and write, everyone can do nothing

Possible future: config file can specify a python method (TODO: method signature)

Integration with Other Systems

TODO: Specify how to delegate authenatication to user database in some other system.

1309804460000000 1312191646000000
#1209 defect kindly closed fixed sort out rendering of formalchemy package preview

Got broken due to the moderated edits changes #1141.

Need to make sure preview works.

1309817240000000 1310571715000000
#1210 defect dread dread ckan-v1.5-sprint-4 closed fixed POST application/json error handling with newer WebOb

WebOb? from v1.0.7 has some interesting new behaviour with reading request data for different Content-Types:

  • It seems that looking at request.body mangles request.POST. See fix in ckan cset:e83bad9caa1b (problem with WebOb? v1.0.7 but may go back to v1.0.4?)
  • A particular request caused an AssertionError? during looking at request.body, so we need to enclose this in a try/except. I couldn't recreate it in a test unfortunately (problem with WebOb? v1.0.7 but may go back to v1.0.2?)

Example:

Module ckan.controllers.api:206 in create
<<          log.debug('create: %s' % (context))
               try:
                   request_data = self._get_request_data()
               except ValueError, inst:
                   response.status_int = 400
>>  request_data = self._get_request_data()
Module ckan.lib.base:149 in _get_request_data
<<          cls.log.debug('Retrieving request params: %r' % request.params)
               cls.log.debug('Retrieving request POST: %r' % request.POST)
               cls.log.debug('Retrieving request POST body: %r' % request.body)
               if request.POST:
                   try:
>>  cls.log.debug('Retrieving request POST body: %r' % request.body)
Module paste.registry:137 in __getattr__
<<      
           def __getattr__(self, attr):
               return getattr(self._current_obj(), attr)
           
           def __setattr__(self, attr, value):
>>  return getattr(self._current_obj(), attr)
Module webob.request:470 in _body__get
<<          Return the content of the request body.
               """
               self.make_body_seekable() # we need this to have content_length
               r = self.body_file.read(self.content_length)
               self.body_file.seek(0)
>>  self.make_body_seekable() # we need this to have content_length
Module webob.request:697 in make_body_seekable
<<              self.body_file_raw.seek(0)
               else:
                   self.copy_body()
>>  self.copy_body()
Module webob.request:714 in copy_body
<<                  self.body = self.body_file_raw.read(length)
               elif self.is_body_readable:
                   self.body = self.body_file_raw.read()
                   self._copy_body_tempfile()
               else:
>>  self.body = self.body_file_raw.read()
Module webob.request:1190 in read
<<      def read(self, size=-1):
               body = self._get_body()
               if size < 0:
                   v = body[self.position:]
>>  body = self._get_body()
Module webob.request:1207 in _get_body
<<                  self._body = _encode_multipart(self.vars, self.content_type)
                   else:
                       assert 0, ('Bad content type: %r' % self.content_type)
               return self._body
>>  assert 0, ('Bad content type: %r' % self.content_type)
AssertionError: Bad content type: '; charset=utf-8'
1309949451000000 1309974781000000
#1211 enhancement kindly kindly ckan-v1.5 closed fixed [super] Drupal integration

Sub-tasks / tickets:

  • #1335 Action API - access to any logic layer function
  • Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
  • Add package_create_validate & package_update_validate logic functions
  • Allow extensions to provide logic layer functions (IActions)
1309958442000000 1315948703000000
#1214 defect dread dread closed worksforme API improvements (following javascript use)

Notes on the CKAN API from Aron:

  • Server returns text/html for errors even when "Accept: application/json" header is set.
  • DELETE package request requires a Content-Length? Should be 405 Method Not Allowed?
  • Tag returned as a JSON object when updating but as a string when requesting. Same with "extras" content.
  • How to remove key from "extras" object? Passing None as described in the docs is invalid JSON and using null doesn't work.
  • Tags return package names rather than ids when querying GET /tag/{id}
  • Doesn't return rendered_text property on package update.
  • Relationships use "object" key rather than id or package_id.
1309975222000000 1314029628000000
#1215 defect dread closed fixed Edit Resources table - removing all rows prevents adding them again

I'm not sure if this was reported earlier, but I have found a minor bug in the "Add new package" form. As far as I can see this bug exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). This is a problem with adding new resources in "Resources" section. To reproduce the bug do the following:

  1. Go to "Add new package", for instance http://ckan.net/package/new
  2. Scroll down to "Resources" section.
  3. Click on "Remove this row" link (red circle).
  4. Say "Yes" to "Are you sure you wish to remove this row?"
  5. Now there should be no rows. Only the header "URL* Format Description"
  6. Click on "Add row to table"
  7. All you get now is another row with table header "URL* Format Description". You will not be able to add any resources now. Solution is to reload the page.

The simplest solution to this bug will be to hide the "Remove this row" link if there is only one row.

1310133224000000 1310385485000000
#1217 defect dread dread ckan-sprint-2011-11-07 closed invalid Diff exception

Exception raised doing this particular diff.

http://ckan.net/revision/diff/sudocfr?diff_entity=package&oldid=a18b2520-09e0-44be-80dd-4fea94ffe7d1&diff=3934ff16-a3a7-4481-b390-72944903b945

...
Module ckan.controllers.revision:142 in diff
<<          if c.diff_entity == 'package':
                   c.pkg = model.Package.by_name(id)
                   diff = c.pkg.diff(c.revision_to, c.revision_from)
               elif c.diff_entity == 'group':
                   c.group = model.Group.by_name(id)
>>  diff = c.pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:449 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'

Seems to be to do with this particular extra key:

[<PackageExtraRevision id=ba1f3b3e-f61f-4924-bae0-ec3dd2163ae4 package_id=febe4da9-5874-4411-a7da-b497fdebf739 key=namespace value=http://www.sudoc.fr/ state=active revision_id=5ec8e730-c73d-4f1c-989b-2e015eb3d315 continuity_id=ba1f3b3e-f61f-4924-bae0-ec3dd2163ae4 expired_id=None revision_timestamp=2011-07-08 13:48:11.027260 expired_timestamp=9999-12-31 00:00:00 current=True>]

Is it because it was created between the from_revision and to_revision?

1310375498000000 1320663277000000
#1218 enhancement dread [email protected] ckan-sprint-2011-10-28 closed fixed Colour the History tab icon

Change the color of the "History" tab icon to yellowish, like the rest of the icons in the other tabs ("View" and "Edit", "Authorization").

Reasoning

When you view a data package, for instance http://ckan.net/package/thesaurus-w, you will see "View", "Edit", "History" tabs on the top. "History" tab has a black-and-white icon which makes it look like an inactive/disabled tab (since the text is also grayed out when the tab is not selected). This is not a major issue, but it is a little bit confusing for the users. This icon exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net).

The simple solution would be to change the color of the "History" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. "View" and "Edit").

1310375768000000 1310389390000000
#1219 defect timmcnamara closed fixed CSS issues on IE7

As reported on ckan-dev:

items in the footer of CKAN ("Packages", "Groups & Tags", "About", "Language", etc.) are shown vertically instead of horizontally in IE7. This works fine in later browsers like IE8, IE9, FF4, and latest Opera and Chrome.

This seems to exist in all recent CKAN versions up to 1.4.1a.

1310423688000000 1310740534000000
#1221 defect dread dread ckan-sprint-2011-10-28 closed fixed Password reset tidy up
  • No tests.
  • 500 error given when user is not found.
1310475661000000 1310556607000000
#1222 defect dread dread ckan-sprint-2011-10-28 closed fixed Mailer has no tests 1310479136000000 1310556618000000
#1223 enhancement pudo pudo closed fixed Caching of static files

StaticURLParser can have caching - use it

1310573854000000 1310573893000000
#1225 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Tool for mass tag changes

Using ckanapi

1310642161000000 1310642407000000
#1226 defect dread closed worksforme Genshi stream exception

IndexError? exception in genshi.filters.i18n keeps popping up daily.

e.g.

...
Module ckan.controllers.authorization_group:28 in index
<<              items_per_page=20
               )
               return render('authorization_group/index.html')
       
           def read(self, id):
>>  return render('authorization_group/index.html')
Module ckan.lib.base:74 in render
<<      
           return cached_template(template_name, render_template, cache_key=cache_key, 
                                  cache_type=cache_type, cache_expire=cache_expire)
                                  #, ns_options=('method'), method=method)
>>  cache_type=cache_type, cache_expire=cache_expire)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:63 in render_template
<<              stream = item.filter(stream)
               
               return literal(stream.render(method=method, encoding=None))
           
           if 'Pragma' in response.headers:
>>  return literal(stream.render(method=method, encoding=None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
       
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out=out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:339 in __call__
<<          for filter_ in self.filters:
                   stream = filter_(stream)
               for kind, data, pos in stream:
                   cached = cache_get((kind, data))
                   if cached is not None:
>>  for kind, data, pos in stream:
Module genshi.output:670 in __call__
<<          _gen_prefix = _gen_prefix().next
       
               for kind, data, pos in stream:
                   output = cache_get((kind, data))
                   if output is not None:
>>  for kind, data, pos in stream:
Module genshi.output:771 in __call__
<<          push_text = textbuf.append
               pop_text = textbuf.pop
               for kind, data, pos in chain(stream, [(None, None, None)]):
       
                   if kind is TEXT:
>>  for kind, data, pos in chain(stream, [(None, None, None)]):
Module genshi.output:586 in __call__
<<      def __call__(self, stream):
               prev = (None, None, None)
               for ev in stream:
                   if prev[0] is START:
                       if ev[0] is END:
>>  for ev in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module ckanext.googleanalytics.plugin:62 in download_adder
<<              (downloaded %s times)</span>'''
                   count = None
                   for mark, (kind, data, pos) in stream:
                       if mark and kind == START:
                           href = data[1].get('href')
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1175 in __call__
<<          """
               callable_value = hasattr(self.value, '__call__')
               for mark, (kind, data, pos) in stream:
                   if mark is ENTER:
                       if callable_value:
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
       
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.markup:378 in _match
<<                      for event in self._match(self._flatten(template, ctxt,
                                                                  **vars),
                                                    ctxt, start=idx + 1, **vars):
                               yield event
>>  ctxt, start=idx + 1, **vars):
Module genshi.template.markup:327 in _match
<<                      break
       
               for event in stream:
       
                   # We (currently) only care about start and end events for matching
>>  for event in stream:
Module genshi.template.base:545 in _flatten
<<          while 1:
                   for kind, data, pos in stream:
       
                       if kind is START and data[1]:
>>  for kind, data, pos in stream:
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range
1311008861000000 1314029434000000
#1228 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Display OpenID login errors

When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again.

1311086562000000 1311325161000000
#1229 enhancement dread thejimmyg ckan-sprint-2011-10-24 closed fixed Refactor all database code out of the controllers and into the logic layer

Progress can be tracked in this document:

https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdFQzRnVoUHV6ckoyblI5TUdEcnJLVlE&hl=en_US#gid=0

1311093924000000 1319645778000000
#1230 enhancement kindly ckan-sprint-2011-10-28 closed fixed Make sure plugins tables work in tests.

Dgu tests failed as harvesting table are not created at the correct point.

1311100811000000 1311154142000000
#1231 requirement kindly thejimmyg ckan-backlog closed wontfix [super] Management Information Reporting

Child tickets:

  • #1101 Integrate stats and googlanalytics into site nav

We have a spreadsheet from UKLP of statistics we'd like to generate

1311173919000000 1325474447000000
#1234 enhancement dread dread ckan-sprint-2011-10-28 closed fixed Setting locale error conditions handled badly

Needs tests for error conditions for bad locale and redirect parameters.

1311179871000000 1311183328000000
#1236 enhancement dread dread ckan-sprint-2011-11-07 closed fixed View a package at a given revision

As a user I want to view a package at a given revision:

  • When I visit /package/[email protected] or /package/[email protected] I should be shown package at date yyyy-mm-dd or revision rrrrrrr.
  • viewing this page, it should say it is the old revision, just like wikipedia
  • package history page should provide links to these pages
1311269333000000 1330019788000000
#1238 defect kindly dread ckan-sprint-2011-10-28 closed invalid Viewing a package @date misses non-PackageRevisions

There's a problem with getting old versions of packages using @date. The query looks for the PackageRevision? related to the date, but ignores TagRevisions? etc.

So for example if you compare this table: http://ckan.net/package/history/osm with this view: http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830 you actually see the 2010-01-13 11:13 revision - the wrong revision.

1311351303000000 1311759784000000
#1239 defect rgrp rgrp ckan-sprint-2011-10-28 closed fixed Remove and re-add tag on package not working

See reports on list at:

Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix.

1311441020000000 1312453156000000
#1241 task timmcnamara timmcnamara closed wontfix Overview video of http://thedatahub.org

Produce a video explaining some of the things which can be done on the datahub.org

1311567813000000 1340632688000000
#1242 enhancement timmcnamara closed fixed thedatahub.org's login page

The right hand side of thedatahub.org login screen is a little confusing:

    Not a member?

    Join CKAN to contribute packages under your own name.

     - Register
     - Reset your password
     - Privacy Policy

Suggested improvements to make:

  • replace CKAN with thedatahub.org
  • in general, edit the copy to be more inviting
  • remove "Reset your password" from that bubble
  • move the privacy policy link to somewhere less prominent
1311569218000000 1323173292000000
#1243 enhancement timmcnamara closed invalid [thedatahub.org] "Upload Data" should provide a nicer

If someone who is not logged in clicks on "Upload Data", they are presented with

    Not authorized to upload files.

This is in emboldened red text. That is, the system is telling a user that they're doing something illegitimate, even though they're actually following an invitation from the main menu.

I propose two options for resolving this:

  1. only display the "Upload Data" tab if someone is authorised to upload data
  2. if someone isn't logged int, provide a nice welcome message thanking them for wanting to upload data, but we only allow registered users to do so

ENV: thedatahub.org, CKAN 1.4.2b

1311577466000000 1328000405000000
#1245 task rgrp ckan-v1.5 closed fixed Reorganise ckan.org pages

Here is Rufus' plan for ckan.org reorganisation from http://ckan.okfnpad.org/documentation (22/7/11) (ticket made by dread):

/solutions
  /overview <--- current 'about' page (the software)
  /customers <-- say "in addition to our clients..."
  /features <-- needs most work (pictures etc)
/pricing/  <-- simplest thing possible ... talk with Ira (cover pricing and support)
/developers <--- links to http://docs.ckan.org/, getting the software, the wiki, the mailing list /contact
/blog

Tasks:

  • add tagline in header - "the data portal software"
  • make and add glossy PDF
  • change "data hub" to "data portal" passim (if only using one term)
  • add [email protected]
  • image banner on home page - redo screenshots without menus
1311861708000000 1317422640000000
#1246 defect pudo [email protected] ckan-sprint-2011-10-28 closed fixed Search results on ckan.net are mistakenly all 'open'

All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: http://ckan.net/package

1311863353000000 1311892816000000
#1247 task rgrp dread ckan-sprint-2011-10-28 closed fixed Put docs on ReadTheDocs

Put docs on ReadTheDocs? instead of eu13.

1311941948000000 1311955185000000
#1248 defect dread dread ckan-sprint-2011-11-07 closed fixed License exception

Exception received on test.ckan.net at 10.12 on 29/7/11

http://test.ckan.net/package?__cache=2759373&page=4
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:153 in search
<<                          }
       
                   query = get.package_search(context,data_dict)
       
                   c.page = h.Page(
>>  query = get.package_search(context,data_dict)
Module ckan.logic.action.get:514 in package_search
<<      for package in query.results:
               result_dict = table_dictize(package, context)
               result_dict = _extend_package_dict(result_dict,context)
       
               results.append(result_dict)
>>  result_dict = _extend_package_dict(result_dict,context)
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: None'

and a similar one:

 http://test.ckan.net/tag/barbecue
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:118 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.tag:2 in read
Module ckan.lib.cache:167 in wrapper
<<      log = __import__("logging").getLogger("proxy_cache")
           def wrapper(func, *args, **kwargs):
               result = func(*args, **kwargs)
       
               pylons = get_pylons(args)
>>  result = func(*args, **kwargs)
Module ckan.controllers.tag:64 in read
<<          data_dict = {'id':id}
               try:
                   c.tag = get.tag_show(context,data_dict)
               except NotFound:
                   abort(404, _('Tag not found'))
>>  c.tag = get.tag_show(context,data_dict)
Module ckan.logic.action.get:341 in tag_show
<<      extended_packages = []
           for package in tag_dict['packages']:
               extended_packages.append(_extend_package_dict(package,context))
       
           tag_dict['packages'] = extended_packages
>>  extended_packages.append(_extend_package_dict(package,context))
Module ckan.logic.action.get:537 in _extend_package_dict
<<      license_id = package_dict['license_id']
           if license_id:
               isopen = model.Package.get_license_register()[license_id].isopen()
               package_dict['isopen'] = isopen
           else:
>>  isopen = model.Package.get_license_register()[license_id].isopen()
Module ckan.model.license:59 in __getitem__
<<              return default
               else:
                   raise KeyError, "License not found: %s" % key
       
           def get(self, key, default=None):
>>  raise KeyError, "License not found: %s" % key
KeyError: u'License not found: CreativeCommonsAttributionCCBY25'
1311954136000000 1319797763000000
#1249 enhancement pudo rgrp ckan-v1.5 closed fixed Exclude script tag from extraction for i18n

Currently have this script section put in for i18n. It shouldn't be.

<script type="text/javascript">
//<![CDATA[
(function($){
    $.fn.ajaxCreateSlug = function(name, url) {
        var title = this;
        var updater = {
            init: function(title, name) {
                // Add a new element where the validity of the package name can be displayed
                this.name_field = name;
                this.title_field = title;
                this.name_field.parent().append('<div id="package_name_valid_msg"></div>');
                this.title_field.blur(this.title_change_handler())
                this.title_field.keyup(this.title_change_handler())
                this.name_field.keyup(this.name_change_handler());
                this.name_field.blur(this.name_blur_handler());
                this.url = url;
            },
....
1311958536000000 1311959356000000
#1250 enhancement pudo rgrp ckan-v1.5 closed fixed Search results should be sorted by score rather than alphabetical

At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases.

Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard).

Options:

  • Default this in solr (no need to touch code) but fragile and affects everything ...
  • Do it in code and default to score
  • Do it in code and have alphabetical (on name or title?) when no criteria otherwise score

Aside: may also wish to support search in query api but that is for later!

1311964752000000 1312370201000000
#1251 enhancement rgrp rgrp ckan-backlog closed fixed Rename wiki.ckan.net to wiki.ckan.org

Inline with general naming policy going forward.

1311965147000000 1317315159000000
#1252 enhancement rgrp rgrp ckan-v1.5 closed fixed Deprecate ckan.net in favour of thedatahub.org

Deprecate ckan.net in favour of thedatahub.org

  • Announce
  • Create thedatahub.org and have it run same data as ckan.net
  • Announce closing down of ckan.net with 302
  • Implement 302 permanent

Also redirect wiki.ckan.net to wiki.ckan.org

1311965339000000 1318160334000000
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Note: See TracReports for help on using and creating reports.