{22} Trac tickets (2647 matches)

E.g. uploading a file with spaces in it: OECD Monthly Exchange Rates.zip

Gives a 404 Not Found response to the following file: ​http://test.ckan.org/api/storage/metadata/2011-12-05T193046/OECD%20Monthly%20Exchange%20Rates.zip

The upload wheel keeps spinning and the user is not informed that the upload has failed.

Analysis: turns out that google storage (possibly s3 but not checked) replaces with ' ' in keys with '+' on upload. This breaks things because we try and look up metadata about upload using the filename/key we put in but of course that does not exist because google has changed name.

Fix is trivial: replace ' ' in keys / filenames with '-'.

This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied':

​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html

The problem was he didn't have access to the group. A message like 'You don't have access to the group "LODcloud"' would have been clearer.

CKAN returns HTML when an Accept header should make it return RDF/XML:

​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002237.html

On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

"The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.

If a dataset has no resources the resources list currently says '(none)'.

Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'

There doesn't seem to be any documentation yet for Atom feeds.

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

On a dataset page on demo.ckan.org, the left sidebar is confusing.

• It starts with some random links. Actually they are links to groups which the dataset is in, but this isn't clear.
• The sidebar elements that are actually part of the dataset are 'Datset extent' and 'License', so these should be right at the top (if they belong in the sidebar at all), instead of which they are right at the bottom in the junk part of the page (i.e. probably lower than the bottom of the main page, and hence lower than anyone will scroll).

Revised text for 'What are groups?' box at demo.ckan.org/group (after discussion with IB re #2812):

What are groups?

Groups allow you to group users and data together so that they are easier to manage. Group owners can assign roles and authorisations, giving each project or department control of its own data publishing.

Users can browse or facet by groups, which could be an organisation (for example, the Department of Health) or topic (e.g. Transport, Health), making it easier to find the data they are looking for.

no need for separate files

e.g. license, owner etc

as above

Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g.

mycssfile.css?v=12345678

A possible bug in CKAN when I tried deleting users using "paster --plugin=ckan user delete" command.

To reproduce the bug do the following:

1. Create a user with an ID (which in my case was a user's full name)

that contains non-unicode caracters like Norwegian "æ", "ø", or "å".

2. Make sure that you can see something like the example below:

(pyenv) rm@mycomputer:$ paster --plugin=ckan user Users: name=Rustæm

3. Then try deleting the user with following command:

(pyenv) rm@mycomputer:$ paster --plugin=ckan user delete "Rustæm"

You should now get a python encoding error. I know that this is quite rare case, but in our case it caused some trouble. Could you guys have a look at this bug?

CKAN ver. 1.3.3.

Investigate and list (on the wiki at wiki:UiReviewNotes) CKAN's UI issues, in particular:

• Form usability
  • best example: ​http://ckan.net/package/new
  • "Extras" fields
  • Adding to groups (from group page or from package page?)

• Overview pages: reduction in visual noise and unnecessary elements.
  • best example: package overview page
  • consider openness/downloadability icons
  • display common metadata fields in a more structured fashion?

Est cost: 2 hrs.

The "Preview" button is a nice idea, but it doesn't seem to actually "preview" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file.

This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of application/x-some-junk-here in order to force a download.

I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org.

.

... or, if the CKAN packages do contain architecture-specific binary code, build packages for i386 too.

Currently, ​http://apt-alpha.ckan.org/debian only offers packages for amd64, but e.g. "m1.small" EC2 instances are i386.

We would need this in order to migrate the community instances to a packaged based CKAN.

Rufus, pls prioritise.

The debian package "ckan" with the two scripts "ckan-create-instance" and "ckan-instance-maintenance" depends against "postgresql". But "ckan-create-instance" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs.

Please add a flag "--without-local-db" to "ckan-create-instance" and remove the postgres dependancy from the debain package.

Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3).

That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's "<VirtualHost?>", or tracd). Depending on the "Host:" header in the HTTP1.1 request, it would choose which local ckan ini file to obey.

I see two ways to constitute the map hostname-to-ini-file map:

• ckan reads a set of ini files, and each ini file declares which servers names it is responsible for
• In a global ini file, there are directives mapping servernames to ini files.

In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc.

There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (​ticket) we experienced extraordinary memory usage peaks (​ticket). Here are the observed value with Apache default settings:

• eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
• s055, mpm-prefork: base level ~1GB, peaks up to 4GB
• s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (​changeset). This (together with two other tweaks) changed the situation drastically:

• s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to processes=3), and then observing the relation between maximum-requests=25...500 and memory consumption.

On 14/09/11 17:49, David Read wrote:

Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

• research ways to limit requests-per-process with the different web servers (see below for a start);
• explain these safeguards in the CKAN documentation and encourage users to apply them;
• consider the helper script ​ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of ​commented out. Or at least let it warn the user to use MaxRequestsPerChild;
• Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

• Apache:
  • Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on ​datahub.io and the ​ckan farm
  • (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
• nginx/supervisord: to be researched.

This will be useful as an example consumer of the new queue notifications (#325).

• A user can register any URL to be notified upon an event.
• Possible extensions: GET with ID only vs. POST with serialized object
• E-Mail notifications are also hooks, essentially.

We need to have a UI module that allows for the hook CRUD and some feedback (e.g. non 200 status codes)

We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app)

Create a thin web service that will allow users to upload files to OFS on S3 without knowing S3 credentials. This could also provide static urls for all stored resources.

• relates to #307

• As sysadmin/user of CKAN I want to integrate a CMS to support easy addition of additional content ...
• I want to integrate with navigation
• As a minimum I want to be able to add content pages (more stuff would be e.g. comments, sidebars, main menu ...)

Why: CMS are great at creating content and have very good editors and theme support ...

Previous work

​http://groups.google.com/group/pylons-discuss/browse_thread/thread/f8969e28b0a007c7

• deliverance: retheming on the fly

​http://knowledgeforge.net/shakespeare/hg/file/677eef9402e6/shakespeare/controllers/template.py

1. Rewrite urls for proxying
2. You need a theme to substitute into
3. pylons

Extreme version:

• 2 sources -> pylons/ckan app, cms
• 1 theme source: ckan or cms

1 frontend/proxy app which re-themes everything

Owned by CKAN:

• package/
• tag/
• group/
• api/
• /

Everything else proxied.

• Need to figure out cache invalidation for CKAN
• Evaluate against an internal cache (i.e. memcached)

There seem to be a few conditions under which either queue processing or the indexer fail in their current state. To get a more systematic picture of these failures, we should have a small testing tool to compare the index to the database of a live CKAN instance.

cf. Tariqs mail

Updated version: ​https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&hl=en#gid=0

Original Version

Publishing Entity:
Publishing Entity Type: (Donor, Recipient, Community Data..)
Donor
Country
Activity period:
Verification status: enumeration of statuses (checked, not checked etc)
Resource links: to the actual IATI record
Number of activities: ...
Date record updated:
Date data updated:
License: Need this field even if it may be a standard license

So naively mashing these together, we get something like:

Identifier
Title
Abstract
Donor
Country
Publisher
Publisher Type
Verification Status
Department
Contact
Contact e-mail
Licence
Resource format
Resource URL
Resource ID
Activity period
Number of activities
Date record updated
Date data updated

• Use what is on iatiregistry.org at the moment.

Read/write APIs for users and publishing entities to access and register data

• Users have read access
• Publishing entities have both (though can only write to 'their' records)
• Priority: 4 for write (read access is 3)

• So some kind of confirmation or approval by someone more senior?
• Alternative: just have an account request.
• Priority: 3

From lopad:

Publishing Entity (often/always? equal to the logged in user)
    * Domain logic: only can add entries for "yourself"
Donor
Country
Activity period: period to which this record relates
Verification status: enumeration of statuses (checked, not checked etc)
Resource links: to the actual IATI record
Number of activities: ...
Date record updated:
Date data updated: 
License: Need this field even if it may be a standard license

Displaying siblings is not needed for link_to/from.

Lookup is needed to add a package but won't show up at some times.

Originally reported by Anja Jentzsch, re questions.

reported by LOD folks.

subsidary bug: error messages from OpenID providers are not shown on the form return page.

• copy exisiting tests, modify authz in setUp, adapt and extend tests
  • Problem: default_role_actions is read by init_db
  • Solution: nuke db after monkey-patching
  • role = model.Role('Reader'), del role.actions[...]

self.PRE_MAUTHZ_RULES = copy(mauthz.default_role_actions) mauthz.default_role_actions.remove((Role.READER, Action.CREATE)) #raise Exception(mauthz.default_role_actions) model.Session.remove() model.repo.rebuild_db()

3. Start from the functional
   • Both the api controllers and wui controllers to do
   • tests/function/test_authz.py extended to check CREATE in lockedDown mode. fails nicely. ​http://bitbucket.org/pudo/ckan-authz2

is_authorized(user, Action.Create, model.Package)

-> Doing this will put 'Package' in the context field of the user_object_role table. This will trigger SQLAlchemy to attempt a join towards PackageRole? in all queries. Since for class-level role assignments there never is a PackageRole? join table entry, this will never return any results.

• cf branch "classes": ​http://bitbucket.org/pudo/ckan-authz2/src/6fd0475e0c66
• No possibility of including "instance" column in inheritance decision.
• Even if this can be cirumvented, it must happen on a per-query level and would require a major re-modelling.

• have a ckan install that would not allow visitors to either list packages or list groups
  • two paths: create the listing, but for each group/pkg decide that you cannot show this
  • lock down the whole page (/package/list)
    • this is class-based, not object-based

is_authorized(user, Action.Package_Create, model.System()) is_authorized(user, Action.Group_Create, model.System())

[Separating Package and Group roles may be useful going forward: PackageEditor?, GroupEditor? etc]

4. Find a standard way to lock down classes
   • possibly add default rows in user_object_role
   • introduce lock-down into controllers: group new, package new, REST equivalents,
   • confirm tests

• Copy & adapt Group controller tests to suit AuthorizationGroups?
• Copy & adapt Group controllers and templates for Authorization Groups

Much more natural place to do this, even at the risk of overloading the package edit form.

Perhaps using deliverance?

• Move About to the right of the nav tabs and make it an about IATI page
• Make a Help / FAQ section linked of main menu
• Front page more dynamic - latest data registered and news (microblog thing)
• make the front page blurb noticable / shorter etc.
• link to iatistandard.org from the front page blurb

• Change "Publishing Entity" to "Publisher"

Move from hetzner box to s3.