Custom Query (2152 matches)

Reported by amercader, 2 years ago.

Tom Kralidis kindly told us on IRC [1] that the latest trunk of pycsw includes support for wsgi and different backends, which sounds like could be really useful for CKAN integration.

[1] ​http://irclogs.okfn.org/ckan/2012-04-25.txt 17:16

Reported by nils.toedtmann, 3 years ago.

There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (​ticket) we experienced extraordinary memory usage peaks (​ticket). Here are the observed value with Apache default settings:

• eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
• s055, mpm-prefork: base level ~1GB, peaks up to 4GB
• s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (​changeset). This (together with two other tweaks) changed the situation drastically:

• s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to processes=3), and then observing the relation between maximum-requests=25...500 and memory consumption.

On 14/09/11 17:49, David Read wrote:

Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1

Reported by rgrp, 4 years ago.

Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page.

Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish.

Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat.