id	summary	reporter	owner	description	type	status	priority	milestone	component	resolution	keywords	cc	repo	theme
1345	Investigate possible memory leak	nils.toedtmann	kindly	"There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) ([http://trac.okfn.org/ticket/882 ticket]) we experienced extraordinary memory usage peaks ([http://trac.okfn.org/ticket/904 ticket]). Here are the observed value with Apache default settings:
 * eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
 * s055, mpm-prefork: base level ~1GB, peaks up to 4GB
 * s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests ([https://bitbucket.org/okfn/sysadmin/changeset/126290aead28 changeset]). This (together with two other tweaks) changed the situation drastically:

 * s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to ''processes=3''), and then observing the relation between ''maximum-requests=25...500'' and memory consumption.


On 14/09/11 17:49, David Read wrote:
> Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1"	enhancement	closed	awaiting triage		ckan	fixed		rgrp pudo	ckan	none