http://localhost/ticket/1345 <p> There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions: </p> <p> When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (<a class="ext-link" href="http://trac.okfn.org/ticket/882"><span class="icon">​</span>ticket</a>) we experienced extraordinary memory usage peaks (<a class="ext-link" href="http://trac.okfn.org/ticket/904"><span class="icon">​</span>ticket</a>). Here are the observed value with Apache default settings: </p> <ul><li>eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB </li><li>s055, mpm-prefork: base level ~1GB, peaks up to 4GB </li><li>s055, mpm-worker: base level ~1.5GB, peaks up to 6GB </li></ul><p> William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (<a class="ext-link" href="https://bitbucket.org/okfn/sysadmin/changeset/126290aead28"><span class="icon">​</span>changeset</a>). This (together with two other tweaks) changed the situation drastically: </p> <ul><li>s055, mpm-event: base level ~1.4GB, no peaks </li></ul><p> This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak. </p> <p> To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to <em>processes=3</em>), and then observing the relation between <em>maximum-requests=25...500</em> and memory consumption. </p> <p> On 14/09/11 17:49, David Read wrote: </p> <blockquote class="citation"> <p> Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin. </p> </blockquote> <p> +1 </p> en-us http://assets.okfn.org/p/ckan/img/ckan_logo_shortname.png http://localhost/ticket/1345 Trac 0.12.3 nils.toedtmann Fri, 25 May 2012 15:55:53 GMT http://localhost/ticket/1345#comment:1 http://localhost/ticket/1345#comment:1 <p> This becomes an actual problem on the CKAN farm s057, see <a class="ext-link" href="http://trac.okfn.org/ticket/1245"><span class="icon">​</span>http://trac.okfn.org/ticket/1245</a> </p> Ticket nils.toedtmann Fri, 01 Jun 2012 11:58:46 GMT http://localhost/ticket/1345#comment:2 http://localhost/ticket/1345#comment:2 <p> See <a class="new ticket" href="http://localhost/ticket/2485" title="defect: Encourage leak containment by limiting the number of requests a CKAN ... (new)">#2485</a> for workarounds containing the leak(s). </p> Ticket ross Wed, 20 Jun 2012 11:12:17 GMT http://localhost/ticket/1345#comment:3 http://localhost/ticket/1345#comment:3 <ul> <li><strong>cc</strong> <em>thejimmyg</em> removed </li> <li><strong>owner</strong> changed from <em>dread</em> to <em>kindly</em> </li> <li><strong>status</strong> changed from <em>new</em> to <em>assigned</em> </li> </ul> <p> Moved out of milestone so that it gets looked at again soon </p> Ticket amercader Wed, 25 Jul 2012 11:24:49 GMT http://localhost/ticket/1345#comment:4 http://localhost/ticket/1345#comment:4 <ul> <li><strong>status</strong> changed from <em>assigned</em> to <em>closed</em> </li> <li><strong>resolution</strong> set to <em>fixed</em> </li> </ul> <p> See <a class="ext-link" href="https://github.com/okfn/ckan/pull/73"><span class="icon">​</span>https://github.com/okfn/ckan/pull/73</a> </p> Ticket