http://localhost/ticket/537 <p> There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests. </p> <p> Introducing caching has to be done carefully and should be done in a <em>standards compliant manner</em>. </p> <h2 id="Generalstrategy">General strategy</h2> <ul><li>Where possible, cache output within the pylons app (beaker). </li><li>Facilitate external caching in an end-user's web browser or a caching proxy </li><li>Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes). </li><li>When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data. </li></ul><h2 id="Tasks">Tasks</h2> <p> These tasks should be broken into sub-tickets: </p> <ul><li>caching of parts of templates that are expensive to render (package list, tag list, group list) </li><li>caching of entire output using beaker particularly for API read operations. </li><li>need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches. </li><li>cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: <a class="ext-link" href="http://www.mnot.net/cache_docs/#CACHE-CONTROL"><span class="icon">​</span>http://www.mnot.net/cache_docs/#CACHE-CONTROL</a> <ul><li>Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well </li><li>Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner </li></ul></li></ul><h2 id="FutureWork">Future Work</h2> <ul><li>Investigate ckanclient library maintaining a local cache as a web browser would </li><li>Investigate using a CDN like Google Storage or Amazon for serving cached data. </li></ul> en-us http://assets.okfn.org/p/ckan/img/ckan_logo_shortname.png http://localhost/ticket/537 Trac 0.12.3 wwaites Mon, 30 Aug 2010 16:11:34 GMT http://localhost/ticket/537#comment:1 http://localhost/ticket/537#comment:1 <ul> <li><strong>type</strong> changed from <em>requirement</em> to <em>task</em> </li> </ul> Ticket johnbywater Mon, 30 Aug 2010 16:44:00 GMT http://localhost/ticket/537#comment:2 http://localhost/ticket/537#comment:2 <ul> <li><strong>owner</strong> set to <em>johnbywater</em> </li> <li><strong>milestone</strong> <em>ckan v1.2</em> deleted </li> </ul> Ticket johnbywater Mon, 30 Aug 2010 16:52:27 GMT http://localhost/ticket/537#comment:3 http://localhost/ticket/537#comment:3 <ul> <li><strong>description</strong> modified (<a href="/ticket/537?action=diff&amp;version=3">diff</a>) </li> </ul> Ticket johnbywater Mon, 30 Aug 2010 16:53:31 GMT http://localhost/ticket/537#comment:4 http://localhost/ticket/537#comment:4 <ul> <li><strong>description</strong> modified (<a href="/ticket/537?action=diff&amp;version=4">diff</a>) </li> </ul> Ticket johnbywater Mon, 30 Aug 2010 17:21:56 GMT http://localhost/ticket/537#comment:5 http://localhost/ticket/537#comment:5 Ticket johnbywater Mon, 30 Aug 2010 17:23:37 GMT http://localhost/ticket/537#comment:2 http://localhost/ticket/537#comment:2 Ticket johnbywater Mon, 30 Aug 2010 17:24:05 GMT http://localhost/ticket/537#comment:3 http://localhost/ticket/537#comment:3 <ul> <li><strong>type</strong> changed from <em>task</em> to <em>defect</em> </li> </ul> Ticket johnbywater Mon, 30 Aug 2010 17:24:48 GMT http://localhost/ticket/537#comment:4 http://localhost/ticket/537#comment:4 <ul> <li><strong>type</strong> changed from <em>defect</em> to <em>task</em> </li> </ul> Ticket johnbywater Mon, 30 Aug 2010 17:25:38 GMT http://localhost/ticket/537#comment:5 http://localhost/ticket/537#comment:5 <ul> <li><strong>owner</strong> changed from <em>johnbywater</em> to <em>wwaites</em> </li> </ul> Ticket wwaites Wed, 01 Sep 2010 07:12:36 GMT http://localhost/ticket/537#comment:6 http://localhost/ticket/537#comment:6 <p> See <a class="closed ticket" href="http://localhost/ticket/540" title="story: Implement caching in a systematic manner (closed: fixed)">#540</a> for a story about Varnish. Strongly favouring squid at this juncture. </p> Ticket dread Mon, 25 Oct 2010 16:11:37 GMT http://localhost/ticket/537#comment:7 http://localhost/ticket/537#comment:7 <p> Can this ticket be updated? Were any tasks listed here done? Anything remaining still planned? </p> Ticket thejimmyg Wed, 20 Jul 2011 16:22:09 GMT http://localhost/ticket/537#comment:8 http://localhost/ticket/537#comment:8 <ul> <li><strong>priority</strong> set to <em>awaiting triage</em> </li> <li><strong>repo</strong> set to <em>ckan</em> </li> <li><strong>theme</strong> set to <em>none</em> </li> <li><strong>status</strong> changed from <em>new</em> to <em>closed</em> </li> <li><strong>resolution</strong> set to <em>duplicate</em> </li> </ul> <p> Consolidation of caching has been moved in ticket <a class="closed ticket" href="http://localhost/ticket/995" title="defect: consolidate caching (closed: wontfix)">#995</a>. </p> Ticket