http://localhost/ticket/1641 <p> The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the "Content-Type" and "Content-Length" headers differ from the values stored to see if the resource needs to be updated <a class="missing changeset" title="No default repository defined">[1]</a>. </p> <p> The "Content-Length" header, although widely used, is not mandatory and some servers don't provide it, e.g.: </p> <pre class="wiki">$ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG HTTP/1.1 200 OK Cache-Control: private Transfer-Encoding: chunked Content-Type: text/xml Vary: Accept-Encoding Server: Microsoft-IIS/7.5 Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly X-AspNetMvc-Version: 3.0 content-disposition: attachment; filename=AFG_IATI_12012012.xml X-AspNet-Version: 4.0.30319 X-Powered-By: ASP.NET Date: Thu, 12 Jan 2012 12:36:43 GMT </pre><p> Also worth noting that <a class="ext-link" href="http://docs.python-requests.org/"><span class="icon">​</span>requests</a>, the python library that uses ckanext-archiver, sets an "Accept-Encoding: gzip" header by default, which depending on the configuration of the remote web server, may prevent the "Content-Length" server from being sent, e.g.: </p> <pre class="wiki">$ curl -H "Accept-Encoding: gzip" -I http://iatistandard.org/published-temp/adb-activities.xml HTTP/1.1 200 OK Date: Thu, 12 Jan 2012 12:12:46 GMT Server: Apache Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Content-Type: application/xml curl -I http://iatistandard.org/published-temp/adb-activities.xml HTTP/1.1 200 OK Date: Thu, 12 Jan 2012 11:56:23 GMT Server: Apache Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT Accept-Ranges: bytes Content-Length: 2686720 Vary: Accept-Encoding Content-Type: application/xml </pre><p> All this can lead to some resources never getting updated, and of course the size property of the resource not being set. </p> <p> As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it). </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72"><span class="icon">​</span>https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72</a> </p> en-us http://assets.okfn.org/p/ckan/img/ckan_logo_shortname.png http://localhost/ticket/1641 Trac 0.12.3 amercader Thu, 12 Jan 2012 13:59:37 GMT http://localhost/ticket/1641#comment:1 http://localhost/ticket/1641#comment:1 <ul> <li><strong>status</strong> changed from <em>new</em> to <em>closed</em> </li> <li><strong>resolution</strong> set to <em>fixed</em> </li> </ul> <p> Fixed on 77fa6483c32 </p> Ticket