CKAN: Ticket #851: Link Checking

status changed

wwaites — Mon, 20 Dec 2010 16:02:37 GMT

status changed from new to assigned

link checker above uses the queue. queue not running generally
quickest way forward is just to put the curate tool in a cron job and make a suitable rule. shall do this soonest

wwaites — Tue, 21 Dec 2010 00:48:35 GMT

Ready to put into cron job. cf:

http://groups.google.com/group/fuxi-discussion/browse_thread/thread/47f131fc2e3817e3 (Actions)
http://groups.google.com/group/fuxi-discussion/browse_thread/thread/bf955620a6ae77d8 (denoted/calculated functions)
http://groups.google.com/group/fuxi-discussion/browse_thread/thread/71a94191e9fef384 (FuXi? 1.2)
https://github.com/wwaites/curate/commit/042a96c1589c0fa4980aca733c64c080e02f111e (curate tool update)

status changed; resolution set

wwaites — Tue, 21 Dec 2010 01:24:21 GMT

status changed from assigned to closed
resolution set to fixed

currently running against ckan.net, adding broken_link tag if a broken link is found. perhaps something more elaborate should be done? works for now anyhow...

status changed; resolution deleted

wwaites — Tue, 21 Dec 2010 14:21:43 GMT

status changed from closed to reopened
resolution fixed deleted

urllib2 is good for http(s) urls but not, unfortunately, for other types most prominently ftp.

change the httpReq action to use http://curl.haxx.se/libcurl/python/

status changed; resolution set

wwaites — Wed, 22 Dec 2010 13:38:32 GMT

status changed from reopened to closed
resolution set to fixed

Implemented curlReq that does a curl request and returns statements similar to or analogous to httpReq. Require curate<=0.8

This in a cron job is sufficient to go through all the packages and update them with a broken link tag now:

curate -r https://github.com/wwaites/curate/raw/master/examples/tagging.n3 -s -k API_KEY