Link Checking

[wwaites]

revisit link checker from  http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py

revisit ollyc's parallel work

look at how this gets used in practice (maybe deprecate in favour of curate tool (suitably wrapped to hide details from the user)

[wwaites]

• link checker above uses the queue. queue not running generally
• quickest way forward is just to put the curate tool in a cron job and make a suitable rule. shall do this soonest

[wwaites]

Ready to put into cron job. cf:

•  http://groups.google.com/group/fuxi-discussion/browse_thread/thread/47f131fc2e3817e3 (Actions)
•  http://groups.google.com/group/fuxi-discussion/browse_thread/thread/bf955620a6ae77d8 (denoted/calculated functions)
•  http://groups.google.com/group/fuxi-discussion/browse_thread/thread/71a94191e9fef384 (FuXi? 1.2)
•  https://github.com/wwaites/curate/commit/042a96c1589c0fa4980aca733c64c080e02f111e (curate tool update)

[wwaites]

currently running against ckan.net, adding broken_link tag if a broken link is found. perhaps something more elaborate should be done? works for now anyhow...

[wwaites]

urllib2 is good for http(s) urls but not, unfortunately, for other types most prominently ftp.

change the httpReq action to use  http://curl.haxx.se/libcurl/python/

[wwaites]

Implemented curlReq that does a curl request and returns statements similar to or analogous to httpReq. Require curate<=0.8

This in a cron job is sufficient to go through all the packages and update them with a broken link tag now:

curate -r https://github.com/wwaites/curate/raw/master/examples/tagging.n3 -s -k API_KEY