CKAN: Ticket #371: Implement QoS monitoring

owner set

johnbywater — Fri, 23 Jul 2010 11:55:51 GMT

owner set to johnbywater

description changed

johnbywater — Tue, 27 Jul 2010 09:39:12 GMT

description modified (diff)

type, description changed; businessvalue set

johnbywater — Wed, 28 Jul 2010 19:41:05 GMT

type changed from enhancement to requirement
businessvalue set to 300
description modified (diff)

status changed

johnbywater — Wed, 28 Jul 2010 20:19:44 GMT

status changed from new to closed

status changed

johnbywater — Wed, 28 Jul 2010 20:20:35 GMT

status changed from closed to reopened

businessvalue changed

johnbywater — Wed, 28 Jul 2010 22:52:15 GMT

businessvalue changed from 300 to 1200

milestone set

johnbywater — Fri, 30 Jul 2010 09:43:18 GMT

milestone set to v1.1

summary changed

johnbywater — Fri, 30 Jul 2010 09:49:17 GMT

summary changed from Implement QoS monitoring to The system shall monitor QoS against SLA

cc set

nils.toedtmann — Mon, 13 Dec 2010 16:19:49 GMT

cc nils.toedtmann@… added

(I know the term "QoS" as a very specific networking term about classifying and prioritising network traffic. I assume here it means uptime, availability, performance monitoring?)

There seem to be are at least three monitors already in place:

http://munin.okfn.org/ on eu1 monitoring eu[0-7] and us1, gathering additional health information via locally installed daemons. Munin's notification subsystem is not configured.
http://nagios.hmg.ckan.net/ on hmg.ckan.net monitoring the CKAN-HMG service group (network monitoring only). Notfications are not configured (or?)
We have a http://wasitup.com/ account which is watching some OKFN services (e.g. {ca,de,www}.ckan.org, {blog,www}.okfn.org) and sending loads of alerts to sysadmin@…. Only checking for "HTTP 200 OK" and whether the response contains a configurable string.

My 2ct: We should consolidate. What do we want?

A webservice like https://www.pingdom.com/ ($40/month incl 30 checks and 200 SMS, $0.5/month per extra check, $0.14-20/SMS) or http://www.serverdensity.com/ ($10/server-month plus 5-10p/SMS)?
Or run our own monitor (nagios, opsview, monin)?

In the latter case we want to have a separate machine which is not in on EC2 (but e.g. ByteMark?), dedicated to monitoring only.

We should also include root mails into the alert/notification policies. Root mails should be trimmed down to important warnings and errors only.

nils.toedtmann — Mon, 13 Dec 2010 16:23:09 GMT

The nagios fork OPSview might be worth a look.

nils.toedtmann — Sat, 18 Dec 2010 20:38:36 GMT

Replying to nils.toedtmann:

There seem to be are at least three monitors already in place:

Correction: at least four, we seem to have a Montastic account, too:

On 18/12/10 15:03, noreply@… wrote:

Dear okfn,
This is a monthly reminder that you have an account on Montastic, the
website monitor service.
### ACCOUNT INFORMATION
Signup date: 2009-10-06
Email you signup with: sysadmin@okfn.org
### 20 WEBSITES MONITORED
[OK] - http://www.ckan.net/
[OK] - http://www.knowledgeforge.net/
[OK] - http://okfn.org/
[not monitored] - http://blog.okfn.org/
[...]
### EMAIL ALERT RECIPIENTS
- sysadmin@okfn.org
- rufus.pollock@okfn.org
- martin.keenan@okfn.org
[...]
To make changes to your account or contact us, go to www.montastic.com.
[...]

owner, status changed

thejimmyg — Fri, 07 Jan 2011 14:23:45 GMT

owner changed from johnbywater to nils.toedtmann
status changed from reopened to new

thejimmyg — Fri, 07 Jan 2011 14:52:19 GMT

It is implied in this that the performance of sites should beat the QoS criteria, therefore closing #485. Ensuring this happens is an ongoing process.

thejimmyg — Fri, 07 Jan 2011 16:23:54 GMT

From #440 we'll also need to "Write and pass comprehensive performance tests"

thejimmyg — Fri, 07 Jan 2011 16:25:53 GMT

From #395:

At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls.

This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance.

NB: We should also agree on a maximum request latency.

TODO: Read up on all those QoS tickets to avoid overlapping efforts.

anonymous — Mon, 10 Jan 2011 16:14:53 GMT

Mainly handled in http://knowledgeforge.net/okfn/tasks/ticket/564 now. Close here?

status changed; resolution set

thejimmyg — Tue, 15 Mar 2011 19:37:00 GMT

status changed from new to closed
resolution set to fixed

Marking as closed since http://knowledgeforge.net/okfn/tasks/ticket/600 now takes on this ticket.

I will check nils has added the new DGU Bytemark servers are added to Nagios.