initialize metric mox_panic_total with 0, so the alerting rule also catches the first panic for a label

increase() and rate() don't seem to assume a previous value of 0 when a vector
gets a first value for a label. you would think that an increase() on a
first-value mox_panic_total{"..."}=1 would return 1, and similar for rate(), but
that doesn't appear to be the behaviour. so we just explicitly initialize the
count to 0 for each possible label value. mox has more vector metrics, but
panics feels like the most important, and it's too much code to initialize them
all, for all combinations of label values. there is probably a better way that
fixes this for all cases...
This commit is contained in:
Mechiel Lukkien
2023-09-15 16:47:17 +02:00
parent af71e9855b
commit 3620d6f05e
16 changed files with 75 additions and 22 deletions

View File

@ -44,7 +44,7 @@ func monitorDNSBL(log *mlog.Log) {
if x != nil {
log.Error("monitordnsbl panic", mlog.Field("panic", x))
debug.PrintStack()
metrics.PanicInc("serve")
metrics.PanicInc(metrics.Serve)
}
}()