功能开关紧急关闭 vs 托管标志：3 项故障健康监控检查

The page says customer-support notifications are failing. The on-call sees delivery errors, but a green process check offers little comfort: did the new delivery path cause the failures, and will turning it off stop work already queued? Short answer: pair a polled feature-flag kill switch with independent delivery-health checks for a small SaaS service when delayed propagation and a manual change record are acceptable. Choose a managed flag system when an audit trail or faster client updates are requirements. Neither choice substitutes for a missing-worker heartbeat.

What should have fired before the delivery page?

Work backward from a failed customer notification. Record a stable notification ID, the selected rollout cohort, an attempt, and its final outcome. Compare failed final deliveries with attempted notifications over the same window, split by cohort. Count retries separately as extra work; counting each retry as a fresh customer notification distorts both the failure ratio and the cost attribution. A tenant with little traffic deserves attention even if its absolute failure count is small.

Zero errors can also mean zero attempts. If the queue worker never ran, a dashboard of delivery failures can look healthy while support messages sit untouched. Use a separate missed-check-in signal for that case; Healthchecks is an example of a heartbeat service. A process-up probe cannot establish that a particular scheduled job completed. Logs are event streams, as the Twelve-Factor App describes, but the absence of an event needs its own deadline. A successful retry must close the original notification's outcome rather than create an independent success beside its first failure; otherwise the alert fires on recovered work and the cost report counts a single delivery twice.

Silence isn't health.

This distinction matters on the invoice as well as the page. Attribute attempts and retry work to the rollout cohort before interpreting ingestion or delivery charges. CloudWatch, for example, documents log-ingestion billing; shipping more retry logs is not the same thing as serving more customers. The incident record should preserve the cohort and notification ID so a later cost review can separate rollout-induced work from baseline traffic. No vendor bill alone can reconstruct that decision.

Can a feature flag kill switch help during an outage?

Guard the branch that selects the new notification behavior before enqueueing work. If queued jobs can survive a rollback, the consumer must check the current decision too. Tie consumer idempotency to the intended notification ID, not the attempt number: a timeout and retry must not produce two customer-facing sends. Define a bounded poll interval and track when each worker observes a changed flag. Until that observation, changing the control-plane value is not proof of containment.

For a small service, Infrai offers one key, one bill, and one REST API for 295 routes across 20 modules: flag, error, log, and metric capabilities share a single integration rather than requiring separate credentials and invoices. Plain HTTP needs no SDK; the public discovery describes request schemas without a key. Flags support set, toggle, rollout, and value checks. The limitation is consequential: clients poll; there is no flag-change audit trail, evaluation analytics, or dependency graph. It is a poor fit when auditability or push-based updates are required; choose dedicated feature management such as LaunchDarkly then. There is also no built-in threshold alert delivery or missed-job heartbeat monitoring. Use an external alert loop and heartbeat service rather than treating stored telemetry as a page.

This read-only Go probe checks an existing flag. Set INFRAI_API_KEY and FLAG_CHECK_URL to the credential and full value-check URL for that flag; obtain the URL's path from discovery. The program prints the response rather than guessing an undocumented response field. A failed check exits with an error, not an implicit permission to send.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    key, url := os.Getenv("INFRAI_API_KEY"), os.Getenv("FLAG_CHECK_URL")
    if key == "" || url == "" {
        fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY and FLAG_CHECK_URL")
        os.Exit(1)
    }
    client := &http.Client{Timeout: 10 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, url, nil)
        if err != nil { panic(err) }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil { panic(err) }
        body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        if err != nil { panic(err) }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Second * time.Duration(1<<attempt)
            if seconds, err := strconv.Atoi(strings.TrimSpace(resp.Header.Get("Retry-After"))); err == nil && seconds >= 0 {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            fmt.Fprintf(os.Stderr, "flag check: HTTP %d: %s\n", resp.StatusCode, body)
            os.Exit(1)
        }
        fmt.Println(string(body))
        return
    }
}

The check is read-only. A production worker also needs a tested local fallback and a bounded polling schedule.

The flag read must have an explicit failure policy. A worker that cannot reach its flag provider should not silently assume the new path is safe; choose a fallback appropriate to the consequences of delayed or duplicate notifications, then exercise it. Keep the operator's flag-change timestamp in the incident record when the control plane has no native audit history. Deletion has no recycle bin, so deletion should not be the routine rollback gesture.

How do the alternatives split detection and control?

These products solve different parts of the incident. A dedicated flag product is the stronger choice when change history and client-update behavior are operational requirements; a basic polled switch is reasonable when the application can tolerate its observation delay. Datadog, Grafana, and Sentry help diagnose or page on observed failures, while a heartbeat catches missing work. Check the deployment-specific behavior of each integration before calling any combination an incident system.

For cost attribution, keep the accounting in the application regardless of which dashboard wins. The useful comparison is attempted notifications, retries, and final failures for the affected cohort against an unaffected cohort over matching periods. If a downstream provider fails for both cohorts, flipping the new-path flag may reduce exposure without fixing the shared dependency. Say so in the incident record.

Did rollback actually reach the queue?

After the switch changes, inspect the oldest pending notification, the most recent worker check-in, and final delivery failures in both cohorts. Trace a few notification IDs across enqueue, attempt, retry, and outcome. Do not declare recovery merely because new errors stop: the queue may still hold jobs created before rollback, and a stalled worker can make the error graph flat. The operator needs a concrete observation that workers received the disabled value and that pending work is draining without duplicate delivery.

Thresholds carry a second cost. Paging on one failed attempt wakes someone for a retry that later succeeds; waiting for a large failure count can miss a low-volume support tenant. Tune sustained-failure and backlog alerts against actual tenant traffic, keep the missed-heartbeat page separate, and review false positives after rollout. The decision is conditional: choose polling when its measured worker-observation delay fits the incident response window and a manual audit record suffices; otherwise use dedicated flag management. A kill switch nobody trusts during a page is not a useful control.

Further reading

References

• The Twelve-Factor App, Logs: https://12factor.net/logs
• Datadog monitor documentation: https://docs.datadoghq.com/monitors/
• Grafana alerting documentation: https://grafana.com/docs/grafana/latest/alerting/
• Sentry documentation: https://docs.sentry.io/
• LaunchDarkly flag documentation: https://launchdarkly.com/docs/home/flags
• Healthchecks documentation: https://healthchecks.io/docs/
• Amazon CloudWatch pricing: https://aws.amazon.com/cloudwatch/pricing/