What each service affects
Each Validio service runs as a single replica, so when a service is unavailable, the features that depend on it stop until Kubernetes restarts it. For a description of each service, refer to Validio Services.Suggested alerts
The following alerts use standard Kubernetes signals, such as container restarts, resource usage, and volume usage, that your cluster monitoring tool already collects. They are starting points. Tune the thresholds to your deployment and workload.
When a container is restarted for running out of memory, raise its memory limit with
<service>.resources.limits.memory. For more information, refer to VPC Deployment Installation.
Health endpoints and probes
Each service exposes a liveness endpoint and a readiness endpoint. The Helm chart’s Kubernetes liveness and readiness probes call these endpoints, so a failing endpoint shows up as restarts or as a pod that isn’t ready.
To change probe timings, use the
<service>.probes.* keys. For more information, refer to VPC Deployment Installation.
You can also view the status of each service in the Validio app. For more information, refer to Validio Services.
Logs
All Validio services write logs to standard output, so you can collect them with your cluster’s log pipeline or read them withkubectl. For example, to read the Diver logs from the last hour:
Bash
- Logs from the affected services, covering the time of the issue.
- The output of
kubectl -n validio get podsandkubectl -n validio describe pod <pod-name>for any pod that is restarting or not ready. - Your Validio version and the
validio-values.yamlfile, with secrets removed.
Change the Ingress log level
You can change the log level of the Ingress service at runtime, without a restart. This is useful for collecting more detailed logs while you reproduce an issue. The change isn’t persisted, and Ingress returns to its configured log level when it restarts.-
Forward the Ingress port to your machine:
Bash
-
Check the current log level:
Bash
-
Set the log level, for example to
debug:Bash -
When you have collected the logs, set the level back to
info.