> ## Documentation Index
> Fetch the complete documentation index at: https://docs.validio.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring VPC Deployments

> Monitor a self-managed Validio VPC deployment with Kubernetes-level alerts, health endpoints, and logs.

Monitoring a VPC deployment means tracking the health, resource usage, and logs of the Validio services in your Kubernetes cluster and alerting on them. In a [Customer Virtual Private Cloud](https://docs.validio.io/docs/deployment/virtual-private-cloud.md) deployment, Validio has no access to your infrastructure, so your DevOps team is responsible for this monitoring.

This guide describes what each service does when it is unavailable, which alerts to start with, and what to collect when you contact [Validio support](https://docs.validio.io/docs/other/support.md). For installation and configuration, refer to [VPC Deployment Installation](https://docs.validio.io/docs/deployment/virtual-private-cloud/vpc-deployment.md).

## What each service affects

Each Validio service runs as a single replica, so when a service is unavailable, the features that depend on it stop until Kubernetes restarts it. For a description of each service, refer to [Validio Services](https://docs.validio.io/docs/deployment/validio-services.md).

| Service | What users notice when the service is unavailable |
| - | - |
| `Surface` | The Validio web app and API are unreachable. |
| `Diver` | Validators, catalog refreshes, and other scheduled work don't run, notifications aren't sent, and most pages in the app fail to load data. |
| `Ingress` | Validio can't connect to sources, so validator runs, metadata and catalog updates, and source health checks fail. |
| `Pipelines` | New validator results and incidents aren't produced, because data statistics aren't processed. |
| `Postgres` | The whole platform is unavailable, since all platform data and configuration is stored in Postgres. |
| `Redis` | Data stops moving between services, so validator runs stall. Streaming sources are affected first. |

## Suggested alerts

The following alerts use standard Kubernetes signals, such as container restarts, resource usage, and volume usage, that your cluster monitoring tool already collects. They are starting points. Tune the thresholds to your deployment and workload.

| Alert | Signal | Suggested starting threshold |
| - | - | - |
| Service unavailable | Ready replicas of a Validio StatefulSet or Deployment compared with the desired count, which is always 1 | Below the desired count for more than 15 minutes |
| Container restarts | Restart count of any Validio container | More than 3 restarts in 15 minutes |
| Out of memory | Container terminated with reason `OOMKilled` | Any occurrence |
| Memory near limit | Container memory working set compared with its memory limit | Above 90% of the limit for 10 minutes |
| CPU throttling | Share of CPU periods throttled, only if you set CPU limits | Above 25% for 15 minutes |
| Postgres disk usage | Used space on the Postgres persistent volume | Above 80% |
| Failed migrator job | The `ve-migrator` or `surface-migrator` Job fails or doesn't complete after a release that includes migrations | Any failure, or not complete after 30 minutes |

When a container is restarted for running out of memory, raise its memory limit with `<service>.resources.limits.memory`. For more information, refer to [VPC Deployment Installation](https://docs.validio.io/docs/deployment/virtual-private-cloud/vpc-deployment.md#4-configure-the-validio-values-yaml-file).

## Health endpoints and probes

Each service exposes a liveness endpoint and a readiness endpoint. The Helm chart's Kubernetes liveness and readiness probes call these endpoints, so a failing endpoint shows up as restarts or as a pod that isn't ready.

| Service | Port | Liveness | Readiness |
| - | - | - | - |
| `Diver` | `10001` | `/healthz/live` | `/healthz/ready` |
| `Ingress` | `8080` | `/healthz/live` | `/healthz/ready` |
| `Pipelines` | `8080` | `/healthz/live` | `/healthz/ready` |
| `Surface` | `8889` | `/healthz/live` | `/healthz/ready` |

To change probe timings, use the `<service>.probes.*` keys. For more information, refer to [VPC Deployment Installation](https://docs.validio.io/docs/deployment/virtual-private-cloud/vpc-deployment.md#4-configure-the-validio-values-yaml-file).

You can also view the status of each service in the Validio app. For more information, refer to [Validio Services](https://docs.validio.io/docs/deployment/validio-services.md).

## Logs

All Validio services write logs to standard output, so you can collect them with your cluster's log pipeline or read them with `kubectl`. For example, to read the Diver logs from the last hour:

```bash title="Bash" theme={null}
kubectl -n validio logs statefulset/validio-diver --since=1h
```

When you contact [Validio support](https://docs.validio.io/docs/other/support.md), include:

* Logs from the affected services, covering the time of the issue.
* The output of `kubectl -n validio get pods` and `kubectl -n validio describe pod <pod-name>` for any pod that is restarting or not ready.
* Your Validio version and the `validio-values.yaml` file, with secrets removed.

### Change the Ingress log level

You can change the log level of the Ingress service at runtime, without a restart. This is useful for collecting more detailed logs while you reproduce an issue. The change isn't persisted, and Ingress returns to its configured log level when it restarts.

1. Forward the Ingress port to your machine:

   ```bash title="Bash" theme={null}
   kubectl -n validio port-forward statefulset/validio-ingress 8080
   ```

2. Check the current log level:

   ```bash title="Bash" theme={null}
   curl http://localhost:8080/settings/log/level
   ```

3. Set the log level, for example to `debug`:

   ```bash title="Bash" theme={null}
   curl -X PUT -H "Content-Type: application/json" -d '{"level":"debug"}' http://localhost:8080/settings/log/level
   ```

4. When you have collected the logs, set the level back to `info`.

The other Validio services don't support changing the log level at runtime.
