Check Managed Inference endpoint health
Check the status, success rate, and latency of a Managed Inference endpoint from the web interface or the CLI.
Check the status, success rate, and latency of your Managed Inference endpoints, and probe an endpoint when you need a fresh reading.
Prerequisites
You need the following before you start:
- A running Managed Inference Job with a serving endpoint. See Create a Managed Inference Job.
- Access to the CosmicAC web interface, if you check health there.
- The CosmicAC CLI installed and configured, if you check health from a terminal. See Install the CLI.
Steps
Open the health view
CosmicAC probes the replicas every five minutes by default, and both methods show the result of the last probe.
In the left sidebar, open Model Health. The page re-reads the results every 30 seconds while it's open.
To change the time range, choose one at the top of the page. The choices are 1H, 6H, 24H, 7D, and 30D. The page defaults to 7D.
The web interface and the CLI default to different ranges, so the same endpoint can report different numbers in each.
Read the results
Each endpoint reports the following:
- Status shows Healthy, Degraded, or Down. For what each value means, see Model health.
- Success rate shows the share of requests that succeeded.
- Traffic shows the total requests handled.
- Failures shows how many of those requests failed.
- Avg response shows the average response time in milliseconds. The CLI prints it as Latency (avg).
- Last health check shows when CosmicAC last probed the endpoint. The CLI prints it as Last Check At.
- Last updated shows when CosmicAC generated the reading. The CLI prints it as Timestamp.
To find an unhealthy replica, read the per-replica table that the CLI prints, which gives the replica ID, status, traffic, failures, and average latency for each one. The web interface shows a replica health count on each endpoint card.
Probe an endpoint on demand
Probe when you need a reading now rather than at the next scheduled one.
To probe every endpoint, click Run health check at the top of the Model Health page. To probe one endpoint, click Health check on its card.
The figures don't update when you click. They update on the next refresh, within 30 seconds.
CosmicAC probes a replica by sending it a real inference request, so a probe uses serving capacity. Probing every endpoint sends a request to every replica you run. On a large deployment, probe a single endpoint instead.
To trigger a probe from a script or another tool, see Run health checks.