Set up webhook notifications

Send CosmicAC job and model health events to an HTTPS endpoint, including a Slack webhook.

Send CosmicAC events to an HTTPS endpoint you control. CosmicAC posts a payload when a job fails, degrades, or recovers, and when a served model's health changes. One webhook covers the whole deployment.

Job lifecycle events also depend on the settings of the job that raises them. See What controls delivery.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
An HTTPS endpoint that accepts POST requests. For Slack, create an incoming webhook, then copy its URL.

Steps

Open the Notifications page

In the left navigation, click Settings. On the Settings page, under Instance, click Notifications.

Save the webhook URL

In Webhook URL, enter the HTTPS URL that receives the events, then click Save. CosmicAC sends nothing until you save a URL.

The first save also generates the signing secret. Copy it now, because CosmicAC shows it only once and your receiver needs it to verify signatures.

Select the payload format

Under Format, select one of the following:

Generic JSON: a flat event object carrying id, type, schema, timestamp, instance, and the fields for that event type.
Slack-compatible: a Slack incoming webhook body carrying text and a color-coded attachments block. Red marks a failure, green a recovery, and orange a degraded state.

Select Slack-compatible when the URL points at Slack. CosmicAC posts the body straight to the incoming webhook URL, so it needs no Slack workspace credentials, bot token, or OAuth 2.0 authorization.

Select the events to send

Under Events, turn on each event you want CosmicAC to send. Delivery is opt-in, so CosmicAC never sends an event you leave off.

The four job.* events also need the matching preference on the job that raises them. See What controls delivery.

Event	Fires when
`job.failed`	A job transitions to Failed, and the event carries the failure reason.
`job.degraded`	Healthy replicas drop below the count you set, and the endpoint stays live.
`job.recovered`	A job returns to Active from Degraded or Failed.
`job.restart_storm`	Any replica restarts three times within 10 minutes.
`model.health.down`	A served model's health enters Down.
`model.health.recovered`	A served model's health leaves Down.

Click Save to apply the selection.

Send a test event

Click Send test event. CosmicAC posts a test.ping payload to the saved URL.

Confirm the delivery

Check Recent Deliveries. Each row lists the event type, the target URL, the response status, and how long ago CosmicAC sent it. A successful delivery shows the response code, such as 200 OK.

A delivery counts as failed when the receiver answers with a status outside the 200 range or doesn't answer in time. CosmicAC retries a delivery that fails with a 5xx status, a 429, a network error, or a timeout, up to three times with a growing delay. It doesn't retry any other status, and it discards the event after the last attempt.

Verify the signature on your receiver

Verify signatures when your receiver is a service you run. A Slack incoming webhook can't run verification code, so it ignores the signature header.

Your receiver needs the signing secret to confirm that a payload came from CosmicAC. CosmicAC shows the secret once, when you first save the webhook URL. If you didn't copy it then, click Regenerate to issue a new one.

CosmicAC sends these headers:

X-Cosmic-Event: the event type, such as job.failed.
X-Cosmic-Delivery: the unique event identifier. Use it to discard duplicates.
X-Cosmic-Timestamp: the send time in milliseconds.
X-Cosmic-Attempt: which attempt this delivery is, starting at 1.
X-Cosmic-Signature: sha256= followed by the HMAC-SHA256 digest of the raw request body, keyed with the signing secret. CosmicAC sends this header, and the identical X-Cosmic-Signature-256, only when a signing secret exists.

Recompute the digest over the raw body and compare it before you trust a payload.

const crypto = require('crypto')

const expected = 'sha256=' + crypto
  .createHmac('sha256', process.env.COSMICAC_WEBHOOK_SECRET)
  .update(rawBody)
  .digest('hex')

const signature = req.headers['x-cosmic-signature']
const valid = expected.length === signature.length &&
  crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(signature))

What controls delivery

Every job has four notification preferences, one for each lifecycle event. CosmicAC turns on all four when you create the job. To set them, see Create a GPU Container Job, Create a vLLM Managed Inference Job, and Create a Parakeet Managed Inference Job.

An event reaches your webhook only if both of these are on:

The job's notification preference, which controls whether CosmicAC creates the event.
The event under Events on the Settings → Notifications page, which controls whether CosmicAC posts it.

If either one is off, the event doesn't reach your webhook. Turning it on later doesn't deliver the events that CosmicAC already skipped.

CosmicAC delivers each event it creates to two destinations.

Destination	Requirements
The notification bell in the web interface	None. CosmicAC shows every event it creates.
Your webhook	A saved webhook URL, and the event turned on under Events on the Settings → Notifications page.

The Events selection applies only to the webhook. If you turn off an event under Events, it still appears in the notification bell.

The CLI and the web interface spell the same event differently. Pass the underscore form, such as job_failed, to cosmicac jobs create --notify. The web interface shows job.failed.

The model.health.down and model.health.recovered events have no per-job preference, because no job raises them. The Events selection is the only setting that controls them.

Rotate the signing secret

Click Regenerate to replace the signing secret. Copy the new secret before you leave the page, because CosmicAC shows it only once. The previous secret stops working immediately, so update your receiver in the same maintenance window.