CosmicAC Logo

Managed Inference Job

Guides for creating Managed Inference Jobs and connecting clients to them.

Step-by-step guides for common tasks with a Managed Inference Job. These guides assume you already have CosmicAC running. If you don't, follow the installation guide.

Guides

Create a Managed Inference Job in the web interface

Serve a model behind an OpenAI-compatible endpoint.

Create an API key in the web interface

Create a key for endpoints that require an authorization header.

Connect to a vLLM Managed Inference endpoint

Send a chat-completion request from the CLI or an OpenAI-compatible client.

Create a Parakeet Managed Inference Job

Create a speech-to-text Parakeet job from the CLI.

Transcribe audio with a Parakeet endpoint

Upload an audio file to a Parakeet endpoint and get a transcription.

Check Managed Inference endpoint health

Check the status, success rate, and latency of an endpoint.

Managed Inference Job

Understand what a Managed Inference Job is and when to use it.

Create a Managed Inference Job

Create a Managed Inference Job from the CLI.

Inference commands

Look up the CLI commands to run inference on a model.

Model commands

Look up the CLI commands to inspect deployed models.

Set up SSH and Tailscale on a GPU Container Job

Previous Page

Create a Managed Inference Job in the web interface

Next Page

On this page

GuidesRelated