{
  "hf_id": "nvidia/Cosmos3-Super",
  "meta": {
    "title": "Cosmos3-Super",
    "slug": "cosmos3-super",
    "provider": "NVIDIA",
    "description": "Frontier-scale 64B omnimodal world model (Mixture-of-Transformers) for advanced multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI",
    "date_added": "2026-06-02",
    "date_updated": "2026-06-02",
    "tasks": [
      "omni"
    ],
    "performance_headline": "64B omnimodal world model — ~55s/video on 8×H200 with HSDP + ulysses",
    "related_recipes": [
      "nvidia/Cosmos3-Nano",
      "nvidia/Cosmos3-Super-Text2Image",
      "nvidia/Cosmos3-Super-Image2Video"
    ],
    "hardware": {
      "h200": "verified",
      "h100": "verified"
    }
  },
  "model": {
    "model_id": "nvidia/Cosmos3-Super",
    "min_vllm_version": "0.21.0",
    "docker_image": "vllm/vllm-omni:cosmos3",
    "architecture": "dense",
    "parameter_count": "64B",
    "active_parameters": "64B",
    "context_length": 262144,
    "base_args": [
      "--host",
      "0.0.0.0",
      "--port",
      "8000",
      "--cfg-parallel-size",
      "2",
      "--ulysses-degree",
      "4",
      "--use-hsdp",
      "--hsdp-shard-size",
      "8",
      "--init-timeout",
      "1800"
    ],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-omni:cosmos3",
        "note": "Release-tested vLLM-Omni image bundling the Cosmos3 omni handlers."
      }
    }
  },
  "omni": {
    "tasks": [
      {
        "id": "t2v",
        "description": "Text → video (audio muxed into the MP4)",
        "curl": "curl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"prompt=$(cat assets/example_t2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=123\" \\\n  --output output.mp4\n"
      },
      {
        "id": "i2v",
        "description": "Image → video (temporally coherent, audio muxed in)",
        "curl": "curl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"input_reference=@assets/example_i2v_input.jpg\" \\\n  -F \"prompt=$(cat assets/example_i2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=1111\" \\\n  --output output.mp4\n"
      }
    ]
  },
  "features": {},
  "opt_in_features": [],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 154,
      "description": "BF16 weights — frontier-scale; 8×H200/H100/A100 node recommended"
    }
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\n**Cosmos3-Super** is the frontier-scale (64B) member of NVIDIA's Cosmos3 family\nof **omnimodal world models**. Built on a Mixture-of-Transformers (MoT)\narchitecture — an autoregressive tower for discrete-token generation paired\nwith a diffusion tower for continuous modalities — it generates coherent text,\nimages, video, audio, and action commands from combinations of text, image,\nvideo, and action-trajectory inputs, for advanced world understanding,\nsimulation, future prediction, and action reasoning.\n\nThis recipe covers the **vLLM-Omni** serving path (video + muxed-audio\ngeneration). The same checkpoint also serves a text/multimodal **Reasoner**\nendpoint via standard vLLM — see [Reasoner mode](#reasoner-mode-multimodal-understanding)\nbelow.\n\n## Prerequisites\n\n- 8× H200, H100, or A100 for the documented full-node profile\n- The release-tested `vllm/vllm-omni:cosmos3` container, or vLLM-Omni installed\n  on top of final `vllm==0.33.0` for this source checkout. The published\n  Cosmos3 container and its historical results use an older runtime and do\n  not qualify the current source. The separate Reasoner environment below,\n  where present, retains its own plugin/runtime requirements.\n- Prompts should be JSON-upsampled for best quality (see\n  [cosmos-framework prompt upsampling](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/prompt_upsampling.md))\n\n## Launch command (vLLM-Omni)\n\nRecommended configuration on 8×H200, 8×H100, or 8×A100:\n\n```bash\ndocker pull vllm/vllm-omni:cosmos3\n\nvllm serve nvidia/Cosmos3-Super \\\n  --omni \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --cfg-parallel-size 2 \\\n  --ulysses-degree 4 \\\n  --use-hsdp \\\n  --hsdp-shard-size 8 \\\n  --init-timeout 1800\n```\n\nWith this configuration, 50-step video generation takes ~55 seconds on H200.\nFor **2×H200**, use `--cfg-parallel-size 2 --use-hsdp --hsdp-shard-size 2`\n(a video takes ~3 minutes). Tensor parallelism is also supported via\n`--tensor-parallel-size`. On memory-constrained GPUs,\n`--enable-layerwise-offload` reduces VRAM usage at a performance cost.\n\n### Example: text-to-video\n\n```bash\ncurl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"prompt=$(cat assets/example_t2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=123\" \\\n  --output output.mp4\n```\n\nImage-to-video uses the same endpoint with an `input_reference` file part.\nAudio is generated jointly and muxed into the output MP4 (48 kHz stereo AAC).\n\n## Reasoner mode (multimodal understanding)\n\nThe same checkpoint serves a text/multimodal **Reasoner** via the standard\n`vllm` package (not vLLM-Omni):\n\n```bash\n# CUDA 13 drivers:\nuv pip install --torch-backend=cu130 \"vllm==0.21.0\" \\\n  \"vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3\" \\\n  openai\n# CUDA 12.8 drivers: use --torch-backend=cu128 \"vllm==0.19.1\"\n\nvllm serve nvidia/Cosmos3-Super \\\n  --hf-overrides '{\"architectures\": [\"Cosmos3ReasonerForConditionalGeneration\"]}' \\\n  --tensor-parallel-size 4 \\\n  --mm-encoder-tp-mode data \\\n  --async-scheduling \\\n  --allowed-local-media-path / \\\n  --media-io-kwargs '{\"video\": {\"num_frames\": -1}}' \\\n  --port 8000\n```\n\nThen query the OpenAI-compatible `/v1/chat/completions` endpoint with text +\nimage/video content.\n\n## References\n\n- [Model card](https://huggingface.co/nvidia/Cosmos3-Super)\n- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)\n- [cosmos-framework](https://github.com/NVIDIA/cosmos-framework)\n- [Cosmos](https://github.com/nvidia/cosmos)\n",
  "source": {
    "source": "vllm-omni",
    "repo": "vllm-project/vllm-omni",
    "ref": "main",
    "sha": "02286829737a54167e890185bcbb6948ed9afbcf",
    "path": "recipes/models/nvidia/Cosmos3-Super.yaml",
    "edit_url": "https://github.com/vllm-project/vllm-omni/edit/main/recipes/models/nvidia/Cosmos3-Super.yaml",
    "overlays": []
  }
}