{
  "hf_id": "nvidia/Cosmos3-Nano",
  "meta": {
    "title": "Cosmos3-Nano",
    "slug": "cosmos3-nano",
    "provider": "NVIDIA",
    "description": "Compact 16B omnimodal world model (Mixture-of-Transformers) for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI",
    "date_added": "2026-06-02",
    "date_updated": "2026-06-02",
    "tasks": [
      "omni"
    ],
    "performance_headline": "16B omnimodal world model — single-GPU H200 video/audio generation",
    "related_recipes": [
      "nvidia/Cosmos3-Super",
      "nvidia/Cosmos3-Super-Text2Image",
      "nvidia/Cosmos3-Super-Image2Video"
    ],
    "hardware": {
      "h200": "verified",
      "h100": "verified"
    }
  },
  "model": {
    "model_id": "nvidia/Cosmos3-Nano",
    "min_vllm_version": "0.21.0",
    "docker_image": "vllm/vllm-omni:cosmos3",
    "architecture": "dense",
    "parameter_count": "16B",
    "active_parameters": "16B",
    "context_length": 262144,
    "base_args": [
      "--host",
      "0.0.0.0",
      "--port",
      "8000",
      "--init-timeout",
      "1800"
    ],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-omni:cosmos3",
        "note": "Release-tested vLLM-Omni image bundling the Cosmos3 omni handlers."
      }
    }
  },
  "omni": {
    "tasks": [
      {
        "id": "t2v",
        "description": "Text → video (audio muxed into the MP4)",
        "curl": "curl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"prompt=$(cat assets/example_t2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=123\" \\\n  --output output.mp4\n"
      },
      {
        "id": "i2v",
        "description": "Image → video (temporally coherent, audio muxed in)",
        "curl": "curl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"input_reference=@assets/example_i2v_input.jpg\" \\\n  -F \"prompt=$(cat assets/example_i2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=1111\" \\\n  --output output.mp4\n"
      }
    ]
  },
  "features": {},
  "opt_in_features": [],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 39,
      "description": "BF16 weights — single-GPU H200 omni generation"
    }
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\n**Cosmos3-Nano** is the compact (16B) member of NVIDIA's Cosmos3 family of\n**omnimodal world models**. Built on a Mixture-of-Transformers (MoT)\narchitecture — an autoregressive tower for discrete-token generation paired\nwith a diffusion tower for continuous modalities — it generates coherent text,\nimages, video, audio, and action commands from combinations of text, image,\nvideo, and action-trajectory inputs. It targets Physical AI: world\nunderstanding, world simulation, future prediction, and action reasoning.\n\nThis recipe covers the **vLLM-Omni** serving path (video + muxed-audio\ngeneration). The same checkpoint also serves a text/multimodal **Reasoner**\nendpoint via standard vLLM — see [Reasoner mode](#reasoner-mode-multimodal-understanding)\nbelow.\n\n## Prerequisites\n\n- 1× H200 (or H100 / A100) for the documented single-GPU profile\n- The release-tested `vllm/vllm-omni:cosmos3` container, or vLLM-Omni installed\n  on top of final `vllm==0.33.0` for this source checkout. The published\n  Cosmos3 container and its historical results use an older runtime and do\n  not qualify the current source. The separate Reasoner environment below,\n  where present, retains its own plugin/runtime requirements.\n- Prompts should be JSON-upsampled for best quality (see\n  [cosmos-framework prompt upsampling](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/prompt_upsampling.md))\n\n## Launch command (vLLM-Omni)\n\n```bash\ndocker pull vllm/vllm-omni:cosmos3\n\nvllm serve nvidia/Cosmos3-Nano \\\n  --omni \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --init-timeout 1800\n```\n\nTo speed up inference with additional GPUs, enable context parallelism with\n`--ulysses-degree` or switch to tensor parallelism with\n`--tensor-parallel-size`. On memory-constrained GPUs, `--enable-layerwise-offload`\nreduces VRAM usage at a performance cost.\n\n### Example: text-to-video\n\n```bash\ncurl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"prompt=$(cat assets/example_t2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=123\" \\\n  --output output.mp4\n```\n\nImage-to-video uses the same endpoint with an `input_reference` file part.\nAudio is generated jointly and muxed into the output MP4 (48 kHz stereo AAC) —\nthere is no separate audio endpoint.\n\n## Reasoner mode (multimodal understanding)\n\nThe same checkpoint serves a text/multimodal **Reasoner** for world\nunderstanding, future prediction, and action reasoning via the standard `vllm`\npackage (not vLLM-Omni):\n\n```bash\n# CUDA 13 drivers:\nuv pip install --torch-backend=cu130 \"vllm==0.21.0\" \\\n  \"vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3\" \\\n  openai\n# CUDA 12.8 drivers: use --torch-backend=cu128 \"vllm==0.19.1\"\n\nCUDA_VISIBLE_DEVICES=0 \\\nvllm serve nvidia/Cosmos3-Nano \\\n  --hf-overrides '{\"architectures\": [\"Cosmos3ReasonerForConditionalGeneration\"]}' \\\n  --tensor-parallel-size 1 \\\n  --mm-encoder-tp-mode data \\\n  --async-scheduling \\\n  --allowed-local-media-path / \\\n  --media-io-kwargs '{\"video\": {\"num_frames\": -1}}' \\\n  --port 8000\n```\n\nThen query the OpenAI-compatible `/v1/chat/completions` endpoint with text +\nimage/video content.\n\n## References\n\n- [Model card](https://huggingface.co/nvidia/Cosmos3-Nano)\n- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)\n- [cosmos-framework](https://github.com/NVIDIA/cosmos-framework)\n- [Cosmos](https://github.com/nvidia/cosmos)\n",
  "source": {
    "source": "vllm-omni",
    "repo": "vllm-project/vllm-omni",
    "ref": "main",
    "sha": "02286829737a54167e890185bcbb6948ed9afbcf",
    "path": "recipes/models/nvidia/Cosmos3-Nano.yaml",
    "edit_url": "https://github.com/vllm-project/vllm-omni/edit/main/recipes/models/nvidia/Cosmos3-Nano.yaml",
    "overlays": []
  }
}