dgx-spark-playbooks/nvidia/station-sglang-inference/endpoint-test.yaml

kind: Playbook
metadata:
  name: station-sglang-inference
  displayName: LLM Inference with SGLang
  shortDescription: Serve LLMs with SGLang on DGX Station for prefix-cached multi-turn and structured output inference

  publisher: nvidia
  description: |
    # REPLACE THIS WITH YOUR MODEL CARD
    https://gitlab-master.nvidia.com/api-catalog/examples/-/blob/main/modelcard-example-mixtral8x7b.md?ref_type=heads
        
  labelsV2:
  - gpuType:playbook:gpu_type_station
  - DGX Station
  - GB300
  - Inference
  - SGLang
  - Blackwell
  - Structured Output
  - RadixAttention
  
  attributes:
  - key: DURATION
    value: 25 MIN
  
spec:
  artifactName: station-sglang-inference
  nvcfFunctionId: None
  attributes:

    showUnavailableBanner: false
    apiDocsUrl: None
    termsOfUse: |
            
    cta:
      text: View on GitHub
      url: https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/station-sglang-inference/
      

    tabs:
    - 
      id: overview
      
      label: Overview
      content: |
        # Basic idea
        
        SGLang is a high-performance serving framework for large language models, optimized for workloads where requests share common prefixes — multi-turn conversations, RAG pipelines, and agentic workflows. Its core innovation, **RadixAttention**, automatically caches and reuses KV cache entries across requests using a radix tree, eliminating redundant prefill computation. SGLang also provides best-in-class **structured output generation** (JSON, regex, grammar-constrained decoding) through its xGrammar backend, running up to 3x faster than standard guided decoding.
        
        - **RadixAttention** — Automatically reuses KV cache across requests sharing common prefixes. Multi-turn conversations and repeated system prompts skip re-computation entirely, reducing first-token latency and increasing throughput.
        - **Structured output** — Compressed finite-state machine decoding with grammar mask generation overlapped with the LLM forward pass. Produces valid JSON, regex-matched, or grammar-constrained output with minimal overhead.
        - **OpenAI-compatible API** — Drop-in replacement for OpenAI and vLLM endpoints. Supports `/v1/chat/completions`, `/v1/completions`, and `/v1/embeddings`.
        - **Blackwell optimized** — SGLang includes optimizations for SM100+ GPUs and CUDA 13, with NVFP4 GEMM support and accelerated softmax kernels.
        
        # What you'll accomplish
        
        Launch SGLang on DGX Station to serve an LLM, then exercise its two key differentiators: prefix-cached multi-turn chat and structured JSON output generation. You will also benchmark multi-turn throughput to see RadixAttention's effect.
        
        - Serve Qwen3-8B with SGLang's Blackwell-optimized backend
        - Send multi-turn conversations and observe prefix cache hits in server metrics
        - Generate structured JSON output using schema-constrained decoding
        - Benchmark multi-turn throughput with and without prefix caching
        
        # What to know before starting
        
        - Basic Docker container usage
        - Familiarity with REST APIs (curl or Python requests)
        
        # Prerequisites
        
        - NVIDIA DGX Station with GB300 GPU (Blackwell SM103)
        - Docker installed: `docker --version`
        - NVIDIA Container Toolkit configured: `nvidia-smi` should show the GB300
        - HuggingFace account with access token
        - Network access to HuggingFace and Docker Hub
        
        # Ancillary files
        
        - `assets/benchmark_multiturn.py` — Python script that benchmarks multi-turn conversation throughput and demonstrates structured output generation
        
        # Time & risk
        
        * **Duration:** 20–25 minutes (including model download)
        * **Risks:** Model download requires HuggingFace authentication
        * **Rollback:** Stop and remove the container to restore state
        * **Last Updated:** 04/06/2026
          * First Publication
                
      

    - 
      id: instructions
      
      label: Instructions
      content: |
        # Step 1. Set up Docker permissions
        
        If you haven't already, add your user to the docker group to run Docker without sudo:
        
        ```bash
        sudo usermod -aG docker $USER
        newgrp docker
        ```
        
        # Step 2. Set up environment variables
        
        ```bash
        # HuggingFace token (required)
        # Get a token from https://huggingface.co/settings/tokens
        export HF_TOKEN="your_huggingface_token"
        
        # Model to serve
        export MODEL_HANDLE="Qwen/Qwen3-8B"
        
        # Maximum context length
        export MAX_MODEL_LEN=8192
        ```
        
        # Step 3. Pull the SGLang container
        
        Pull the SGLang container image with CUDA 13.0 support (required for Blackwell SM103):
        
        ```bash
        docker pull lmsysorg/sglang:latest-cu130
        ```
        
        # Step 4. Identify the GB300 GPU
        
        On DGX Station with multiple GPUs, identify the GB300's device index:
        
        ```bash
        nvidia-smi --query-gpu=index,name --format=csv,noheader
        ```
        
        Look for the row showing `NVIDIA GB300`. Note its index (e.g., `1`).
        
        # Step 5. Start SGLang server
        
        Launch the SGLang server:
        
        ```bash
        # Replace device=1 with your GB300's index from Step 4
        docker run -d \
          --name sglang-server \
          --gpus '"device=1"' \
          --ipc host \
          --ulimit memlock=-1 \
          --ulimit stack=67108864 \
          -p 30000:30000 \
          -e HF_TOKEN="$HF_TOKEN" \
          -v "$HOME/.cache/huggingface/hub:/root/.cache/huggingface/hub" \
          lmsysorg/sglang:latest-cu130 \
          sglang serve --model-path "$MODEL_HANDLE" \
            --host 0.0.0.0 \
            --port 30000 \
            --context-length $MAX_MODEL_LEN \
            --mem-fraction-static 0.85
        ```
        
        Check the server logs:
        
        ```bash
        docker logs -f sglang-server
        ```
        
        Wait for the server to show it is ready:
        
        ```
        INFO:     Uvicorn running on http://0.0.0.0:30000
        ```
        
        Press `Ctrl+C` to exit the log view.
        
        > [!NOTE]
        > First launch downloads the model and compiles kernels. Subsequent starts are faster thanks to cached weights and compiled artifacts.
        
        # Step 6. Test basic inference
        
        Send a chat completion request using the OpenAI-compatible API:
        
        ```bash
        curl http://localhost:30000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
            "model": "'"$MODEL_HANDLE"'",
            "messages": [{"role": "user", "content": "Explain quantum computing in simple terms."}],
            "max_tokens": 256
          }'
        ```
        
        The response follows the standard OpenAI format with a `choices` array containing the model's answer.
        
        # Step 7. Multi-turn conversation with prefix caching
        
        SGLang's RadixAttention automatically caches the KV cache for processed tokens. When follow-up messages share the same conversation prefix, the cached entries are reused — skipping prefill for all previously seen tokens.
        
        Send a multi-turn conversation. The system prompt is deliberately long so the shared prefix exceeds SGLang's page size (64 tokens), which is the minimum unit for cache reuse:
        
        ```bash
        # Turn 1
        curl -s http://localhost:30000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
            "model": "'"$MODEL_HANDLE"'",
            "messages": [
              {"role": "system", "content": "You are an expert physics tutor who explains concepts clearly and concisely. You use real-world analogies and everyday examples to make abstract ideas concrete. When answering, first state the key concept in one sentence, then give a short explanation with an example."},
              {"role": "user", "content": "What is the difference between speed and velocity?"}
            ],
            "max_tokens": 256
          }' | python3 -m json.tool
        
        # Turn 2 — extends the same conversation
        curl -s http://localhost:30000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
            "model": "'"$MODEL_HANDLE"'",
            "messages": [
              {"role": "system", "content": "You are an expert physics tutor who explains concepts clearly and concisely. You use real-world analogies and everyday examples to make abstract ideas concrete. When answering, first state the key concept in one sentence, then give a short explanation with an example."},
              {"role": "user", "content": "What is the difference between speed and velocity?"},
              {"role": "assistant", "content": "Speed is a scalar quantity that measures how fast an object moves, while velocity is a vector quantity that includes both speed and direction. For example, a car driving at 60 km/h has a speed of 60 km/h regardless of where it is headed. But if that car is driving 60 km/h north, that is its velocity — change direction to south and the velocity changes even though the speed stays the same."},
              {"role": "user", "content": "Can you give me another example that shows why the distinction matters in real physics problems?"}
            ],
            "max_tokens": 256
          }' | python3 -m json.tool
        ```
        
        The second request reuses the KV cache from the shared prefix (system message + first user turn + assistant response), only computing attention for the new user message. This reduces first-token latency for follow-up turns.
        
        Check the cache hit rate in the server logs. SGLang logs each prefill batch with the number of cached tokens reused:
        
        ```bash
        docker logs sglang-server 2>&1 | grep "cached-token" | tail -10
        ```
        
        Look for `#cached-token` values greater than 0 on later turns — this confirms RadixAttention is reusing the KV cache from the shared prefix.
        
        # Step 8. Structured JSON output
        
        SGLang's constrained decoding guarantees valid JSON output matching a schema. This uses the xGrammar backend to overlap grammar mask generation with the model's forward pass, adding minimal latency.
        
        Generate a structured response:
        
        ```bash
        curl -s http://localhost:30000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
            "model": "'"$MODEL_HANDLE"'",
            "messages": [
              {"role": "user", "content": "List three programming languages with their primary use case and year created."}
            ],
            "max_tokens": 512,
            "response_format": {
              "type": "json_schema",
              "json_schema": {
                "name": "languages",
                "schema": {
                  "type": "object",
                  "properties": {
                    "languages": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "properties": {
                          "name": {"type": "string"},
                          "primary_use": {"type": "string"},
                          "year_created": {"type": "integer"}
                        },
                        "required": ["name", "primary_use", "year_created"]
                      }
                    }
                  },
                  "required": ["languages"]
                }
              }
            }
          }' | python3 -m json.tool
        ```
        
        The response content is guaranteed to be valid JSON matching the provided schema. Parse the `choices[0].message.content` field — it will contain a well-formed JSON object.
        
        # Step 9. Benchmark multi-turn throughput
        
        Run the included benchmark script to measure how prefix caching improves multi-turn latency. The script is in the `assets/` directory of this playbook.
        
        ```bash
        python3 -m venv .venv && source .venv/bin/activate
        pip install requests
        ```
        
        ```bash
        python3 assets/benchmark_multiturn.py \
          --base-url http://localhost:30000 \
          --model "$MODEL_HANDLE" \
          --num-conversations 20 \
          --turns-per-conversation 5
        ```
        
        The script sends parallel multi-turn conversations and measures:
        - **Per-turn latency** for turn 1 vs subsequent turns (shows prefix caching effect)
        - **Total throughput** in tokens per second
        - **Cache statistics** from server metrics
        
        You should see latency decrease for later turns in each conversation as the shared prefix grows, demonstrating RadixAttention's cache reuse.
        
        > [!TIP]
        > If you downloaded this playbook as a zip, the `assets/` directory is already present. If you cloned the full repository, navigate to `nvidia/station-sglang-inference/` first.
        
        # Step 10. Cleanup
        
        Stop and remove the container:
        
        ```bash
        docker stop sglang-server
        docker rm sglang-server
        ```
        
        Optionally remove the image:
        
        ```bash
        docker rmi lmsysorg/sglang:latest-cu130
        ```
                
      

    - 
      id: troubleshooting
      
      label: Troubleshooting
      content: |
        # Common issues
        
        | Symptom | Cause | Fix |
        |---------|-------|-----|
        | "permission denied" when running docker | User not in docker group | Run `sudo usermod -aG docker $USER && newgrp docker` |
        | Container fails to start with GPU error | NVIDIA Container Toolkit not configured | Run `nvidia-ctk runtime configure --runtime=docker` and restart Docker |
        | `device >= 0 && device < num_gpus INTERNAL ASSERT FAILED` | Using `--gpus all` on a multi-GPU system | Use `--gpus '"device=N"'` to target the GB300 specifically (check index with `nvidia-smi`) |
        | "Token is required" or 401 error | Missing HuggingFace token | Ensure `HF_TOKEN` is exported before running the docker command |
        | Server exits with OOM error | Model too large for available GPU memory | Lower `--mem-fraction-static` (e.g., 0.7) or reduce `--context-length`. Check GPU memory with `nvidia-smi` |
        | `json_schema` response_format returns error | SGLang version too old | Ensure you are using `lmsysorg/sglang:latest-cu130`. Older versions may not support `json_schema` format |
        | Server starts but CUDA errors on inference | Wrong CUDA version for Blackwell | Use the `latest-cu130` image tag. SM103 requires CUDA 13.0+ |
        | Model runs on wrong GPU | Default GPU selection | Use `--gpus '"device=N"'` to select the GB300 specifically |
        | Slow first request after server start | Kernel JIT compilation | First request triggers kernel compilation. Subsequent requests are fast. Wait ~30 seconds |
        | Connection refused on port 30000 | Server still loading model | Check `docker logs sglang-server` — wait for the Uvicorn startup message |
        | `/server_info` shows no cache stats | Endpoint may differ across versions | Try `curl http://localhost:30000/v1/models` to verify the server is responsive. Cache metrics may be under `/metrics` (requires `--enable-metrics` server flag) |
        
        > [!NOTE]
        > On DGX Station, the GB300 is typically device 1 (device 0 is the RTX Pro 6000 workstation GPU). Always verify with `nvidia-smi --query-gpu=index,name --format=csv,noheader`.
                
      


    resources:
    - name: SGLang (GitHub)
      url: https://github.com/sgl-project/sglang
      

    - name: SGLang Documentation
      url: https://docs.sglang.io/
      

    - name: SGLang OpenAI API Reference
      url: https://docs.sglang.io/basic_usage/openai_api_completions.html