dgx-spark-playbooks/nvidia/unsloth

Unsloth on DGX Spark

Optimized fine-tuning with Unsloth

Table of Contents

• Overview
• Instructions

---

Overview

Basic idea

• Performance-first: It claims to speed up training (e.g. 2× faster on single GPU, up to 30× in multi-GPU setups) and reduce memory usage compared to standard methods.
• Kernel-level optimizations: Core compute is built with custom kernels (e.g. with Triton) and hand-optimized math to boost throughput and efficiency.
• Quantization & model formats: Supports dynamic quantization (4-bit, 16-bit) and GGUF formats to reduce footprint, while aiming to retain accuracy.
• Broad model support: Works with many LLMs (LLaMA, Mistral, Qwen, DeepSeek, etc.) and allows training, fine-tuning, exporting to formats like Ollama, vLLM, GGUF, Hugging Face.
• Simplified interface: Provides easy-to-use notebooks and tools so users can fine-tune models with minimal boilerplate.

What you'll accomplish

You'll set up Unsloth for optimized fine-tuning of large language models on NVIDIA Spark devices, achieving up to 2x faster training speeds with reduced memory usage through efficient parameter-efficient fine-tuning methods like LoRA and QLoRA.

What to know before starting

• Python package management with pip and virtual environments
• Hugging Face Transformers library basics (loading models, tokenizers, datasets)
• GPU fundamentals (CUDA/GPU vs CPU, VRAM constraints, device availability)
• Basic understanding of LLM training concepts (loss functions, checkpoints)
• Familiarity with prompt engineering and base model interaction
• Optional: LoRA/QLoRA parameter-efficient fine-tuning knowledge

Prerequisites

• NVIDIA Spark device with Blackwell GPU architecture
• nvidia-smi shows a summary of GPU information
• CUDA 13.0 installed: nvcc --version
• Internet access for downloading models and datasets

Ancillary files

The Python test script can be found here on GitHub

Time & risk

Duration: 30-60 minutes for initial setup and test run

Risks:

• Triton compiler version mismatches may cause compilation errors
• CUDA toolkit configuration issues may prevent kernel compilation
• Memory constraints on smaller models require batch size adjustments

Rollback: Uninstall packages with pip uninstall unsloth torch torchvision.

Instructions

Step 1. Verify prerequisites

Confirm your NVIDIA Spark device has the required CUDA toolkit and GPU resources available.

nvcc --version

The output should show CUDA 13.0.

nvidia-smi

The output should show a summary of GPU information.

Step 2. Get the container image

docker pull nvcr.io/nvidia/pytorch:25.09-py3

Step 3. Launch Docker

docker run --gpus all --ulimit memlock=-1 -it --ulimit stack=67108864 --entrypoint /usr/bin/bash --rm nvcr.io/nvidia/pytorch:25.09-py3

Step 4. Install dependencies inside Docker

pip install transformers peft datasets "trl==0.19.1"
pip install --no-deps unsloth unsloth_zoo

Step 5. Build and install bitsandbytes inside Docker

pip install --no-deps bitsandbytes

Step 6. Create Python test script

Curl the test script here into the container.

curl -O https://gitlab.com/nvidia/dgx-spark/temp-external-playbook-assets/dgx-spark-playbook-assets/-/blob/main/${MODEL}/assets/test_unsloth.py

We will use this test script to validate the installation with a simple fine-tuning task.

Step 7. Run the validation test

Execute the test script to verify Unsloth is working correctly.

python test_unsloth.py

Expected output in the terminal window:

• "Unsloth: Will patch your computer to enable 2x faster free finetuning"
• Training progress bars showing loss decreasing over 60 steps
• Final training metrics showing completion

Step 8. Next steps

Test with your own model and dataset by updating the test_unsloth.py file:

## Replace line 32 with your model choice
model_name = "unsloth/Meta-Llama-3.1-8B-bnb-4bit"

## Load your custom dataset in line 8
dataset = load_dataset("your_dataset_name")

## Adjust training parameter args at line 61
per_device_train_batch_size = 4
max_steps = 1000

Visit https://github.com/unslothai/unsloth/wiki for advanced usage instructions, including:

• Saving models in GGUF format for vLLM
• Continued training from checkpoints
• Using custom chat templates
• Running evaluation loops