Fine-tune a model

This guide explains how to fine-tune a large language model using Runpod and Axolotl. You’ll learn how to select a base model, configure your training environment, and start the fine-tuning process.

Prerequisites

Before you begin fine-tuning, ensure you have:

• A Runpod account with access to the Fine Tuning feature
• (Optional) A Hugging Face access token for gated models

Select a base model

To start fine-tuning, you’ll need to choose a base model from Hugging Face:

1. Navigate to the Fine Tuning section in the sidebar

2. Enter the Hugging Face model ID in the Base Model field

   • Example: NousResearch/Meta-Llama-3-8B

3. For gated models (requiring special access):

   1. Generate a Hugging Face token with appropriate permissions
   2. Add your token in the designated field

Select a dataset

You can choose a dataset from Hugging Face for fine-tuning:

1. Browse available datasets on Hugging Face
2. Enter your chosen dataset identifier in the Dataset field
   • Example: tatsu-lab/alpaca

Deploy the fine-tuning pod

Follow these steps to set up your training environment:

1. Click Deploy the Fine Tuning Pod

2. Select a GPU instance based on your model’s requirements:

   • Smaller models: Choose GPUs with less memory
   • Larger models/datasets: Choose GPUs with higher memory capacity

3. Monitor the system logs for deployment progress

4. Wait for the success message: "You've successfully configured your training environment!"

Connect to your training environment

After your pod is deployed and active, you can connect using any of these methods:

1. Go to your Fine Tuning pod dashboard

2. Click Connect and choose your preferred connection method:

   • Jupyter Notebook: Browser-based notebook interface
   • Web Terminal: Browser-based terminal
   • SSH: Local machine terminal connection

To use SSH, add your public SSH key in your account settings. The system automatically adds your key to the pod’s authorized_keys file.

Configure your environment

Your training environment includes this directory structure in /workspace/fine-tuning/:

• examples/: Sample configurations and scripts
• outputs/: Training results and model outputs
• config.yaml: Training parameters for your model

The system generates an initial config.yaml based on your selected base model and dataset.

Review and modify the configuration

The config.yaml file controls your fine-tuning parameters. Here’s how to customize it:

1. Open the configuration file:

   nano config.yaml
   

2. Review and adjust the parameters based on your specific use case

Here’s an example configuration with common parameters:

base_model: NousResearch/Meta-Llama-3.1-8B

# Model loading settings
load_in_8bit: false
load_in_4bit: false
strict: false

# Dataset configuration
datasets:
  - path: tatsu-lab/alpaca
    type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./outputs/out

# Training parameters
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

# Weights & Biases logging (optional)
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

# Training optimization
gradient_accumulation_steps: 8
micro_batch_size: 1
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5

# Additional settings
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
evals_per_epoch: 2
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: <|end_of_text|>

The config.yaml file contains all hyperparameters needed for fine-tuning. You may need to iterate on these settings to achieve optimal results.

For more configuration examples, visit the Axolotl examples repository.

Start the fine-tuning process

Once your configuration is ready, follow these steps:

1. Start the training process:

   axolotl train config.yaml
   

2. Monitor the training progress in your terminal

Push your model to Hugging Face

After completing the fine-tuning process, you can share your model:

1. Log in to Hugging Face:

   huggingface-cli login
   

2. Create a new repository on Hugging Face if needed

3. Upload your model:

   huggingface-cli upload <your-username>/<model-name> ./output
   

Replace <your-username> with your Hugging Face username and <model-name> with your desired model name.

Additional resources

For more information about fine-tuning with Axolotl, see:

• Axolotl Documentation