Fine-tune your first large language model (LLM) with LoRA, llama.cpp, and KitOps in 5 easy steps

Getting started with LLMs can be intimidating, in this tutorial we will show you how to fine-tune a large language model using LoRA, facilitated by tools like llama.cpp and KitOps.

LoRA, or Low-Rank Adaptation, is a technique for efficiently adapting pre-trained models with minimal computational overhead. This tutorial aims to guide you through the setup, adaptation process, and deployment using easy-to-follow steps.

Prerequisites

Before we get started, you’ll need to install Kit CLI and download an executable from Llama.

• Kit CLI installed on your system. Installation Guide
• finetune executable from the llama.cpp project. Download here

1.Environment setup

In step 1, we will, set up your environment:

1. Create a project directory:

   mkdir /lora_finetuning
   

2. Log in to the GitHub Container Registry using the Kit CLI. This will require your GitHub username and password Alternatively, you can use any other registry that is compatible with OCI artifacts:

   kit login ghcr.io
   

2. Create your Kitfile and refer to your base model

1. In your project directory (/lora_finetuning), create a Kitfile with the following contents:

   manifestVersion: "1.0"
   package:
   name: llama3 fine-tuned
   version: 3.0.0
   authors: ["Jozu AI"]
   model:
   name: llama3-8B-instruct-q4_0
   path: ghcr.io/jozu-ai/llama3:8B-instruct-q4_0
   description: Llama 3 8B instruct model
   license: Apache 2.0
   

[!NOTE] Notice how the path refers to another ModelKit? In this case ghcr.io/jozu-ai/llama3:8B-instruct-q4_0.

1. Let's create our initial ModelKit by invoking.

   kit pack /lora_finetuning -t fine-tuning:untuned
   

2. Now let's unpack our ModelKit to our work folder. Notice that kit CLI will also unpack the model that you have referenced to your work folder so that you can work with it.

   kit unpack fine-tuning:untuned -d /lora_finetuning --overwrite
   

3. Create your Lora Adapter

1. Prepare your training data and store it in a text file, ideally using a common string to separate each example that isn't found in your training data, e.g., "<s>".

    Example one text here.
    Example two text here.
   

2. Run the fine-tuning command from your project directory:

   cd /lora_finetuning
   llama.cpp\finetune.exe --model-base ./llama3-8B-instruct-q4_0.gguf --train-data ./training-data.txt --threads 8 --sample-start "" --lora-out lora_adapter.gguf
   

manifestVersion: "1.0"
package:
  name: llama3 fine-tuned
  version: 3.0.0
  authors: ["Jozu AI"]
model:
  name: llama3-8B-instruct-q4_0
  path: ghcr.io/jozu-ai/llama3:8B-instruct-q4_0
  description: Llama 3 8B instruct model
  license: Apache 2.0
  parts:
    - path: ./lora-adapter.gguf
      type: lora-adapter
datasets:
  - name: fine-tune-data
    path: ./training-data.txt