Suzume

This Suzume 8B, a multilingual finetune of Llama 3 (meta-llama/Meta-Llama-3-8B-Instruct).

Llama 3 has exhibited excellent performance on many English language benchmarks. However, it also seemingly been finetuned on mostly English data, meaning that it will respond in English, even if prompted in other languages.

We have fine-tuned Llama 3 on almost 90,000 multilingual conversations meaning that this model has the smarts of Llama 3 but has the added ability to chat in more languages.

Please feel free to comment on this model and give us feedback in the Community tab!

We will release a paper in the future describing how we made the training data, the model, and the evaluations we have conducted of it.

How to use

The easiest way to use this model on your own computer is to use the GGUF version of this model (lightblue/suzume-llama-3-8B-multilingual-gguf) using a program such as jan.ai or LM Studio.

If you want to use this model directly in Python, we recommend using vLLM for the fastest inference speeds.

from vllm import LLM, SamplingParams

sampling_params = SamplingParams(temperature=0.0, max_tokens=100)
llm = LLM(model="lightblue/suzume-llama-3-8B-multilingual")

messages = []
messages.append({"role": "user", "content": "Bonjour!"})
prompt = llm.llm_engine.tokenizer.tokenizer.apply_chat_template(conversation=messages, add_generation_prompt=True, tokenize=False)
prompts = [prompt]

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Evaluation scores

We achieve the following MT-Bench scores across 6 languages:

	meta-llama/Meta-Llama-3-8B-Instruct	lightblue/suzume-llama-3-8B-multilingual	Nexusflow/Starling-LM-7B-beta	gpt-3.5-turbo
German 🇩🇪	NaN	7.26	6.99	7.68
French 🇫🇷	NaN	7.66	7.29	7.74
Japanese 🇯🇵	NaN	6.56	6.22	7.84
Russian 🇷🇺 *	NaN	8.19	8.28	7.94
Chinese 🇨🇳	NaN	7.11	6.97	7.55
English 🇺🇸	7.98	7.73	7.92	8.26

* (Note the Russian scores exclude code, reasoning and math problems due to not having any translated reference answers for these questions.)

We observe minimal degredation of Llama 3's English ability while achieving best-in-class multilingual abilities compared to the top rated 7B model (Nexusflow/Starling-LM-7B-beta) on the Chatbot Arena Leaderboard.

Here is our evaluation script.

Training data

We train on three sources of data to create this model:

lightblue/tagengo-gpt4 - 76,338 conversations
- A diverse dataset of initial inputs sampled from lmsys/lmsys-chat-1m and then used to prompt gpt-4-0125-preview
megagonlabs/instruction_ja - 669 conversations
- A hand-edited dataset of nearly 700 Japanese conversations taken originally from translations of the kunishou/hh-rlhf-49k-ja dataset.
openchat/openchat_sharegpt4_dataset - 6,206 conversations
- Multilingual conversations of humans talking to GPT-4.

We prepare our data like so:

import pandas as pd
from datasets import Dataset, load_dataset, concatenate_datasets

### Tagengo
gpt4_dataset = load_dataset("lightblue/tagengo-gpt4", split="train")
gpt4_dataset = gpt4_dataset.filter(lambda x: x["response"][1] == "stop")
####

### Megagon
megagon_df = pd.read_json(
    "https://raw.githubusercontent.com/megagonlabs/instruction_ja/main/data/data.jsonl",
    lines=True,
    orient="records"
    )
role_map = {"user": "human", "agent": "gpt"}
megagon_df["conversations"] = megagon_df.utterances.apply(lambda x: [{"from": role_map[y["name"]], "value": y["text"]} for y in x])
megagon_df["language"] = "Japanese"
megagon_df = megagon_df[["conversations", "language"]]
megagon_dataset = Dataset.from_pandas(df)
###

### Openchat
openchat_df = pd.read_json("https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset/resolve/main/sharegpt_gpt4.json?download=true")
openchat_df["conversations"] = openchat_df["items"]
openchat_dataset = Dataset.from_pandas(openchat_df)
###


dataset = concatenate_datasets([gpt4_dataset, megagon_dataset, openchat_dataset])
dataset = dataset.filter(lambda x: not any([y["value"] is None for y in x["conversations"]]))
dataset.select_columns(["conversations"]).to_json("/workspace/llm_training/axolotl/llama3-multilingual/tagengo_openchat_megagon.json")

workspace/llm_training/axolotl/llama3-multilingual/output_tagengo_openchat_megagon_8B_llama3

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the above described dataset. It achieves the following results on the evaluation set:

Loss: 0.6595

Training procedure

See axolotl config

axolotl version: 0.4.0

base_model: meta-llama/Meta-Llama-3-8B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer  # PreTrainedTokenizerFast

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: /workspace/llm_training/axolotl/llama3-multilingual/tagengo_openchat_megagon.json
    ds_type: json # see other options below
    type: sharegpt
    conversation: llama-3
dataset_prepared_path: /workspace/llm_training/axolotl/llama3-multilingual/prepared_tagengo_openchat_megagon
val_set_size: 0.01
output_dir: /workspace/llm_training/axolotl/llama3-multilingual/output_tagengo_openchat_megagon_8B_llama3

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

use_wandb: true
wandb_project: wandb_project
wandb_entity: wandb_entity
wandb_name: wandb_name

gradient_accumulation_steps: 2
micro_batch_size: 2
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 5
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed: /workspace/axolotl/deepspeed_configs/zero2.json
weight_decay: 0.0
special_tokens:
  pad_token: <|end_of_text|>

Note - we added this Llama 3 template to fastchat directly as the Llama 3 chat template was not supported when we trained this model.

from fastchat.conversation import Conversation
from fastchat.conversation import register_conv_template
from fastchat.conversation import SeparatorStyle

register_conv_template(
    Conversation(
        name="llama-3",
        system_template="<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}",
        roles=("<|start_header_id|>user<|end_header_id|>\n", "<|start_header_id|>assistant<|end_header_id|>\n"),
        sep_style=SeparatorStyle.ADD_NEW_LINE_SINGLE,
        sep="<|eot_id|>",
        stop_token_ids=[128009],
        stop_str="<|eot_id|>",
    )
)

Training hyperparameters

This model was trained using 4 x A100 (80GB) for roughly 2.5 hours.

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 2
eval_batch_size: 2
seed: 42
distributed_type: multi-GPU
num_devices: 4
gradient_accumulation_steps: 2
total_train_batch_size: 16
total_eval_batch_size: 8
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 10
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.1894	0.0	1	1.0110
0.8493	0.2	73	0.7057
0.8047	0.4	146	0.6835
0.7644	0.6	219	0.6687
0.7528	0.8	292	0.6615
0.7794	1.0	365	0.6595

Framework versions

Transformers 4.38.2
Pytorch 2.2.1+cu121
Datasets 2.18.0
Tokenizers 0.15.0

Developer

Peter Devine - (ptrdvn)

Zoyd
/

suzume-llama-3-8B-multilingual-3_5bpw-exl2