Meta-Llama-3-8B_pct_ortho

This model is a fine-tuned version of unsloth/llama-3-8b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.2219

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.02
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
2.2594	0.0206	8	2.2433
2.2616	0.0412	16	2.2377
2.2041	0.0618	24	2.2343
2.2802	0.0824	32	2.2533
2.2832	0.1030	40	2.2618
2.2484	0.1236	48	2.2566
2.2735	0.1442	56	2.2577
2.293	0.1648	64	2.2784
2.2518	0.1854	72	2.2818
2.2922	0.2060	80	2.2914
2.3199	0.2266	88	2.2720
2.3807	0.2472	96	2.2788
2.3528	0.2678	104	2.2831
2.3144	0.2884	112	2.2958
2.3652	0.3090	120	2.2950
2.3637	0.3296	128	2.2845
2.3014	0.3502	136	2.2781
2.3067	0.3708	144	2.2830
2.3242	0.3914	152	2.2788
2.3184	0.4120	160	2.2659
2.3574	0.4326	168	2.2782
2.3006	0.4532	176	2.2733
2.3082	0.4738	184	2.2699
2.3097	0.4944	192	2.2615
2.3003	0.5150	200	2.2649
2.3027	0.5356	208	2.2594
2.3262	0.5562	216	2.2491
2.3118	0.5768	224	2.2599
2.2904	0.5974	232	2.2623
2.2519	0.6180	240	2.2495
2.2907	0.6386	248	2.2526
2.2864	0.6592	256	2.2512
2.242	0.6798	264	2.2492
2.2941	0.7004	272	2.2415
2.2799	0.7210	280	2.2383
2.2881	0.7416	288	2.2358
2.2797	0.7621	296	2.2381
2.3197	0.7827	304	2.2255
2.2507	0.8033	312	2.2284
2.236	0.8239	320	2.2313
2.2667	0.8445	328	2.2200
2.2763	0.8651	336	2.2255
2.2915	0.8857	344	2.2229
2.2554	0.9063	352	2.2211
2.2237	0.9269	360	2.2241
2.2446	0.9475	368	2.2229
2.2926	0.9681	376	2.2224
2.2813	0.9887	384	2.2219

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1