Best Small Models for Fine Tuning

X:fine_tune_small_language_models_LLM_SLM

Best models to fine-tune

Ranking: 1. llama-3-8b 2. phi-3-4k 3. zephyr-7b-beta 4. llama-3-8b-instruct 5. mistral-7b

Source: The Fine-tuning Index - Predibase

Related Webinar

https://www.youtube.com/watch?v=kFdfDJ1fQxY

https://pbase.ai/3x4JB6G

In this on-demand webinar, Staff Software Engineer Justin Zhao, and ML Engineer Timothy Wang, lead an in-depth discussion on our findings:

• How we fine-tuned open-source LLMs that rival GPT-4 • How you can implement Parameter-Efficient Fine-Tuning (PEFT) methods like Low Rank Adaptation (LoRA) • Which tasks are best suited for fine-tuning based on our benchmarks • Which popular LLMs—namely Phi, Gemma, and Mistral—perform best and worst across tasks • How we implemented an evaluation harness for fine-tuning at scale