Comparing Low-Cost and High-Quality LLM Training Data Approaches

You don't need a 2 trillion parameter model to tell you the capital of France is Paris. Be smart and route between a panel of models according to query difficulty and model specialty! New paper proposes a framework to train a router that routes queries to the appropriate LLM to optimize the trade-off b/w cost vs. performance. Overview: Model inference cost varies significantly: Per one million output tokens: Llama-3-70b ($1) vs. GPT-4-0613 ($60), Haiku ($1.25) vs. Opus ($75) The RouteLLM paper propose a router training framework based on human preference data and augmentation techniques, demonstrating over 2x cost saving on widely used benchmarks. They define the problem as having to choose between two classes of models: (1) strong models - produce high quality responses but at a high cost (GPT-4o, Claude3.5) (2) weak models - relatively lower quality and lower cost (Mixtral8x7B, Llama3-8b) A good router requires a deep understanding of the question’s complexity as well as the strengths and weaknesses of the available LLMs. Explore different routing approaches: - Similarity-weighted (SW) ranking - Matrix factorization - BERT query classifier - Causal LLM query classifier Neat Ideas to Build From: - Users can collect a small amount of in-domain data to improve performance for their specific use cases via dataset augmentation. - Can expand this problem from routing between a strong and weak LLM to a multiclass model routing approach where we have specialist models(language vision model, function calling model etc.) - Larger framework controlled by a router - imagine a system of 15-20 tuned small models and the router as the n+1'th model responsible for picking the LLM that will handle a particular query at inference time. - MoA architectures: Routing to different architectures of a Mixture of Agents would be a cool idea as well. Depending on the query you decide how many proposers there should be, how many layers in the mixture, what the aggregate models should be etc. - Route based caching: If you get redundant queries that are slightly different then route the query+previous answer to a small model to light rewriting instead of regenerating the answer

Understanding LLM Adaptation: Full Training vs Fine-Tuning vs LoRA/QLoRA 🚀 Which approach really moves the needle in today’s AI landscape? As large language models (LLMs) become mainstream, I frequently get asked: “Should we train from scratch, full fine-tune, or use LoRA/QLoRA adapters for our use case?” Here’s a simple breakdown based on real-world considerations: 🔍 1. Full Training from Scratch What: Building a model from the ground up with billions of parameters. Who: Only major labs/Big Tech (OpenAI, Google, etc.) Cost: 🏦 Millions—requires massive clusters and huge datasets. Why: Needed ONLY if you want a truly unique model architecture or foundation. 🛠️ 2. Full Fine-Tuning What: Take an existing giant model and update ALL its weights for your task. Who: Advanced companies with deep pockets. Cost: 💰 Tens of thousands to millions—need multiple high-end GPUs. Why: Useful if you have vast domain data and need to drastically “re-train” the model’s capabilities. ⚡ 3. LoRA/QLoRA (Parameter-Efficient Tuning) What: Plug low-rank adapters into a model, updating just 0.5-5% of weights. Who: Startups, researchers, almost anyone! Cost: 💡 From free (on Google Colab) to a few hundred dollars on cloud GPUs. Why: Customize powerful LLMs efficiently—think domain adaption, brand voice, or private datasets, all without losing the model’s general smarts. 🤔 Which one should YOU use? For most organizations and projects, LoRA/QLoRA is the optimal sweet spot Fast: Results in hours, not weeks Affordable: Accessible to almost anyone Flexible: Update or revert adapters with ease Full fine-tuning and from-scratch training make sense only for the biggest players—99% of AI innovation today leverages parameter-efficient tuning! 💬 What’s your experience? Are you using full fine-tunes, or has LoRA/QLoRA met your business needs? Share your project (or frustrations!) in comments.

The bottleneck isn't GPUs or architecture. It's your dataset. Three ways to customize an LLM: 1. Fine-tuning: Teaches behavior. 1K-10K examples. Shows how to respond. Cheapest option. 2. Continued pretraining: Adds knowledge. Large unlabeled corpus. Extends what model knows. Medium cost. 3. Training from scratch: Full control. Trillions of tokens. Only for national AI projects. Rarely necessary. Most companies only need fine-tuning. How to collect quality data: For fine-tuning, start small. Support tickets with PII removed. Internal Q&A logs. Public instruction datasets. For continued pretraining, go big. Domain archives. Technical standards. Mix 70% domain, 30% general text. The 5-step data pipeline: 1. Normalize. Convert everything to UTF-8 plain text. Remove markup and headers. 2. Filter. Drop short fragments. Remove repeated templates. Redact PII. 3. Deduplicate. Hash for identical content. Find near-duplicates. Do before splitting datasets. 4. Tag with metadata. Language, domain, source. Makes dataset searchable. 5. Validate quality. Check perplexity. Track metrics. Run small pilot first. When your dataset is ready: All sources documented. PII removed. Stats match targets. Splits balanced. Pilot converges cleanly. If any fail, fix data first. What good data does: Models converge faster. Hallucinate less. Cost less to serve. The reality: Building LLMs is a data problem. Not a training problem. Most teams spend 80% of time on data. That's the actual work. Your data is your differentiator. Not your model architecture. Found this helpful? Follow Arturo Ferreira.

One ongoing challenge in leveraging LLMs for data annotation is the implications of compute cost as data volumes scale. But, with recent model releases, this is starting to change. --- At Deasie, we ran a data annotation experiment to compare classification performance of GPT-4o with it's cheaper counterparts: Gemini 1.5 Flash Preview and GPT-4o mini. 🔬 Set-up: We assessed the ability for each model to correctly extract the presence or absence of clauses from ~180 complex legal documents, measuring accuracy, recall, F1 and precision in every case (e.g., Was a minimum commitment term specified? Was an exclusivity clause included?) 📊 Results: For simpler metadata attributes, the performance across the 3 models was comparable. For classifications that were slightly more nuanced, we observed a marginal drop in all performance metrics for both GPT-4o Mini and Gemini 1.5 Flash Preview compared to GPT-4o. --- Despite the performance delta, we believe that the respective 7x and 33x cost differentials of Gemini 1.5 Flash Preview and GPT-4o Mini compared to GPT-4o will lead to increasing opportunity to support LLM-based classification at scale, without compromising on performance. With the recent release of Llama 3.1, this will only become more true.