14. Phase 12 — Multimodal AI, Fine-Tuning & Open Models
These are important, but learn them after the core stack.
14.1 Multimodal AI
- Vision: image understanding, document screenshots, diagrams and UI analysis.
- Audio: speech-to-text, text-to-speech and voice workflows.
- Document intelligence: combine layout extraction, OCR, vision and structured outputs.
- Multimodal evaluation: compare extracted facts against ground truth; test image quality and edge cases.
- Privacy and media retention considerations.
Definition. A multimodal model accepts and/or produces more than one type of content — text, images, audio — within a single architecture, rather than requiring separate specialized models glued together. Document intelligence in particular often combines several techniques from earlier chapters: OCR/layout extraction (Chapter 7's ingestion step) feeding into a vision-capable model, whose output is then constrained by the structured-output pattern (Chapter 5).
14.2 Fine-Tuning: When to Use It
🚧 Fine-tuning is not the default solution
Fine-tuning is not the default solution for missing current facts or private documents. Those are usually retrieval/tool problems (Chapters 7–8). Fine-tuning is more useful for behavior/style/task specialization, structured patterns, domain-specific classification/extraction or improving smaller models when you have high-quality labeled examples.
Before Fine-Tuning, Try
- Better task definition and prompting.
- Structured outputs.
- Few-shot examples.
- Retrieval/RAG for knowledge.
- Tool calling for live/system data.
- A different base model.
- Evaluation-driven error analysis.
This ordered checklist reflects a real cost/complexity hierarchy: prompting and structured outputs are cheap to iterate on; RAG and tool calling require moderate engineering; fine-tuning requires labeled data, training infrastructure, and evaluation before/after — the most expensive option, reserved for when cheaper approaches provably fail.
14.3 Fine-Tuning Topics
- Dataset design and train/validation/test split.
- Label quality and leakage.
- Supervised fine-tuning (SFT).
- Parameter-efficient methods such as LoRA/QLoRA conceptually.
- Function-calling/tool-use fine-tuning.
- Evaluation before/after tuning.
- Overfitting, catastrophic forgetting and safety concerns.
Prerequisite Concept — LoRA/QLoRA (Conceptual)
Definition. Full fine-tuning updates all of a model's (potentially billions of) parameters, requiring significant compute and memory. LoRA (Low-Rank Adaptation) instead freezes the original weights and trains small additional "adapter" matrices inserted into the model, dramatically reducing the number of trainable parameters and the compute/memory needed. QLoRA combines this with quantization (representing the frozen base weights in lower precision) to further reduce memory requirements, making fine-tuning feasible on much more modest hardware.
Catastrophic Forgetting
Definition. A risk where fine-tuning a model heavily on a narrow new task degrades its previously good performance on other, unrelated tasks/knowledge — the model "forgets" general capability while specializing. This is one reason evaluation before and after fine-tuning (on both the target task and a broader capability check) is listed as mandatory.
14.4 Open-Weight / Local Models
- Model sizes, quantization and hardware requirements.
- Local inference runtimes (learn one).
- Serving endpoints and batching.
- Latency/throughput vs model quality.
- Privacy/control benefits vs infrastructure complexity.
- License and commercial-use constraints.
- Benchmark on your own eval set — not only public leaderboards.
Hosted vs Open-Weight — Comparison
| Aspect | Hosted proprietary API | Open-weight / self-hosted |
|---|
| Infrastructure | None — managed by provider | You manage GPUs/serving infrastructure |
| Data control | Depends on provider's retention policy | Full control; data never leaves your environment |
| Fine-tuning flexibility | Limited to provider's fine-tuning API, if offered | Full control over training process |
| Cost model | Pay per token/call | Pay for infrastructure regardless of usage (until scale favors it) |
| Operational burden | Low | High — serving, scaling, monitoring all your responsibility |
⚠️ Common Mistake
Trusting public leaderboard rankings as a proxy for how a model will perform on your specific task. Public benchmarks measure general capability on their own test distributions; your production task may stress entirely different skills (e.g. following a narrow extraction schema) where leaderboard rank and real task performance diverge significantly. Always benchmark candidates on your own eval set (Chapter 11).
14.5 PyTorch Depth for Applied AI
Know tensors, devices, model loading/inference, basic training loops and how to read model code. Go deeper into optimization, distributed training, CUDA and model architecture only if you want to move toward ML Engineering or model training (see Chapter 22, Option C).
Common Questions
Q: When is fine-tuning actually the right call?
When the problem is genuinely about behavior/style/format specialization or classification on a narrow domain with high-quality labeled data — not when the problem is missing current facts or private documents, which RAG solves more cheaply and updatably.
Q: What's the practical benefit of LoRA over full fine-tuning?
It trains a small number of additional parameters instead of the full model, drastically cutting compute/memory requirements while still adapting model behavior meaningfully for many tasks.
Q: Why benchmark an open-weight model on your own eval set instead of trusting a public leaderboard?
Leaderboards measure general capability on their own benchmarks, which may not reflect performance on your specific task/schema/domain — only your own eval set tells you how the model will actually perform in your product.
Interview / Viva Questions
Q: Give two reasons fine-tuning is not the default fix for "the model doesn't know our internal data."
Fine-tuning is slow and expensive to update whenever data changes, and it does not reliably teach discrete new facts the way retrieval does — RAG keeps knowledge external, current, and citable.
Q: What is catastrophic forgetting, and how do you guard against it?
It's the degradation of a model's general/prior capabilities after aggressive fine-tuning on a narrow task; guard against it by evaluating both the target task and broader capability before and after tuning, and by preferring parameter-efficient methods (LoRA) that touch fewer weights.
Q: What trade-off does choosing a self-hosted open-weight model involve?
Greater control over data privacy and fine-tuning flexibility, at the cost of owning all serving/scaling/monitoring infrastructure that a hosted API provider would otherwise manage.
Chapter Summary
- Multimodal capability lets one model architecture handle text, vision, and audio together.
- Fine-tuning is reserved for behavior/style/domain specialization — try prompting, structured outputs, RAG, and tool calling first.
- LoRA/QLoRA make fine-tuning feasible on modest hardware by training small adapter matrices instead of the full model.
- Catastrophic forgetting requires evaluating both target-task and general capability before/after fine-tuning.
- Choosing hosted vs open-weight models is a trade-off between operational simplicity and data/fine-tuning control.
- Always benchmark candidate models on your own eval set, not public leaderboards alone.