5. Phase 3 — Model APIs, Prompting & Structured Outputs

Estimated time: 2 weeks.

5.1 Use Raw Model SDKs Before Frameworks

Build direct integrations first. This forces you to understand messages, instructions, model responses, tool calls, errors, tokens, latency and provider-specific behavior. Frameworks make much more sense after this layer is clear.

⚠️ Common Mistake Learning a high-level orchestration framework before ever calling a model API directly. When the framework misbehaves — and every framework eventually does — you need to understand what is happening underneath to debug it. Frameworks are addressed properly starting in Chapter 9 (LangGraph), deliberately after this raw-API phase.

5.2 Capabilities to Practice

Prerequisite Concept — System vs User Content

Definition. A system (or "developer") instruction is trusted text written by the engineer that configures the model's behavior; user content is untrusted text supplied by the end user or pulled from external documents. Keeping these separate is not just a code-organization habit — it is a security boundary, since a model can be manipulated by instructions hidden inside user-supplied or retrieved content (a threat formally covered as "prompt injection" in Chapter 12).

Streaming Responses — Under the Hood

Client Server Model API │ POST /chat │ │ │ ─────────────────────────▶│ │ │ │ request (stream=true) │ │ │ ────────────────────────▶│ │ │◀───── token chunk 1 ─────│ │◀── SSE: "The" ───────────│ │ │ │◀───── token chunk 2 ─────│ │◀── SSE: " capital" ───────│ │ │ ... │ ... │ │◀── SSE: [DONE] ───────────│◀───── stream end ─────────│
Fig 5.1 — Streaming sends partial output as it's generated (commonly via Server-Sent Events), reducing perceived latency for the end user even though total generation time is unchanged.

Usage Accounting

Providers typically report input tokens, output tokens, and (increasingly) cached tokens — repeated prompt prefixes that the provider can reuse from a previous call at reduced cost/latency. Tracking these per request, per feature, and per user/tenant is what makes the cost dashboards in Chapter 11 (Observability) and Chapter 13 (Cost Engineering) possible.

Retries, Timeouts, Rate Limits, and Fallback Models

Why it matters. Model APIs are external network services with their own outages, rate limits, and transient errors. A production system must set a timeout on every call, retry only transient failures (e.g. 429/503) with exponential backoff, and — where product requirements allow — fall back to a secondary model or a cached/degraded response rather than failing the whole feature outright. This is the same reliability discipline as any external API integration, detailed further in Chapter 13.

5.3 Prompt and Context Engineering

💡 Interview Tip If asked "What is prompt engineering, really?", a strong answer avoids treating it as a mystical skill: "It's the practice of specifying a task's instructions, context, examples, and output format precisely enough that model behavior becomes reliable and testable — it's one input to a larger system that also needs validation, retrieval, and evaluation to be trustworthy in production."

Prerequisite Concept — Few-Shot Examples

Definition. A few-shot example is a sample input/output pair included directly in the prompt to demonstrate the desired behavior or format, as opposed to "zero-shot" prompting (instructions only, no examples). Examples are powerful for pinning down format and edge-case handling, but each one consumes tokens (cost and context budget) — hence "use examples only when they improve reliability," not by default.

5.4 Structured Output Pattern

User / API input │ ▼ Prompt + JSON Schema │ ▼ Model │ ▼ Validated Pydantic object │ ▼ Business rules │ ▼ Database / downstream service
Fig 5.2 — The structured-output pattern. Note the Pydantic validation step between the raw model response and any business logic — this is non-negotiable.
🚧 Critical Distinction Never treat generated text as trusted executable data. Parse, validate, authorize and constrain outputs before they reach real systems. A model returning JSON that looks well-formed is not the same as JSON that is valid against your schema and business rules — always validate explicitly (e.g. with Pydantic) rather than trusting the raw response.

Worked Example — Extraction with Validation

class ExtractedInvoice(BaseModel): vendor: str total: float = Field(gt=0) invoice_date: date currency: Literal["USD", "EUR", "GBP", "NPR"] response = call_model(prompt, response_schema=ExtractedInvoice) try: invoice = ExtractedInvoice.model_validate_json(response.text) except ValidationError as e: # route to human review queue instead of silently trusting bad data log_and_flag_for_review(response.text, e)
Fig 5.3 — A model that returns an invalid currency code or negative total fails validation and is routed to a human review queue rather than reaching the database — this is the safety net structured outputs provide.

5.5 Project: Document Information Extractor

Real-World Example

Many document-heavy back-office workflows — invoice processing at an accounting firm, resume screening at a recruiting agency, receipt reconciliation for expense management — follow this exact loop: extract → validate → route uncertain cases to a human → feed corrections back into the eval set. This pattern, sometimes called "human-in-the-loop extraction," is one of the highest-ROI, lowest-risk applications of LLMs in production because a human remains the final authority over any data that reaches financial or compliance systems.

Common Questions

Q: Why build with raw SDKs before adopting a framework?
Frameworks abstract away exactly the details (message structure, token accounting, error handling) you need to understand to debug production issues; learning the raw API first means the framework becomes a convenience, not a black box.
Q: What happens if a tool/model call times out?
It should be caught explicitly, retried if the failure is transient, and either fall back to a secondary model/cached response or surface a clear error — never left to hang indefinitely, which would exhaust server resources under load.
Q: How would this scale to thousands of concurrent extraction requests?
Move extraction to an async job queue (Chapter 7/13) rather than a synchronous request/response call, so uploads return immediately with a job ID and the client polls or receives a webhook/event when extraction completes.

Interview / Viva Questions

Q: Why should retrieved or user-supplied content never be placed in the "system" role?
The system role is treated by the model (and by developers) as trusted, high-priority instructions; mixing untrusted content into it increases the risk that malicious or manipulative content is followed as if it were a legitimate instruction (prompt injection).
Q: What is the risk of trusting model-generated JSON without schema validation?
The model can produce malformed, incomplete, or maliciously-shaped output (wrong types, unexpected fields, injected values) that could corrupt a database or trigger unintended business logic if it reaches downstream systems unchecked.
Q: Why version prompts like code?
Because prompt changes can silently change model behavior across many users; treating prompts as versioned artifacts with an evaluation gate (Chapter 11) prevents unnoticed regressions in production quality.
Q: When should you use a fallback model?
When the primary model/provider is unavailable, rate-limited, or too slow for a latency-sensitive path, and the product can tolerate a slightly lower-quality (or different-cost) response rather than a hard failure.

Chapter Summary