Translate requirements into model, data, retrieval, prompt, and compliance decisions with measurable quality and traceable governance.
Elicit business, quality, latency, cost, privacy, safety, integration, and operational requirements before choosing a model or pattern.
Compare model capability, context, modalities, latency, price, regional availability, customization, and responsible-use constraints.
Assess provenance, permission, representativeness, quality, privacy, transformation, metadata, retention, and safe ingestion.
Choose embeddings, dimensionality, indexing, metadata filters, distance measures, tenancy, update behavior, and access boundaries.
Design chunking, retrieval, reranking, grounding, citations, fallback, freshness, and evaluation for RAG systems.
Use structured instructions, context boundaries, examples, output contracts, templates, versioning, testing, and approval controls.
Start with the task and its failure cost, not with a foundation model. Define who uses the system, what decision or artifact it supports, which inputs are trusted, which data may enter context, what an acceptable output looks like, and what must happen when the system is uncertain. Convert quality, latency, throughput, cost, privacy, safety, availability, integration, and regional constraints into measurable acceptance criteria before comparing models or architectures.
Use the least invasive technique that meets the need. Prompting changes instructions and presentation. Retrieval-augmented generation supplies external knowledge at request time. Tool use connects the model to deterministic capabilities. Fine-tuning or other customization changes stable model behavior and introduces a larger data, evaluation, deployment, and governance lifecycle.
| Objective | Required judgment | Proof |
|---|---|---|
| 1.1 Determine requirements and design the solution | Turn business and risk requirements into an architecture and evaluation plan | A versioned acceptance rubric covers quality, safety, latency, cost, and failure behavior |
| 1.2 Select and configure foundation models | Compare capability, modality, context, availability, latency, price, customization, and responsible-use constraints | Candidate models are evaluated on the same representative dataset |
| 1.3 Validate and process data | Establish permission, provenance, quality, privacy, transformation, retention, and representativeness | Every dataset has an owner, allowed purpose, lineage, quality checks, and deletion path |
| 1.4 Implement vector stores | Choose embedding, dimensionality, indexing, similarity, metadata, tenancy, and update behavior | Retrieval tests prove relevance, isolation, update, and deletion behavior |
| 1.5 Implement retrieval augmentation | Design chunking, retrieval, filtering, reranking, grounding, citations, fallback, and freshness | Answers are supported by retrieved evidence or safely abstain |
| 1.6 Apply prompt strategies and governance | Version instructions, context boundaries, examples, output contracts, tests, and approvals | Prompt and configuration changes are reviewable, reproducible, and reversible |
Separate task success from model fluency. A helpful-sounding response can be factually wrong, unauthorized, unsafe, too slow, too costly, or impossible to audit. Define representative tasks and critical failures. A customer-support summarizer might require source fidelity and removal of personal data. A code assistant might require repository authorization and secure output review. An agent that changes records needs deterministic authorization and human approval beyond natural-language instructions.
Decide whether the application is synchronous, streaming, asynchronous, batch, or event-driven. Record peak concurrency, response-time budget, token and data volume, Regions, languages, modalities, availability target, retry tolerance, and degradation path. Determine whether the provider may receive the data, which model and service terms apply, whether prompts or outputs may be logged, and which regulatory or contractual obligations constrain processing.
Compare models on the application's own evaluation set. Consider task quality, supported modalities, context window, output limits, structured-output behavior, latency, throughput, price, regional and cross-Region availability, invocation API compatibility, customization options, and responsible-use requirements. A larger model is not automatically the best production choice. Routing simple requests to a smaller model can improve unit economics, but routing logic must be evaluated and monitored.
Inference parameters change output behavior. Lower randomness can improve repeatability but does not guarantee correctness. Maximum-token settings constrain output length but do not control input cost. Stop sequences can shape generation but should not be the only output-safety control. Use schema or parser validation for structured data and define safe handling for malformed, incomplete, or refused output.
Inventory training, evaluation, retrieval, prompt, output, feedback, and telemetry data separately. Confirm provenance, license or permission, intended use, sensitivity, data-subject expectations, retention, geographic requirements, and deletion. Remove unnecessary fields before ingestion. Detect malformed content, duplicates, encoding problems, unsupported formats, poisoned documents, secrets, and personal data. Keep a reproducible transformation record so an evaluation or incident can be traced to the source version.
Representativeness matters. A dataset can be clean and still omit important languages, user groups, document types, edge cases, or attacks. Split evaluation data to avoid tuning on the same examples used for release approval. Restrict who can modify benchmark answers, rubrics, and hard safety cases.
An embedding model, vector store, and generation model form separate compatibility and lifecycle decisions. Choose chunk boundaries from document structure and question behavior rather than one universal size. Overlap may preserve context but increases index size and duplicate retrieval. Record embedding model and version, dimensionality, distance measure, normalization assumptions, index parameters, metadata schema, tenant boundary, encryption, backup, update, and deletion behavior.
Authorization must happen before protected content enters model context. Use identity and resource permissions plus metadata filtering or physically separate stores where needed. A prompt telling the model not to reveal another tenant's data is not an access control. Test direct and indirect cross-tenant retrieval attempts.
Evaluate retrieval separately from answer generation. Measure whether the required evidence appears in the candidate set, whether ranking puts it high enough, whether metadata filters preserve both relevance and isolation, and whether stale or deleted content disappears. Then evaluate answer support, citation accuracy, completeness, and abstention. Reranking can improve relevance but adds latency and cost. Hybrid lexical and semantic retrieval can help exact identifiers or uncommon terms.
Separate system instructions, developer-owned templates, retrieved content, user input, tool results, and examples. Mark untrusted content as data, not instructions. Give the model a clear task, constraints, allowed sources, output contract, uncertainty behavior, and prohibited actions. Few-shot examples can improve consistency but consume context and can encode unintended bias. Chain-of-thought requests are not a substitute for verifiable evidence or deterministic controls.
Version prompts with model identifiers, inference parameters, retrieval settings, tool schemas, guardrails, and evaluation results. Treat a prompt change as a release input. Require review appropriate to impact and retain rollback. Do not log raw prompts by default; design privacy-safe telemetry first.
| Need | Prefer | Why |
|---|---|---|
| Change instructions or format | Versioned prompt and output contract | Lowest lifecycle cost and easy rollback |
| Add current or private knowledge | Governed RAG | Sources remain external, updateable, and citeable |
| Perform deterministic action | Workflow or bounded tool | Authorization and validation stay outside the model |
| Stabilize repeated behavior after evidence | Customization after benchmark comparison | Higher data and governance cost must be justified |
| Retrieve protected multi-tenant content | Pre-context authorization and metadata isolation | The model cannot enforce a boundary on unseen data |
| Support exact codes plus semantic questions | Consider hybrid retrieval and reranking | Embeddings alone may miss exact rare tokens |
A legal operations team wants a contract assistant using private documents across several business units. Answers must cite current clauses, never cross unit boundaries, and return within four seconds.
