title: "AIP-C01 Decision Cheat Sheet" summary: "A compact decision framework for selecting, securing, evaluating, operating, and governing generative AI systems on AWS."

Production decision sequence

  1. Define the task, users, failure cost, quality, safety, privacy, latency, cost, availability, and evidence requirements.
  2. Establish data provenance, permission, classification, retention, and deletion.
  3. Choose prompting, retrieval, customization, workflow, or bounded agency from measured need.
  4. Enforce identity, authorization, tenancy, tool permissions, approval, and budgets outside the model.
  5. Version every release input and evaluate it against a representative baseline.
  6. Observe quality, safety, latency, cost, quota, and business outcomes together.
  7. Roll back the complete release unit when a hard gate fails.

Use the matrices by hiding the recommendation and defending it from constraints. Alter one variable—freshness, action impact, latency, data sensitivity, human review, or regional support—and explain how the design changes.

Internal raw scores are recall and reasoning signals only. They do not predict AWS scaled scores, and the provisional question pools remain noindex until named technical review.

Prompt, RAG, or customization

Choose the least invasive pattern that meets quality and governance needs.

Prompt, RAG, or customization: Choose the least invasive pattern that meets quality and governance needs.
NeedPreferReason
Behavior/instructionsPrompt and output contractFast, versionable, no model training
Fresh private knowledgeRAGKeeps source content external and citeable
Stable task behavior/style at scaleFine-tuning/customization after evaluationHigher lifecycle and governance cost
Actions across systemsWorkflow or bounded agentRequires authorization and failure controls outside the model
Objectives: 1.1, 1.2, 1.5, 1.6

Retrieval design

Treat ingestion, retrieval, generation, and evaluation as separate failure domains.

Retrieval design: Treat ingestion, retrieval, generation, and evaluation as separate failure domains.
DecisionTradeoffMeasure
Chunk size/overlapContext completeness vs noise/costRecall, precision, answer support
Embedding modelSemantic quality vs cost/compatibilityRetrieval benchmark
Metadata filtersAuthorization/focus vs missed contextTenant leakage and relevance
RerankingQuality vs latency/costTop-k relevance and end-to-end task success
Objectives: 1.3, 1.4, 1.5

Agent or workflow

Use deterministic orchestration unless autonomy materially improves the task.

Agent or workflow: Use deterministic orchestration unless autonomy materially improves the task.
ConditionPatternMandatory control
Known sequenceWorkflow/state machineExplicit transitions and retries
Ambiguous plan, bounded toolsAgentTool allowlist, scoped identity, limits, observation
Irreversible/high-impact actionHuman approvalAuthoritative check outside model
Untrusted tool resultValidate and constrainSchema, provenance, size, and policy checks
Objectives: 2.1, 2.3, 2.5

Safety control placement

Do not ask the model to enforce controls that belong in deterministic systems.

Safety control placement: Do not ask the model to enforce controls that belong in deterministic systems.
RiskControl locationEvidence
Prompt injectionInput/context boundaries plus external authorizationAttack-set results and denied actions
Harmful outputGuardrail plus output policy/schemaSafety metrics and review samples
Data leakageIdentity, filtering, tenancy, encryption, minimizationAccess tests and privacy-safe traces
Excessive agencyTool permissions, budgets, approvals, stop conditionsTool audit and blocked mutation
Objectives: 3.1, 3.2, 3.3, 3.4

Evaluation selection

Select metrics that reflect the real failure cost.

Evaluation selection: Select metrics that reflect the real failure cost.
TargetMethodCaution
Structured outputDeterministic parser/schemaPassing structure does not prove correctness
GroundingCitation and entailment checks plus human sampleModel judges can share bias
Task qualityRepresentative rubric and human reviewAverage can hide critical failures
Release decisionBaseline comparison plus hard safety thresholdsNever trade away a hard control for mean quality
Objectives: 5.1, 3.4

Operational optimization

Optimize end-to-end task success rather than tokens in isolation.

Operational optimization: Optimize end-to-end task success rather than tokens in isolation.
SymptomInvestigatePotential response
High latencyRetrieval, context, model, streaming, tools, concurrencyReduce unnecessary context, cache safely, change orchestration
High costTokens, retries, model mix, cache hit rate, failed tasksRoute by complexity and fix retries/failures
Quality regressionDataset drift, prompt/model/config/data changesCompare release inputs and roll back
Quota failureConcurrency, retry storm, service quota, backpressureQueue, rate limit, request quota, degrade safely
Objectives: 4.1, 4.2, 4.3, 5.2