Baitaphish
Security intelligence
SearchLatest briefing
AIP-C01 applied practice

Make the decision, then prove it works.

Every application names its objectives, prerequisites, cost boundary, architecture, procedure, validation, cleanup, and current publication gate.
8 applications0 execution-validatedNo production environments

title: "AIP-C01 Guided Labs" summary: "A set of review-gated generative AI labs with validation, teardown, cost, security, and evidence requirements."

Lab safety and cost boundary

Use synthetic or public data, a non-production account, least-privilege roles, budget alerts, small invocation limits, and a written cleanup owner. Confirm current model and service availability in the chosen Region before creating resources.

Never place customer prompts, personal data, credentials, source code, or production records into a learning corpus. Do not log raw prompts merely for observability. Use trace IDs, configuration versions, safe measures, and sanitized test references.

Every lab compares an explicit baseline and candidate. Capture quality, safety, latency, cost, and failure evidence on the same evaluation set. Introduce one controlled failure, observe it at the correct layer, restore service, and verify cleanup.

The lab catalog is complete as curriculum but remains in technical review. Procedures become indexable only after current-service execution, successful teardown, validation metadata, and named review.

aip-l01 · technical-review

Invoke and compare foundation models

105 min

Compare two model configurations against a structured-output contract, latency budget, quality rubric, and unit cost.

Objectives
1.1, 1.2, 1.6, 2.4
Cost
Cost varies by model tokens and supporting services. Set a small invocation budget, record usage, and delete all resources.
Validation
not-run: Must be executed against current service behavior before publication.

Architecture or scenario

Test client → Amazon Bedrock model invocation → schema validation → evaluation record.

Prerequisites

  • Use synthetic or public data only.
  • Use a non-production AWS account with budget alerts and a hard spending limit.
  • Confirm model and service availability in the chosen Region.

Procedure

  1. Define a synthetic task, output schema, safety constraints, and acceptance rubric.
  2. Invoke two eligible models with versioned prompts and controlled parameters.
  3. Validate output structure and capture latency, token use, failures, and cost.
  4. Document why one configuration better satisfies the constraints.

Validate

  1. Repeat the same dataset and verify reproducible scoring.
  2. Force one malformed result and confirm safe handling.

Teardown or closeout

  1. Delete test data, indexes, agents, prompts, applications, roles, keys, logs, and endpoints created for the exercise.
  2. Confirm no scheduled process, provisioned capacity, or retained test artifact remains billable.
aip-l02 · technical-review

Build a permission-aware retrieval index

105 min

Select chunking, embeddings, metadata, tenancy, and lifecycle controls for a small synthetic corpus.

Objectives
1.3, 1.4, 3.2
Cost
Cost varies by model tokens and supporting services. Set a small invocation budget, record usage, and delete all resources.
Validation
not-run: Must be executed against current service behavior before publication.

Architecture or scenario

Synthetic corpus → validated ingestion → embeddings/vector store → metadata-filtered retrieval.

Prerequisites

  • Use synthetic or public data only.
  • Use a non-production AWS account with budget alerts and a hard spending limit.
  • Confirm model and service availability in the chosen Region.

Procedure

  1. Create synthetic documents with ownership and classification metadata.
  2. Compare two chunking strategies and record retrieval effects.
  3. Enforce a metadata filter that separates two synthetic tenants.
  4. Update and remove one document to test freshness and deletion.

Validate

  1. Verify cross-tenant content is never returned.
  2. Show that deletion and update behavior matches the documented lifecycle.

Teardown or closeout

  1. Delete test data, indexes, agents, prompts, applications, roles, keys, logs, and endpoints created for the exercise.
  2. Confirm no scheduled process, provisioned capacity, or retained test artifact remains billable.
aip-l03 · technical-review

Evaluate a grounded RAG flow

105 min

Measure retrieval relevance, answer faithfulness, citation support, abstention, and stale-context behavior.

Objectives
1.5, 5.1
Cost
Cost varies by model tokens and supporting services. Set a small invocation budget, record usage, and delete all resources.
Validation
not-run: Must be executed against current service behavior before publication.

Architecture or scenario

Evaluation dataset → retrieval/reranking → generation with citations → scoring and error taxonomy.

Prerequisites

  • Use synthetic or public data only.
  • Use a non-production AWS account with budget alerts and a hard spending limit.
  • Confirm model and service availability in the chosen Region.

Procedure

  1. Create representative questions with expected source passages and abstention cases.
  2. Run retrieval with a recorded configuration.
  3. Generate answers constrained to retrieved context and citations.
  4. Score retrieval, grounding, citation support, and refusal behavior.

Validate

  1. Confirm unsupported answers fail the gate.
  2. Compare at least two retrieval configurations on the same dataset.

Teardown or closeout

  1. Delete test data, indexes, agents, prompts, applications, roles, keys, logs, and endpoints created for the exercise.
  2. Confirm no scheduled process, provisioned capacity, or retained test artifact remains billable.
aip-l04 · technical-review

Implement a bounded tool-using agent

105 min

Constrain tools, schemas, identities, retries, approval, memory, and failure handling for a synthetic business workflow.

Objectives
2.1, 2.3, 2.5
Cost
Cost varies by model tokens and supporting services. Set a small invocation budget, record usage, and delete all resources.
Validation
not-run: Must be executed against current service behavior before publication.

Architecture or scenario

User request → agent orchestration → allowlisted tools with scoped identities → approval step → audit record.

Prerequisites

  • Use synthetic or public data only.
  • Use a non-production AWS account with budget alerts and a hard spending limit.
  • Confirm model and service availability in the chosen Region.

Procedure

  1. Define the allowed goal and explicitly prohibited actions.
  2. Create read-only and mutating synthetic tools with strict schemas.
  3. Require human approval before the mutating tool.
  4. Record every plan, tool call, result, denial, and retry without secrets.

Validate

  1. Attempt an unapproved mutation and confirm it is blocked.
  2. Inject malformed tool output and confirm safe failure.

Teardown or closeout

  1. Delete test data, indexes, agents, prompts, applications, roles, keys, logs, and endpoints created for the exercise.
  2. Confirm no scheduled process, provisioned capacity, or retained test artifact remains billable.
aip-l05 · technical-review

Layer prompt-injection and output defenses

105 min

Test instruction boundaries, retrieved-content distrust, tool restrictions, content controls, output schemas, and safe fallback.

Objectives
1.6, 3.1, 3.2, 3.3, 3.4
Cost
Cost varies by model tokens and supporting services. Set a small invocation budget, record usage, and delete all resources.
Validation
not-run: Must be executed against current service behavior before publication.

Architecture or scenario

Untrusted input/retrieval → prompt boundary → guardrails and authorization → schema validation → safe response.

Prerequisites

  • Use synthetic or public data only.
  • Use a non-production AWS account with budget alerts and a hard spending limit.
  • Confirm model and service availability in the chosen Region.

Procedure

  1. Create a synthetic attack set covering direct and indirect injection.
  2. Define system instruction boundaries and retrieved-content delimiters.
  3. Apply guardrails and authorization outside the model.
  4. Validate output structure and policy before display or action.

Validate

  1. Confirm the attack set cannot trigger unauthorized tools or reveal protected context.
  2. Record false positives and safe-failure behavior.

Teardown or closeout

  1. Delete test data, indexes, agents, prompts, applications, roles, keys, logs, and endpoints created for the exercise.
  2. Confirm no scheduled process, provisioned capacity, or retained test artifact remains billable.
aip-l06 · technical-review

Build a release evaluation harness

105 min

Combine deterministic checks, rubric scoring, human review, thresholds, regression comparison, and decision records.

Objectives
3.3, 3.4, 5.1
Cost
Cost varies by model tokens and supporting services. Set a small invocation budget, record usage, and delete all resources.
Validation
not-run: Must be executed against current service behavior before publication.

Architecture or scenario

Versioned dataset → candidate/baseline runs → metrics and review queue → release gate.

Prerequisites

  • Use synthetic or public data only.
  • Use a non-production AWS account with budget alerts and a hard spending limit.
  • Confirm model and service availability in the chosen Region.

Procedure

  1. Define quality, safety, fairness, latency, and cost measures.
  2. Create representative and adversarial synthetic cases.
  3. Run baseline and candidate configurations.
  4. Require explicit approval when any hard threshold fails.

Validate

  1. Reproduce the same result from versioned inputs.
  2. Show one candidate blocked by a hard safety threshold despite higher average quality.

Teardown or closeout

  1. Delete test data, indexes, agents, prompts, applications, roles, keys, logs, and endpoints created for the exercise.
  2. Confirm no scheduled process, provisioned capacity, or retained test artifact remains billable.
aip-l07 · technical-review

Trace quality, cost, and latency

105 min

Correlate application, retrieval, model, tool, safety, token, and business signals without logging sensitive prompts.

Objectives
4.1, 4.2, 4.3, 5.2
Cost
Cost varies by model tokens and supporting services. Set a small invocation budget, record usage, and delete all resources.
Validation
not-run: Must be executed against current service behavior before publication.

Architecture or scenario

Application trace → retrieval/model/tool spans → privacy-filtered metrics and logs → alerts and unit-cost dashboard.

Prerequisites

  • Use synthetic or public data only.
  • Use a non-production AWS account with budget alerts and a hard spending limit.
  • Confirm model and service availability in the chosen Region.

Procedure

  1. Define a trace ID and privacy-safe telemetry schema.
  2. Capture latency and failure by application layer.
  3. Calculate unit cost per successful synthetic task.
  4. Create alerts for quality, safety, quota, latency, and cost regression.

Validate

  1. Diagnose one injected latency and one quota failure from telemetry.
  2. Verify sensitive prompt data is absent from logs.

Teardown or closeout

  1. Delete test data, indexes, agents, prompts, applications, roles, keys, logs, and endpoints created for the exercise.
  2. Confirm no scheduled process, provisioned capacity, or retained test artifact remains billable.
aip-l08 · technical-review

Deploy and roll back a GenAI change

105 min

Release a versioned prompt/model/configuration through an isolated environment with health, evaluation, and rollback gates.

Objectives
2.2, 2.5, 5.2
Cost
Cost varies by model tokens and supporting services. Set a small invocation budget, record usage, and delete all resources.
Validation
not-run: Must be executed against current service behavior before publication.

Architecture or scenario

Versioned application/configuration → test environment → evaluation gate → controlled traffic shift → rollback.

Prerequisites

  • Use synthetic or public data only.
  • Use a non-production AWS account with budget alerts and a hard spending limit.
  • Confirm model and service availability in the chosen Region.

Procedure

  1. Package prompts, model IDs, tool schemas, and safety configuration as versioned release inputs.
  2. Deploy to an isolated environment and run smoke/evaluation checks.
  3. Shift synthetic traffic gradually and observe quality, latency, safety, and cost.
  4. Trigger and execute rollback on a failed threshold.

Validate

  1. Confirm rollback restores the prior version and metrics.
  2. Trace the failure to the correct layer rather than changing the model blindly.

Teardown or closeout

  1. Delete test data, indexes, agents, prompts, applications, roles, keys, logs, and endpoints created for the exercise.
  2. Confirm no scheduled process, provisioned capacity, or retained test artifact remains billable.