Skip to main content
Software Engineering

Workshop 04 - Afternoon Session: Agent Orchestration & Safety

10 Modules
Chapter 9: Learning Objectives - Workshop 04 Afternoon: Agent Orchestration & Safety90%

Learning Objectives - Workshop 04 Afternoon: Agent Orchestration & Safety

Workshop Overview

Move beyond single agents to production multi-agent systems. By the end of this 3-hour session, you will be able to design, build, and deploy orchestrated agent workflows with proper safety controls, cost management, and observability -- the capabilities that make the difference between a demo and a production system.

Primary Learning Outcomes

By the end of this workshop, you will be able to:

1. Orchestration Architecture (Core Knowledge)

You will be able to:

  • Explain why multi-agent orchestration is necessary for complex, real-world tasks
  • Describe the four core orchestration patterns: subagent delegation, pipeline, fan-out/fan-in, and graph-based state machines
  • Identify which pattern suits a given problem (e.g. parallel research vs sequential content creation vs branching workflows)
  • Articulate the trade-offs between frameworks: Claude Code subagents, the Anthropic Agent SDK, LangGraph, and CrewAI
  • Draw an architecture diagram for a multi-agent system, including orchestrator, subagents, safety layer, and result collector

Success Criteria:

  • Can compare at least three orchestration patterns with use cases and weaknesses
  • Can recommend a framework for a given scenario with reasoning
  • Can explain how subagent delegation works in Claude Code
  • Can describe a fan-out/fan-in pattern and when to use it

2. Safety & Guardrails (Core Skills)

You will be able to:

  • Implement a token budget controller that tracks input and output tokens and raises an exception when the budget is exceeded
  • Build an iteration limiter that prevents runaway agent loops
  • Create a human-in-the-loop approval gate for sensitive tool calls (file writes, deployments, destructive actions)
  • Design an audit logger that records every LLM call, tool invocation, and approval decision in structured JSON
  • Configure Claude Code hooks (PreToolUse, PostToolUse) to enforce safety rules automatically
  • Explain the role of sandboxed execution (Docker, E2B) in preventing agents from damaging the host system

Success Criteria:

  • Working token budget controller with tracking and enforcement
  • Iteration limiter that stops agents after a configurable maximum
  • Approval gate that blocks sensitive actions until a human approves
  • Audit log file with structured JSON entries for a complete session
  • Can explain five safety controls and what happens if each is missing

3. Cost Management (Core Skills)

You will be able to:

  • Implement a model routing strategy that classifies task complexity and selects the cheapest suitable model
  • Route simple tasks (classification, formatting, extraction) to Haiku 4.5
  • Route moderate tasks (analysis, coding, synthesis) to Sonnet 4.6
  • Reserve Opus 4.8 for complex reasoning and critical architectural decisions
  • Use prompt caching to reduce costs on repeated system prompts
  • Trim conversation context to avoid paying for stale messages
  • Estimate the token cost of a multi-agent workflow before running it
  • Compare the cost of a routed approach against a naive "all Sonnet" baseline

Success Criteria:

  • Working model router that classifies and dispatches to three model tiers
  • Cost report showing tokens used per model with estimated savings
  • Can estimate the token cost of a given multi-agent workflow on paper
  • Can identify at least three cost reduction strategies

4. Parallel Execution & Synthesis (Applied Skills)

You will be able to:

  • Use Python's concurrent.futures.ThreadPoolExecutor to run subagents in parallel
  • Implement a fan-out/fan-in pattern where multiple agents search independently and results are synthesised
  • Handle partial failures: if one subagent fails, continue with the others and note the gap
  • Build a synthesis agent that combines diverse subtask results into a coherent final output
  • Deduplicate overlapping findings from parallel research agents

Success Criteria:

  • Orchestrator that runs at least 3 subtasks in parallel
  • Synthesis output that is coherent and eliminates redundancy
  • System handles one subagent failure without crashing
  • Cost report shows parallel vs sequential comparison

5. Error Recovery & Resilience (Applied Skills)

You will be able to:

  • Implement exponential backoff for transient API failures (rate limits, network errors)
  • Build model fallback logic: if the primary model fails, retry with a cheaper alternative
  • Design a circuit breaker that stops retrying after consecutive failures
  • Return partial results with clear indication of what succeeded and what failed
  • Log all retries, failures, and fallbacks for post-incident review

Success Criteria:

  • Retry logic with configurable backoff (1s, 2s, 4s)
  • Model fallback from Sonnet to Haiku on failure
  • Partial results returned with status indicators
  • Failure log with timestamps and error details

6. Production Deployment Patterns (Applied Knowledge)

You will be able to:

  • Structure a multi-agent project with proper separation of concerns (orchestrator.py, safety.py, router.py, agents.py)
  • Write tests for safety controllers, model routers, and orchestration logic
  • Use observability tools (LangSmith, Helicone) to trace and debug agent workflows
  • Configure Claude Code hooks for automated safety enforcement in real projects
  • Design an approval-gated deployment pipeline with full audit trail

Success Criteria:

  • Modular project structure with separate safety, routing, and orchestration modules
  • At least 3 passing tests covering safety and routing logic
  • Can explain how to integrate LangSmith or Helicone for production monitoring
  • Can describe the workflow for an approval-gated deployment

Detailed Skill Progression

Hour 1: Concepts and Understanding (0:00--1:00)

Knowledge:

  • Understand why single agents are insufficient for complex tasks
  • Know the four orchestration patterns and their trade-offs
  • Understand the five core safety controls (budget, limits, sandboxing, approval, audit)
  • Know the model routing strategy (Haiku / Sonnet / Opus)
  • Understand prompt caching and context efficiency
  • Recognise the risks of unguarded autonomous agents

Comprehension:

  • Can explain subagent delegation with a concrete example
  • Can describe when to use fan-out/fan-in vs pipeline
  • Can articulate why audit logging is essential, not optional
  • Can explain the cost difference between Haiku and Opus

Hour 2: Hands-On Building (1:00--2:00)

Implementation:

  • Build a subagent orchestrator with parallel execution
  • Implement token budget tracking and enforcement
  • Create an iteration limiter and test it
  • Build an approval gate for sensitive tool calls
  • Set up audit logging in structured JSON format
  • Configure Claude Code hooks for automated safety

Practice:

  • Run the orchestrator with a real task and observe parallel execution
  • Trigger the token budget limit and observe the graceful stop
  • Test the approval gate by attempting a sensitive action
  • Review the audit log to trace what happened during a run

Hour 3: Exercises, Project & Assessment (2:00--3:00)

Application:

  • Complete at least 3 of the 6 progressive exercises
  • Build a model routing orchestrator with cost reporting
  • Implement retry logic with exponential backoff
  • Design and build the final project: a production multi-agent system

Evaluation:

  • Answer conceptual questions about orchestration patterns
  • Identify bugs and missing safety controls in sample code
  • Design a multi-agent system for a given real-world scenario
  • Estimate the token cost of a multi-agent workflow

Profession-Specific Outcomes

For Technical Leads & Architects

  • Design multi-agent architectures for your team's AI workflows
  • Evaluate orchestration frameworks against your requirements
  • Set cost budgets and safety policies for agent deployments
  • Build observability into AI systems from the start

For AI/ML Practitioners

  • Implement production-grade agent orchestration in Python
  • Build custom safety controllers and model routers
  • Use LangGraph for complex workflows with branching and loops
  • Test and debug multi-agent systems systematically

For Product Managers & Strategists

  • Understand the cost structure of multi-agent systems
  • Evaluate which tasks benefit from orchestration vs single agents
  • Set realistic budgets for agent-powered features
  • Communicate safety requirements to engineering teams

For Operations & Compliance Teams

  • Understand audit logging and traceability for AI agents
  • Design approval workflows for sensitive automated actions
  • Set token budgets that balance capability with cost control
  • Review audit trails to verify agent behaviour

Assessment Criteria

Knowledge Assessment (40 points)

You will demonstrate understanding of:

  • Orchestration patterns: subagent delegation, pipeline, fan-out/fan-in (10 points)
  • Safety controls: budgets, limits, sandboxing, approval, audit (10 points)
  • Model routing and cost optimisation (10 points)
  • Error recovery: retries, fallbacks, circuit breakers (10 points)

Code Analysis (30 points)

You will successfully:

  • Identify at least 4 issues in a flawed orchestrator (15 points)
  • Design a safe orchestrator with budget handling and partial results (15 points)

System Design (30 points)

You will:

  • Design a multi-agent system for a real-world scenario (15 points)
  • Estimate and optimise the cost of a multi-agent workflow (15 points)

Bonus: Practical Implementation (20 points)

  • Build a working mini-orchestrator with parallel execution, token tracking, synthesis, and error handling

Passing Score: 70/100 (70%)

Success Indicators

Immediate (End of Session)

  • Built a working multi-agent orchestrator
  • Safety controls tested and operational
  • Model routing producing cost savings
  • Audit log capturing all agent actions
  • Passed the assessment

Short-term (1 Week)

  • Applied orchestration patterns to a real work task
  • Set up token budgets for an existing agent workflow
  • Implemented audit logging in a project
  • Shared learnings with a colleague

Long-term (1 Month)

  • Running a production multi-agent system with safety controls
  • Cost management reducing agent spending measurably
  • Audit trails supporting compliance or debugging needs
  • Evaluating advanced frameworks (LangGraph, CrewAI) for team adoption

Post-Session Goals

Immediate Next Steps

  • Review the Resources page for documentation links
  • Complete any unfinished exercises from the session
  • Apply the orchestration patterns to one real task at work
  • Set up cost monitoring for your API usage
  • Document your safety policy for agent deployments

Continuing Education

  • Explore LangGraph for complex stateful workflows
  • Try CrewAI for role-based agent teams
  • Read the MCP specification for building custom tool servers
  • Build a production dashboard for agent observability
  • Contribute to the workshop community with your experiences

Begin Workshop --> | Check Prerequisites | Back to Workshop Overview