Assessment: Agent Orchestration & Safety

Part 1: Conceptual Understanding (40 points)

Question 1: Orchestration Patterns (10 points)

Compare these three orchestration patterns: subagent delegation, pipeline, and fan-out/fan-in. For each, describe the pattern, give one practical use case, and identify a weakness.

Model Answer:

Pattern	Description	Use Case	Weakness
Subagent delegation	Parent decomposes task, spawns focused child agents in parallel	Codebase refactoring -- each subagent handles one module	Coordination overhead; synthesising diverse results can be difficult
Pipeline	Sequential stages where each feeds the next	Content creation: research -> write -> edit -> publish	No parallelism; bottlenecked by the slowest stage
Fan-out/Fan-in	Parallel search/processing with final synthesis	Multi-source research: academic, news, and technical sources searched simultaneously	Deduplication is hard; synthesis quality depends on overlap handling

---

Question 2: Safety Controls (10 points)

List five safety controls that a production agent system must implement. For each, explain why it is necessary and what happens if it is missing.

Expected Coverage:

1. Token budget -- prevents cost explosions from runaway loops
2. Iteration limit -- stops agents that cannot converge on an answer
3. Sandboxed execution -- prevents code execution from damaging the host system
4. Human approval gates -- prevents destructive actions (file deletion, deployment)
5. Audit logging -- enables debugging and compliance; without it, failures are invisible
6. (Also acceptable: rate limiting, model fallbacks, timeout controls, input validation)

---

Question 3: Model Routing (10 points)

Explain the model routing strategy for a cost-optimised multi-agent system. Which tasks should use Haiku, Sonnet, and Opus? Why?

Expected Answer:

• Haiku 4.5: Classification, routing, simple extraction, formatting -- tasks where speed and cost matter more than deep reasoning
• Sonnet 4.6: General agent work, code generation, analysis, synthesis -- the workhorse model with good quality-to-cost ratio
• Opus 4.8: Complex architectural decisions, nuanced reasoning, final review of critical outputs -- used sparingly because it is the most expensive
• The classifier itself should run on Haiku to avoid the cost overhead of routing

---

Question 4: Error Recovery (10 points)

Design an error recovery strategy for a multi-agent system where one subagent fails. Consider: retry logic, fallback models, partial results, and user notification.

Expected Coverage:

• Retry with backoff: Exponential backoff (1s, 2s, 4s) for transient failures (rate limits, network errors)
• Model fallback: If the primary model fails, retry with a different model (e.g. Sonnet -> Haiku)
• Partial results: If one subagent fails after retries, continue with results from successful subagents; note the gap in the synthesis
• Circuit breaker: After N consecutive failures, stop retrying and report to the user
• User notification: Clearly indicate which subtasks succeeded and which failed

---

Part 2: Code Analysis (30 points)

Question 5: Identify Issues (15 points)

Review this orchestrator code and identify at least 4 problems:

def orchestrate(task):
    subtasks = task.split(". ")

    results = []
    for st in subtasks:
        response = client.messages.create(
            model="claude-opus-4-8",
            max_tokens=4096,
            messages=[{"role": "user", "content": st}]
        )
        results.append(response.content[0].text)

    return "\n".join(results)

Issues:

1. Naive decomposition: Splitting on ". " is not intelligent task decomposition -- use an LLM to plan
2. No model routing: Uses Opus for every subtask, which is expensive and unnecessary for simple tasks
3. Sequential execution: Tasks run one after another; independent subtasks should run in parallel
4. No safety controls: No token budget, no iteration limit, no error handling
5. No synthesis: Results are just joined with newlines instead of being synthesised into a coherent answer
6. No audit logging: No way to debug failures or track costs
7. No error handling: If any API call fails, the entire function crashes

---

Question 6: Design a Safe Orchestrator (15 points)

Given these requirements, design (pseudocode is fine) an orchestrator with proper safety:

• Task: Process 10 customer support tickets
• Budget: Maximum 50,000 tokens total
• Each ticket should be classified (simple/complex) and routed appropriately
• Simple tickets: Haiku, complex tickets: Sonnet
• All actions must be logged
• If budget runs out mid-processing, return partial results

Expected Design:

INIT safety_controller(max_tokens=50000)
INIT results = []

FOR EACH ticket IN tickets:
    IF safety_controller.budget_remaining() < 1000:
        LOG "Budget low, stopping with partial results"
        BREAK

    complexity = classify(ticket, model=haiku)  # cheap classification
    safety_controller.track(classification_response)

    model = haiku IF complexity == "simple" ELSE sonnet
    
    TRY:
        result = process_ticket(ticket, model)
        safety_controller.track(result_response)
        results.append({ticket, result, model, "success"})
    CATCH BudgetError:
        results.append({ticket, "budget exceeded", model, "skipped"})
        BREAK
    CATCH Exception as e:
        results.append({ticket, str(e), model, "error"})
        LOG error

safety_controller.save_audit()
RETURN {results, cost_report, processed_count}

---

Part 3: Design Questions (30 points)

Question 7: Production Multi-Agent System (15 points)

Design a multi-agent system for: "Automated weekly codebase health report"

The system should:

• Run weekly (scheduled)
• Analyse code quality, test coverage, dependency freshness, and security
• Produce a Markdown report
• Cost less than a reasonable budget per run

Specify:

1. Agent roles and their models
2. Orchestration pattern
3. Safety controls
4. Output format

Example Answer:

Agents:
1. Code Quality Agent (Sonnet 4.6) -- runs linter, analyses complexity
2. Test Coverage Agent (Haiku 4.5) -- checks coverage percentages, identifies untested code
3. Dependency Agent (Haiku 4.5) -- checks for outdated/vulnerable dependencies
4. Security Agent (Sonnet 4.6) -- scans for common vulnerabilities
5. Synthesis Agent (Sonnet 4.6) -- combines all findings into report

Orchestration: Fan-out/Fan-in
- Agents 1-4 run in parallel (independent analyses)
- Agent 5 synthesises after all complete

Safety:
- Token budget: 30,000 tokens per run
- Timeout: 5 minutes per agent, 15 minutes total
- Audit log: stored for review
- No destructive actions (read-only)

Output: Markdown report with sections:
- Executive summary (traffic light: green/amber/red)
- Code quality metrics
- Test coverage gaps
- Dependency update recommendations
- Security findings (if any)

---

Question 8: Cost Estimation (15 points)

Estimate the token cost of this workflow and suggest optimisations:

1. Decompose task into 5 subtasks (1 Sonnet call)
2. Classify each subtask (5 Haiku calls)
3. Execute subtasks: 2 with Haiku, 2 with Sonnet, 1 with Opus
4. Synthesise results (1 Sonnet call)

Assume approximate per-call token usage: Haiku ~500 tokens, Sonnet ~1500 tokens, Opus ~2000 tokens.

Expected Analysis:

• Decomposition: 1 x 1500 = 1,500 tokens
• Classification: 5 x 500 = 2,500 tokens
• Execution: (2 x 500) + (2 x 1500) + (1 x 2000) = 1,000 + 3,000 + 2,000 = 6,000 tokens
• Synthesis: 1 x 1500 = 1,500 tokens
• Total: ~11,500 tokens

Optimisations:

1. Use Haiku for decomposition (saves ~1000 tokens at Sonnet rates)
2. Batch classification into a single Haiku call with all 5 subtasks
3. Evaluate whether the Opus subtask truly needs Opus, or if Sonnet suffices
4. Use prompt caching for the synthesis system prompt if running multiple tasks
5. Check the Anthropic pricing page for current token rates to convert to actual cost

---

Part 4: Practical Implementation (Bonus 20 points)

Question 9: Build a Mini-Orchestrator

Implement a working orchestrator that:

1. Accepts a task
2. Decomposes into subtasks
3. Executes at least 2 subtasks in parallel
4. Synthesises results
5. Reports token usage

# Your implementation here
# Must use: concurrent.futures, anthropic SDK, token tracking

Evaluation Criteria:

• Parallel execution works: 5 points
• Token tracking is accurate: 5 points
• Synthesis is coherent: 5 points
• Error handling present: 5 points

Passing Criteria

• Part 1: Minimum 28/40 points
• Part 2: Minimum 21/30 points
• Part 3: Minimum 21/30 points
• Overall: Minimum 70/100 points to pass

Navigation

• Previous: Final Project
• Next: Resources
• Back to Workshop Overview