Managing the Synthetic Workforce: How to Lead AI Teams

You know how to manage people. But how do you manage code that acts like people? A guide for the first generation of 'AI Managers'.

Kevan Shah · · 14 min read

By 2030, the average manager will have 3 human direct reports and 30 AI direct reports. We call this the Synthetic Workforce. Managing AI requires a completely different skillset than managing humans. You don't need empathy; you need logic, structure, and observability.

1. Observability is the New "Standup"

You don't ask an agent "What are you working on?" You look at its logs.

  • The Dashboard: Every AI employee needs a real-time dashboard showing:
    • Throughput (Tasks/hour).
    • Success Rate (Completed vs. Failed).
    • Cost per Task (Token usage).
  • Alerting: Set up thresholds. If your AI SDR books 0 meetings in 24 hours, something is broken. If it books 500, it's hallucinating. Get an alert immediately.

2. Continuous Evaluation (The "Feedback Loop")

AI drifts. A prompt that worked yesterday might fail today if the underlying model changes or data inputs shift.

  • Golden Datasets: Maintain a set of 50 perfect examples (Input -> Output). Run your AI against this set every week (Regression Testing) to ensure it hasn't gotten "stupid."
  • Human-in-the-Loop (HITL): Randomly sample 1% of your AI's work for human review. If the quality drops, pause the agent and refine the prompt.

3. Permissions and Security (The "Least Privilege" Rule)

Never give an AI "Admin" access.

  • Scoped API Keys: Your coding agent should have access to the specific repo it's working on, not your entire GitHub organization.
  • Budget Caps: Hard limit OpenAI/Anthropic API spend. An infinite loop of agents talking to agents can bankrupt you overnight.

4. The Human-AI Culture

How do your human staff feel about their digital colleagues?

  • The Threat: Humans fear replacement.
  • The Reframe: Position AI as an "Intern" or "Assistant" that handles the grunt work.
  • Celebration: When the AI closes a deal, celebrate the human who managed that agent. "Great job, Sarah, on tweaking the SDR Agent's pitch!"

The Future Manager

The manager of the future is 50% Psychologist (for humans) and 50% Systems Architect (for AI). Start building those systems muscles now.