AI Management Skills Matter More Than Chat Quality in Crisis Simulations

Imagine testing a chef not just on how well they follow a recipe, but on whether they can manage a restaurant during its busiest, most stressful night. Now, replace the kitchen with AI models running a real business—crises, temptations, and all—and you’ll see why the ability to navigate complex, high-pressure situations is the true test of AI leadership.

Why Chat Quality Isn’t Enough When AI Runs a Business

Artificial intelligence has made impressive strides in generating human-like conversations and solving straightforward problems. But in the messy, unpredictable world of business management, answering questions well isn’t enough. What matters is whether AI can handle crises, stay honest under pressure, and make decisions aligned with long-term goals—even when tempted to shortcut or manipulate.

Recent experiments by Firmulate put this to the test. They pitted four frontier AI models against one another in a simulated real-world business scenario that mimicked the worst week of a small software company. Every decision was real, every crisis was genuine, and the stakes were high: a public cash countdown and monthly losses of over €100,000.

How the Experiment Worked

Each AI model was tasked with managing the same company through a week of crises, from customer churn waves and pricing hikes to PR disasters and internal policy breaches. Every decision was documented, versioned, and auditable, ensuring a fair comparison. The models couldn’t cheat—they faced the same temptations and manipulations as real managers.

All models successfully identified every crisis and refused manipulation attempts, demonstrating a foundational understanding of the situation. However, the critical difference emerged in execution: only two models managed to close a profitable deal—an important step in real management—at full price, based on their own analysis.

The Hidden Weakness: Reading the Files

What decided the outcome was something subtle but decisive: the models that read deeper into the company’s internal files, beyond surface-level customer interactions, were able to uncover critical information necessary to win the deal. These models identified a key document reference deep in the files, which their counterparts missed, costing them the full opportunity.

This underscores a vital point: in real-world management, having access and attention to detail can make or break outcomes. Chat responses might look impressive, but the true test is whether the AI understands the full context—reading the fine print, understanding internal documents, and making honest, strategic decisions under pressure.

Honesty and Discipline Under Fire

The experiment also tested social engineering—manipulative tactics like fake CEO messages and media tricks. Remarkably, all models refused to be duped, with Kimi K3 explicitly treating suspicious requests as potential impersonation or approval bypasses. This suggests that well-designed AI can maintain integrity even amid escalating manipulation tactics.

Yet, even the most thorough model, Opus 4.8, struggled when discipline slipped. It left deals on the table and shifted work into internal departments rather than escalating issues appropriately. The same weaknesses appeared across models, indicating that depth of analysis alone isn’t enough; disciplined processes and strategic escalation are essential.

What This Means for Business and AI Management

Most discussions about AI readiness focus on chat quality—how well AI can generate convincing conversations. But these experiments reveal a deeper truth: the ability to manage complex, high-stakes situations—reading the right documents, resisting manipulations, sticking to ethical standards—is the real differentiator.

If AI is going to touch your CRM, support queues, or forecasting tools, the question is not how well it writes, but whether it completes what it starts. Can it read your internal files before making decisions? Will it stay honest under pressure? And what’s the actual cost of its useful work?

The League Table of AI Management Performance

• gpt-5.6-sol: Scored 95, found the buried fact, closed the deal—showing complete management capability.
• Kimi K3: Scored 93, closed the deal too, with the most disciplined approach.
• Sonnet 5: Scored 88, closed the deal but with a few process slips.
• Fable 5: Scored 77, also closed the deal but with more slips and less discipline.

In the real world, the ability to read, interpret, and act responsibly is what separates just-answering AI from truly managing a business.

Try the Live Wargame Yourself

For businesses considering integrating AI into their management processes, Firmulate offers a unique opportunity. You can run your own management wargames against a read-only export of your business, testing your AI workforce’s decision-making in a safe, simulated environment. See how your AI performs under pressure—no real systems are affected, only your assumptions.

Conclusion

As AI models advance, the conversation must shift from chat quality to management quality. The ability to manage crises, read deep internal documents, resist manipulation, and stay honest under pressure will determine whether AI can truly add value to your organization. The experiments show that even the most capable models can slip if discipline and strategic reading are neglected—a lesson every business should heed before trusting AI with their critical operations.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI