In 2005, a freestyle chess tournament attracted grandmasters, supercomputers, and everyone in between. The rules were simple: any combination of humans and computers could compete. Chess purists expected the grandmasters with their cutting-edge hardware to dominate. Technology enthusiasts predicted pure chess engines would crush human opponents.
Both groups were wrong.
The winners were Steven Cramton and Zackary Stephen, two amateur players from New Hampshire with chess ratings that wouldn’t qualify them for most local tournaments. Using three consumer-grade PCs, they defeated grandmasters partnered with military-grade supercomputers. They beat pure chess engines running on hardware that cost more than most people’s houses. Their secret wasn’t chess mastery or computational power—it was knowing how to orchestrate their AI tools, when to trust which engine, and which strategic questions to explore.
As Garry Kasparov later observed, “Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.”
That was 2005. Twenty years later, we finally have data on what AI can actually do in the economy. And the pace of change should both terrify and intrigue every middle manager reading this.
The New Benchmark That Changes Everything
In September 2025, OpenAI introduced something called GDPval—a benchmark that measures AI performance not on abstract reasoning or exam-style questions, but on actual economically valuable work. Real tasks created by professionals with an average of 14 years of experience across 44 occupations: legal briefs, engineering blueprints, customer support conversations, sales forecasts, medical assessments, marketing plans.
These aren’t toy problems. They’re the tasks that contribute $3 trillion annually to the U.S. economy. The tasks that define knowledge work. The tasks that, until very recently, we assumed required human expertise, judgment, and years of training.
When GDPval launched in September, the best AI models were matching human experts roughly 50% of the time. Impressive, but still clearly behind human performance overall.
Then, just three months later in December 2025, OpenAI’s GPT-5.2 model achieved something remarkable: a 70.9% win or tie rate against human experts. In ninety days, AI jumped from parity to clear superiority on professional knowledge work tasks. And it does so eleven times faster and at one-hundredth the cost.
If you’re a middle manager responsible for tasks that can be clearly defined, measured, and evaluated—you should be paying attention.
The Number That Should Terrify You (Because of How Fast It’s Moving)
Seventy percent. That’s past the tipping point. That’s not “AI is getting there” or “AI shows promise.” That’s AI demonstrably outperforming humans on most professional knowledge work tasks that can be tested.
But here’s what should really get your attention: the speed. Three months ago, AI was at rough parity with humans. Now it’s clearly superior. That’s not a gradual slope—that’s a vertical climb.
The economic pressure is immediate and real. When you can get expert-level output in minutes instead of days, at a fraction of the cost, why wouldn’t you automate? The spreadsheet practically writes itself: 100x cost reduction, 11x speed improvement, 70% reliability. For any CFO looking at that math, the decision seems obvious.
But here’s where the story gets interesting—and where the chess lesson from 2005 becomes critically important.
Because hidden in that 70.9% success rate is a catastrophic failure mode that changes everything.
The Plot Twist: When AI Fails, It Fails Spectacularly
GDPval’s analysis revealed something that should make every executive pause before clicking “automate everything.” The 70.9% figure doesn’t tell the whole story.
Here’s what matters: of the AI outputs evaluated, roughly 27% were classified as “bad”—meaning not fit for use—and 3% were classified as “catastrophic”—meaning they could cause actual harm if deployed.
But here’s the more subtle issue: even within the 70% that win or tie with human experts, the quality isn’t uniform. A deliverable might be 70% excellent and 30% flawed. An AI-generated legal brief might nail seven arguments but miss a critical precedent. An engineering blueprint might specify correct dimensions but overlook a safety requirement. A customer service response might be 80% perfect but include one sentence that violates company policy.
Think about what that means in practice. Imagine you’re a law firm that starts using AI for brief writing. Most briefs look great—professional, well-researched, properly formatted. But buried in some of them are mistakes that could get your client sanctioned or your firm disciplined. The AI doesn’t flag these errors. It presents everything with the same confidence.
Or you’re a hospital deploying AI for patient documentation. Most notes are thorough and accurate. But occasionally, critical information is omitted or mischaracterized—and nothing in the system signals which notes need extra scrutiny.
This isn’t like having a junior employee who needs supervision. When humans make mistakes, they’re usually within a reasonable margin of error. They might miss something or make a suboptimal choice, but they rarely produce work that’s fundamentally dangerous. AI, by contrast, generates outputs that look confident and competent—until they catastrophically aren’t.
The math suddenly looks different. It’s not just about whether AI can do the work. It’s about whether you can afford the cost of catching the failures before they cause harm.
The Question That Should Keep You Up At Night
So here’s the real question: if AI can handle 70% of professional work tasks but includes subtle or catastrophic flaws that look indistinguishable from quality work—what does that make your job?
The answer is both sobering and liberating: your job becomes judgment.
Not judgment in the sense of deciding whether AI is “good” or “bad.” Not judgment as superiority or gatekeeping. Judgment as the practical skill of evaluating quality, accuracy, and appropriateness—the ability to look at an output and determine whether it’s actually fit for purpose.
Specifically:
- Recognizing which tasks are safe to delegate to AI and which require human handling
- Spotting the subtle errors that look correct but aren’t
- Catching the 3% catastrophic failures before they cause harm
- Evaluating whether an AI-generated deliverable actually solves the problem
- Determining when an output is “good enough” versus when it needs human refinement
This is the new skill gap. Not whether you know how to prompt AI or use the latest tools. Whether you can evaluate the outputs well enough to make the whole system work—like Steven Cramton and Zackary Stephen orchestrating their three chess engines in 2005.
The Centaur Solution (And Why Process Beats Power—For Now)
After Kasparov’s defeat by Deep Blue, he didn’t retreat from AI—he invented a new form of chess called “Advanced Chess” or “Centaur Chess.” The name comes from the mythological centaur: half-human, half-horse, combining the strengths of both. In this context, it means human intelligence guiding and evaluating AI computational power—neither competing against each other, but working as an integrated team.
This is the solution we’re suggesting: not humans versus AI, but humans orchestrating AI through superior process and judgment.
Remember those amateur chess players? They didn’t beat grandmasters because they were better at chess. They beat grandmasters because they had a better process for integrating human judgment with AI capabilities. They knew when to trust which engine. They knew how to combine outputs. They knew which questions to explore.
The grandmasters, ironically, struggled with this. They were so confident in their chess expertise that they either over-trusted the machines or ignored them entirely. The amateurs, by contrast, understood something fundamental: success wasn’t about being smarter than the AI or having more powerful tools. It was about orchestrating the human-AI partnership effectively.
Right now, today, the 70.9% number makes this orchestration essential. Yes, AI can match experts on most tasks. But someone still needs to evaluate which outputs are in the 70% and which are in the catastrophic 3%. Someone needs to determine when AI’s confident answer is actually correct versus when it’s confidently wrong.
That evaluation skill—that judgment—is what makes the system work. It’s what turns a 70% success rate with occasional catastrophic failures into a reliable business process.
The Honest Path Forward
Here’s what we know today, in December 2025:
- AI performance on professional tasks jumped from 50% to 70.9% in just three months
- Quality issues—both subtle and catastrophic—remain present even in “successful” outputs
- The rate of improvement suggests these numbers will continue climbing rapidly
Here’s what we don’t know:
- How quickly AI will continue improving (though recent progress suggests: very fast)
- Whether the quality control challenge will get easier or harder as AI improves
- What the next three months will bring, let alone the next year
But here’s what seems increasingly clear: the future of knowledge work isn’t human versus machine. It’s not even human or machine. It’s human and machine, with humans providing the judgment layer that catches errors, evaluates quality, and determines fitness for purpose.
The question isn’t whether AI will impact your job. At 70.9% and climbing fast, that question is answered. The question is whether you’re developing the judgment skills to evaluate AI outputs effectively—knowing when to trust them, when to refine them, and when to start over.
Twenty years ago, two amateur chess players taught us that weak humans with machines and better process could beat strong humans with machines and inferior process. Today, with AI capabilities advancing every quarter, that lesson about process and judgment has never been more relevant.
The difference is this: in 2005, the chess engines were relatively stable. Today, the AI is improving so fast that the game itself keeps changing. Which means the judgment skills—the ability to evaluate quality, spot errors, and determine appropriateness—become even more critical.
Because in a world where AI can handle 70% of tasks (and counting), the work that remains is precisely the work that requires human judgment about whether the AI’s work is actually good enough.
Sources & Further Reading
- ChessBase: “Dark horse ZackS wins Freestyle Chess Tournament” (June 2005)
- OpenAI: “Measuring the performance of our models on real-world tasks” (GDPval introduction)
- Kasparov, Garry: Various writings on centaur chess and human-AI collaboration
- OpenAI: GPT-5.2 GDPval benchmark results (December 2025)
This is the first in a series exploring what AI’s measured capabilities tell us about the future of knowledge work, human judgment, and the evolving nature of professional expertise.
#HumanPurpose #HumanJudgment #FutureOfWork #AI #ExperienceGap #Leadership #DecisionMaking #ProfessionalDevelopment
