Artificial Intelligence

Better Than What? The Baseline Missing From AI Productivity Claims

People using AI usually outperform people working alone. Against the better of human or AI alone, two meta-analyses find the combination falls short. BBGK on baselines, verification cost and accountability in AI productivity claims.

AI & Judgment | Evidence Review | November 2026

People with AI beat people alone, but not the better of human or AI aloneDiagram of two results from the October 2026 AutoAnalysis preprint. Compared with humans working alone, human-AI combinations scored a Hedges g of plus 0.53. Compared with the better of humans alone or AI alone, across 171 effect sizes, they scored minus 0.35.BBGK ANALYSIS | AI & JUDGMENTPeople with AI beat people alone.They did not beat the best alternative.HUMAN + AI vs HUMAN ALONE+0.53Hedges' g | augmentation≠HUMAN + AI vs THE BETTER OF THE TWO−0.35Hedges' g | synergy | 171 effect sizesThe baseline decides the verdict.SEE DEEPER | KNOW WIDERahsshohelahmed.com
Two results from the same October 2026 preprint. The second comparison rests on 171 of the 1,955 effect sizes, because most studies never measured AI working alone. Source: Vaccaro, Almaatouq and Malone (2026), arXiv 2610.42001, not peer reviewed.

Key findings

  • People using AI usually outperform people working alone. Against the better of human alone or AI alone, two meta-analyses by the same research team find the combination falls short on average.
  • Most studies cannot answer the harder question at all. In the newest evidence review, only 171 of 1,955 effect sizes, about 9%, included an AI-only comparison.
  • A credible AI productivity claim needs three things attached: the baseline it was tested against, the cost of checking the output, and the person accountable for what remains wrong.

Do AI productivity claims hold up? Against people working alone, usually yes. Against the best available alternative, the evidence so far says often no. The difference between those two baselines is the detail most AI productivity claims leave out, and it decides whether an organization has measured real improvement or improvement against a convenient comparison.

Picture three versions of one assignment. An analyst completes it unaided. An AI system completes it independently. A second analyst uses the AI, reviews its answer, adjusts it and submits the result. The organization compares the third result with the first, sees a gain, and declares the pilot a success.

But what if the AI working alone did better still? And what if the time spent checking and correcting the combined output made that workflow more expensive than either alternative? A pilot that never ran the second comparison cannot say.

Two baselines, two verdicts

The clearest evidence comes from a 2024 meta-analysis in Nature Human Behaviour by Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone. It pooled 106 experiments and 370 effect sizes, and it separated two measures that are often treated as one.

Human augmentation asks whether a person using AI performs better than a person working alone. Human-AI synergy asks something harder: does the combination outperform whichever was better in the first place, the human alone or the AI alone?

The authors found augmentation on average, but not synergy. Against the better standalone performer, the combined systems had a negative average effect (Hedges’ g = −0.23; 95% confidence interval −0.39 to −0.07). The direction depended on who was stronger to begin with. Where humans outperformed AI alone, adding AI helped. Where AI outperformed humans alone, adding the human made results worse. Tasks that involved creating content fared better than tasks that involved making decisions.

In an October 2026 preprint called AutoAnalysis, the same three authors extend the question with an automated, self-updating evidence review. Its most recent run covers 420 papers and 1,955 effect sizes. Human-AI combinations again outperformed humans alone (g = 0.53; 95% CI 0.45 to 0.61). In the comparisons that also measured AI working alone, they again fell below the better standalone option (g = −0.35; 95% CI −0.52 to −0.18).

BBGK summary of the two evidence reviews. Negative values mean the combination did worse than the comparison.
Evidence review Human + AI vs human alone Human + AI vs the better of the two Status
Vaccaro et al. (2024) Positive on average g = −0.23, from 370 effect sizes Peer reviewed
AutoAnalysis (2026) g = 0.53 g = −0.35, from 171 effect sizes Preprint, not peer reviewed

One number in that preprint deserves more attention than either headline. Only 171 of the 1,955 effect sizes, about 9%, included an AI-only condition. Most of the research literature, like most corporate pilots, never ran the comparison that would show whether the human-AI arrangement was the best option.

The preprint has limits that belong next to its results. It has not been through journal peer review. The estimates vary enormously across studies, with heterogeneity above 90% on the standard I-squared measure, and its findings by task type are labelled exploratory. Its AI agents did the literature screening and data extraction, and the human check was small: the authors themselves reviewed 30 papers at each of three steps and describe that as a sanity check, not a formal reliability study. A second AI model also audited samples and agreed with 90% of screening decisions, but, as the paper notes, agreement with another model is not accuracy against an independent standard.

That last point is the subject of this article in miniature. A system produced an impressive research synthesis, and the quality of its verification is the part that remains incompletely established.

One profession, three results

The pattern is easiest to see inside a single profession. In a randomized trial published in JAMA Network Open, 50 physicians worked through diagnostic cases with either GPT-4 plus conventional resources or conventional resources alone. Access to the model did not significantly improve their diagnostic reasoning. The model working on its own scored higher than both groups of physicians.

A companion trial, published in Nature Medicine, tested 92 physicians on management reasoning: decisions about treatment, testing and risk. Here physicians with GPT-4 did score higher than those with conventional resources. They did not score significantly differently from the model alone.

So one research group, studying one profession, found no augmentation on one task and augmentation without synergy on another. Both trials used written clinical vignettes, and neither establishes better outcomes for real patients. The lesson is not that one experiment cancels the other. It is that the value of assistance depends on the task, the user, the interface and, above all, the comparison.

The best-known workplace evidence fits the same frame. In a field study in the Quarterly Journal of Economics, Erik Brynjolfsson, Danielle Li and Lindsey Raymond followed 5,172 customer-support agents. Access to a generative AI assistant raised issues resolved per hour by 15% on average, with larger gains for less experienced agents. That is solid evidence of augmentation in a defined workflow. The study had no AI-only arm, so it cannot say whether the combined arrangement was the strongest one available. That is BBGK’s reading of the design, not a criticism the authors make.

Boundaries can also reverse within one experiment. In a Harvard Business School study of 758 BCG consultants, GPT-4 produced strong gains on tasks chosen to sit inside its competence. On one task deliberately chosen to sit outside it, consultants using the model were reported to be 19 percentage points less likely to reach the correct answer. That is one task, not a general failure rate.

Verification is the step nobody prices

A baseline tells you whether the result was better. It does not tell you what it cost to know that. This is where Stanford’s November 2026 AI100 report, AI Goes Mainstream, is most useful. The report documents how far AI has moved since 2021 into reasoning, coding, research and agentic work, and it is direct about the constraint. Its economics section names reliability as the chief bottleneck for deploying AI systems, and its conclusions state that “current AI systems remain jagged.”

The report also describes where checking breaks down. It warns that agentic workflows can exceed effective human oversight, calls for principled methods to validate machine-generated scientific discoveries, and records lawyers being fined for court filings that contained invented cases. It notes, fairly, that partial competence paired with human review is often useful in practice. The open question is what that review costs and whether it is really happening.

Two other sources suggest it often is not. A working paper summarized by MIT Sloan, by Christian Catalini, Xiang Hui and Jane Wu, argues that AI makes it cheap to produce work but not to judge whether the work is any good, and that the limited capacity for human verification will cap how much value organizations capture. And a Sonar survey of more than 1,100 developers, as reported by LeadDev, found that 96% do not fully trust AI-generated code to be functionally correct, while only 48% say they always check it before committing. That is a vendor survey of one profession, so it shows a pattern, not a universal rate.

Rising benchmark scores do not close this distance. The evaluation group METR cautions that a model’s task-completion time horizon is not the length of time it can work independently, and its tests focus largely on well-specified software tasks.

Several researchers and vendors now call this distance a verification gap. BBGK uses the phrase descriptively and does not claim it. In BBGK’s terms it is one instance of the AI Adaptation Gap: capability arrives faster than the institution’s ability to absorb it, and checking is one of the first places the strain shows.

The working rule. An output is not a result until someone other than its producer can check it, at a cost the organization has counted.

Three questions before accepting the result

BBGK’s AI Delegation Boundary Framework settles the first decision: which work to Automate, which to Assist, and which a named person must Own Directly. A second decision follows immediately. Once the work is allocated, what evidence justifies accepting it? The three questions below are not a new BBGK tool. They are the baseline test applied in sequence.

1. Better than what?

Did the AI-assisted workflow outperform both a human-only and an AI-only alternative on the outcome that matters, or did it only beat the old process on speed? Where a full three-way test is not feasible, say so in the claim. Success on a benchmark or in a demonstration is not success in the target setting.

2. Checked how?

Which consequential claims, calculations or actions were tested independently, what could not be checked, and what did detection and correction cost? Some outputs can be validated by execution tests, authoritative records or formal proof. Others rest on contested evidence or judgment with no single right answer. Asking the same system again is not an independent test, which is why the BBGK Independent Verification Check moves verification outside the channel that produced the claim. For grading how much confidence a claim deserves, use the BBGK Evidence Meter.

3. Accountable to whom?

Who can pause, override or reverse the decision, and what evidence would let an affected person challenge it? A reviewer with no time, no access to the underlying records and no power to stop deployment is not a control because a workflow diagram shows a human icon. BBGK’s analysis of the judgment calls nobody made shows how those decisions pass to a system by default.

In practice this means a pilot should record four things together: the quality of the final result, performance against the best feasible alternative, the time and resources spent on verification and correction, and the consequences of the errors that remain. Suppose a team now produces ten analyses in the time it once produced two. Output is up fivefold. The dashboard does not show how many claims were verified, how many exceptions were missed, or what a wrong answer would cost to reverse. That team has measured production, not verified outcomes.

None of this requires manual checking of every low-risk output. Routine tests and independent software checks may do better than human review. The point is to stop treating oversight as free and correctness as implied. The voluntary NIST AI Risk Management Framework makes the same practical demand through its four functions: Govern, Map, Measure and Manage.

The gain that leaves with the tool

There is one more baseline: the person after the tool is taken away. In a 2025 field experiment published in PNAS, nearly a thousand high-school mathematics students were given GPT-4 based tutors. Unrestricted access improved their performance during practice, and those students then did worse than their peers when the tool was removed. A tutor designed to give structured hints largely removed the harm.

That is one educational setting, not every workplace. It matters here because verification is a human skill, and skills that are not exercised fade. BBGK describes this pattern as the Performance-Capability Gap: visible output rises while the underlying ability to produce or check it does not. An organization that automates the work which trains its future reviewers is spending down its own verification capacity.

Two fair objections

Perhaps the AI needs less oversight than the human. That is possible, and the meta-analyses make it hard to ignore. Where AI is demonstrably more dependable, adding an unprepared human reviewer can lower performance. The answer is not to favor human judgment by default. It is to test the whole configuration. Independent automated checks, external evidence and tightly bounded permissions can sometimes do more than a ceremonial human sign-off. An ambiguous, high-stakes choice may still need real human authority even when a model is unusually accurate at parts of the task.

Some decisions have no objectively correct answer. Also true. For that kind of work, verification cannot be a pass or fail score. It means transparent assumptions, documented reasons, clearly stated uncertainty and a way to contest the outcome. This is where evaluation stops being a technical exercise and becomes an institutional one.

What a credible claim looks like

The Stanford report is persuasive about the scale of the technological change. Its deeper warning is about institutional readiness: AI can produce sophisticated work before its users have shown they can judge the results reliably. BBGK made the individual version of this argument in its analysis of bad AI advice. The same discipline now has to apply to whole workflows.

The next AI productivity claim worth believing will not be “AI helped us do more.” It will be: this configuration produced better verified outcomes than the alternatives, at a total cost we counted, with a named person responsible for what might still go wrong.

Frequently asked questions

Does AI make workers more productive?

Often, when the comparison is with people working alone. An October 2026 preprint pooling 1,955 effect sizes from 420 papers found that human-AI combinations outperformed humans alone on average (Hedges’ g of 0.53). A field study of 5,172 customer-support agents found a 15% average rise in issues resolved per hour. Neither result shows that the human-AI arrangement was the best available way to do the work.

What is the difference between human augmentation and human-AI synergy?

Human augmentation means a person using AI performs better than a person working alone. Human-AI synergy is a harder test: the combination must outperform whichever was better in the first place, the human alone or the AI alone. A workflow can show augmentation without showing synergy.

Do humans and AI together outperform AI alone?

On average, the published evidence says not yet. A 2024 meta-analysis in Nature Human Behaviour found that human-AI combinations performed worse than the better of humans or AI alone (Hedges’ g of -0.23 across 370 effect sizes). The October 2026 preprint from the same authors reports -0.35 across 171 effect sizes. Results vary widely by task, so this is an average, not a rule for every workflow.

What is the verification gap in AI?

It is a descriptive phrase, used by several researchers and vendors, for the distance between how fast AI can produce work and how fast people and institutions can check it. BBGK uses it in that plain sense and does not claim it as a framework. BBGK treats it as one instance of the AI Adaptation Gap.

How should an organization evaluate an AI pilot?

Compare the AI-assisted workflow with both a human-only and an AI-only alternative where that is feasible, on the outcome that matters. Record the time and cost of checking and correcting the output. Name the person who can pause, override or reverse the result. A pilot that reports only speed against the old process has measured production, not verified outcomes.

Research note. Figures for the two meta-analyses, the customer-support study and the AI100 report were checked on 17 November 2026 against the papers’ published abstracts, the arXiv full text and the report itself. The physician and consulting results were confirmed through the journal records and university or press summaries. The Sonar survey and the Catalini, Hui and Wu paper were read through LeadDev and MIT Sloan coverage, not at source. BBGK did not recompute any estimate. The three questions are an editorial reading aid, not a tested instrument, and the cited studies do not measure a universal verification gap across industries.

Sources and further reading

  1. Wooldridge, M. et al. (2026). AI Goes Mainstream: The Scaling Era and the Rise of Generative AI. One Hundred Year Study on Artificial Intelligence (AI100), Stanford University, November 2026. Read the source. Expert panel report
  2. Vaccaro, M., Almaatouq, A. and Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293-2303. Read the source. Peer-reviewed meta-analysis
  3. Vaccaro, M., Almaatouq, A. and Malone, T. W. (2026). AutoAnalysis: An automated, self-updating meta-analysis of human-AI collaboration. arXiv 2610.42001, 30 October 2026. Read the source. Preprint; not peer reviewed
  4. Brynjolfsson, E., Li, D. and Raymond, L. (2025). Generative AI at Work. Quarterly Journal of Economics, 140(2), 889-942. Read the source. Peer-reviewed field study
  5. Goh, E. et al. (2024). Large Language Model Influence on Diagnostic Reasoning. JAMA Network Open. Read the source. Randomized trial; clinical vignettes
  6. Goh, E. et al. (2025). GPT-4 assistance for improvement of physician performance on patient care tasks. Nature Medicine. Read the source. Randomized trial; clinical vignettes
  7. Dell’Acqua, F. et al. (2023). Navigating the Jagged Technological Frontier. Harvard Business School working paper, summarized by HBS Working Knowledge. Read the source. Field experiment; outside-frontier figure as reported
  8. Kwa, T. (2026). Clarifying limitations of time horizon. METR, 22 January 2026. Read the source. Research note from the evaluator
  9. National Institute of Standards and Technology (2023). AI Risk Management Framework (AI RMF 1.0), Core. Read the source. Voluntary governance framework
  10. Bastani, H. et al. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. PNAS, 122(26). Read the source. Peer-reviewed field experiment
  11. MIT Sloan (2026). Seeing real value from AI depends on being able to verify its outputs, 8 June 2026, on Catalini, Hui and Wu, “Some Simple Economics of AGI.” Read the source. University summary of a working paper
  12. LeadDev (2026). You can’t verify all the AI-generated code, reporting a Sonar developer survey. Read the source. Trade press report of a vendor survey

Editorial disclosure. This article evaluates the publicly reported methods and findings of the cited studies. The interpretation is BBGK’s own, not a claim made by Stanford, MIT, Harvard Business School, METR, NIST or the study authors. BBGK has no relationship with any of them.

Comments

comments

Get Essays by Email