Skip to main content

METR adds judgment-heavy tasks to its AI capability assessment

21 NOVEMBER 2026·2 MIN READ·2 SOURCES

METR updated its time horizon plot with tasks that test judgment and work relevant to AI research and development. Its report, dated November 20, 2026, presents evaluations of public models relevant to AI R&D acceleration.

Key takeaways · 4

  • 01

    The plot measures models on tasks humans have completed, with difficulty scored using human completion time.

  • 02

    METR included a task asking models to design AI evaluations, but its employees found none useful enough for their research.

  • 03

    Models performed below humans on a task scoring philosophical arguments, though METR employees disagree about its relevance to AI R&D automation.

  • 04

    METR says models perform worse on tasks it estimates are more costly to turn into reinforcement-learning training environments, even after accounting for task length.

What the plot measures

METR’s time horizon plot compares models’ performance on tasks that humans have also completed.[2] The evaluation scores task difficulty according to how long human testers took to finish each task.[2] An earlier version showed frontier models doubling the length of tasks they could complete every four months.[2] The updated report presents evidence from METR’s evaluations of public models relevant to AI R&D acceleration.[1] This approach makes human task time a reference point, but it does not mean that every task is equally easy to assess.[2]

The task set is broader

The update adds tasks that are estimated to take humans far longer than tasks in the original suite, as well as more open-ended work.[2] Some tasks have outcomes that can be checked automatically.[2] METR’s examples include choosing a new research direction, a kind of work where success may be harder to verify through an automatic check.[2] MIT Technology Review noted that AI systems have become strong at coding and math because solutions in those areas can be checked automatically.[2] The expanded set is intended to include judgment-heavy work relevant to whether AI can automate research and development.[2]

Evaluation design remains a gap

One task asked models to design new AI evaluations for METR’s research, with METR employees scoring the results.[2] MIT Technology Review reported that none of the model-designed evaluations was useful enough for that research.[2] That result is a specific finding about this task, not evidence that models cannot help with every part of evaluation work.[2] Sayash Kapoor said AI evaluations need to become more creative and called METR’s effort to broaden the range of evaluations a welcome change.[2] The findings make evaluation design one concrete area to watch as teams assess AI’s potential contribution to research.

Read the growth rate with care

METR says capabilities across TH1.1 and TH2 are doubling roughly every 3.6 months.[1] It also says it does not observe strong capability acceleration across public frontier models on its tasks, as measured by its Capability Index.[1] These statements describe different aspects of the report and should not be collapsed into a claim that acceleration is occurring.[1] METR also reports that models appear to have worse judgment than human experts.[1] MIT Technology Review noted that models were below humans on a philosophical-argument scoring task, while METR employees disagreed about how relevant that task is to AI R&D automation.[2]

Teams evaluating AI for research work should distinguish performance on automatically checkable tasks from performance on open-ended judgment tasks. The report offers reasons to test evaluation design and other judgment-heavy work directly rather than infer capability from coding or math results alone.

Why it matters
For teams

AI training for your team

kju.ai is built for teams: daily sessions, team analytics, and rollout support for your organization.

How this developed

  1. 21 November 2026

    METR adds judgment-heavy tasks to its AI capability assessment

Sources

AI fluency, one session a day, built for your work.