Gold Medal at the Math Olympiad, and It Still Cannot Read a Clock

When you assess almost anything, you test the hardest thing it claims to do and assume the easy things came along for the ride. If an accounting package handles multi-entity consolidation, nobody checks whether it can sum a column. Nobody quizzes a network architect on subnet math. Hard implies easy. That inference is nearly free, and it holds often enough that most of us stopped noticing we were making it.

It does not hold for language models. Capability in these systems is spiky. Strength on one task tells you very little about strength on a neighboring task, even when the neighbor looks obviously easier to a human being. Researchers call the pattern jagged intelligence, and the clock result is the cleanest demonstration anyone has produced. Reading a clock face is something most people finish learning around age seven. It sits far below Olympiad geometry on every intuitive scale we own. The scale is what is broken, not the model.

You cannot infer the boring capabilities from the impressive ones. You have to test the boring ones on purpose.

Here is the failure mode we see most often, and it does not look like failure at first.

A team picks the flashiest step in a workflow for the pilot. Summarize the 40-page vendor contract. Turn six months of ticket notes into a trend report. Those tasks are hard for people, which is why they were chosen, and the tool does them well. The demo lands, and everyone agrees to roll it out across the whole workflow.

Then the workflow hits its actual bottleneck, which is almost never the impressive step. It is a dull, structured, high-volume step nobody thought to test. Pull the PO number out of an email body and match it to the right job. Read the date off a scanned delivery ticket. Sort inbound requests into the four categories your dispatch board uses.

Those steps are the clock. They look beneath testing. They are the ones that break.

We have watched an intake tool handle nuanced customer complaints gracefully, then mis-file routine requests because two of the four categories used similar language. The hard part worked. The sorting did not, and the sorting carried most of the volume.

The same AI Index chapter reports on OSWorld, a benchmark of real computer-use tasks on a desktop operating system. Best-model accuracy rose from roughly 12 percent to 66.3 percent, which the report describes as within 6 percentage points of human performance. That is a large jump in a short window, and it deserves to be taken seriously.

Read the inverse, though, because the inverse is what you have to staff for. The best model still fails roughly one task in three. And it fails that often on a benchmark where every task is pre-specified and graded deterministically, with none of the ambiguity, missing context, or half-legible attachments that define real work. This is not a claim that AI can do 66 percent of your team's computer work. It is a ceiling measured under conditions kinder than your Tuesday.

Chapter 4 of the same report lines up with that. Actual AI agent deployment sits in the single digits across nearly all business functions, even among organizations that report adopting AI. The gap between benchmark capability and production reality is real, and it is mostly made of boring steps.

If a tool fails one item in three, some human has to catch the third one. There are exactly three ways to handle that, and choosing consciously is the whole job.

That third row is the one to watch. Nobody writes "we will not review the output" into a project plan. It arrives three weeks after go-live, when the person doing the reviewing gets pulled onto something urgent and never goes back.

The license cost shows up in a purchase order. The checkpoint cost shows up in somebody's week, unbudgeted, and it is often the larger of the two. Price both before you sign. Our managed AI services work starts with that conversation rather than with tool selection.

Before you deploy anything into a workflow, take the 20 most routine items that workflow handles. Not the interesting ones. Not the edge cases you are proud of. The mundane middle of your volume.

Run all 20 through the tool. Check every output by hand, field by field, against what the correct answer actually was.

Twenty is enough. If there is a jagged edge in that workflow, 20 boring items will usually expose it, and the failures tend to cluster in a way that tells you where. Two hours of somebody's afternoon buys you the answer.

Keep the 20 items. When the model version changes, and it will, run them again. Jaggedness moves between versions, so a capability you verified in March is not automatically still there in September.

Be careful with the numbers in both directions. A 66.3 percent benchmark score does not predict your result. A narrow, well-scoped task with clean inputs can beat it comfortably. An ambiguous task with messy inputs and unwritten rules that live in one person's head will do far worse, and it will do worse quietly.

None of this means the technology is oversold. The Olympiad result is real and remarkable. The point is narrower and more useful: the shape of the capability is uneven, and the unevenness does not follow human intuitions about difficulty. Test accordingly.

The same discipline applies to any tool you put in front of the business, which is why we treat AI pilots like any other change to managed IT. Verify the dull path, name who checks the output, then scale.

What is jagged intelligence?

It is the term researchers use for the uneven capability profile of current AI models: very strong on some tasks and unreliable on others that appear far easier. Stanford HAI's 2026 AI Index illustrates it with a model that can win a gold medal at the International Mathematical Olympiad while reading analog clocks at 50.6 percent accuracy, against 90.1 percent for humans.

Does 66.3 percent on OSWorld mean AI can do two thirds of my team's computer work?

No. OSWorld is a set of pre-specified computer-use tasks graded deterministically. The 66.3 percent figure is a best-model score under controlled conditions, and the useful reading is the inverse: it still fails roughly one task in three, without the ambiguity of real work. Treat it as a ceiling, not a forecast.

How many test items do I need before trusting a tool in a workflow?

Start with the 20 most routine items that workflow handles and check every output by hand. Twenty is usually enough to surface a jagged edge, and the routine items matter more than the difficult ones because they carry the volume.

What should we budget for besides the license?

The checkpoint. Someone has to review output, and you should decide in advance whether that means every item, a sample, or nothing. Checking everything is safe but caps your savings at the gap between doing the work and reviewing it. Sampling is only valid when errors are independent and something downstream catches what you miss.

Is jagged intelligence a reason to wait on AI?

No. It is a reason to change how you evaluate. The same report puts AI agent deployment in the single digits across nearly all business functions, which suggests most organizations are still working out the operational side rather than the technology. Teams that test the boring steps get there sooner.

Where should a Texas business start?

Pick one workflow, find its highest-volume routine step, and test that step rather than the impressive one. If you want a structured look at where automation would actually help, our free IT assessment is a reasonable starting point.

If you are weighing an AI pilot and want a practitioner's read on which step to test first, call us at 713-571-2390. We are based in The Woodlands with a second office in Round Rock, and support is 100 percent Texas-based.