Why better benchmarks don’t always mean better understanding

I’ve been thinking about what it actually means to do science.

We often associate science with learning difficult subjects.

Mathematics.

Physics.

Statistics.

Machine learning.

Solving an integral certainly requires knowledge. Understanding gradient descent requires knowledge. Training a large model requires a tremendous amount of technical expertise.

But none of these, by themselves, mean we are doing science.

Learning what is already known is education.

Applying what is already known can be engineering.

Science begins somewhere else.

The National Academies describes science as a collective and cumulative process for developing reliable knowledge through evidence.[1] I like that definition because it shifts the focus from knowing things to producing knowledge we can trust.

And that requires more than asking:

Why?

It also requires asking:

How?

Under which conditions?

What is this connected to?

When does this explanation stop working?

What evidence would challenge what we believe?

And perhaps most importantly:

What new question becomes visible because of what we just learned?

This last question matters to me.

Because the outcome of science is not always a solution.

Sometimes the most valuable outcome of science is discovering a better problem.

Science and Engineering Are Solving Different Problems

I don’t think science and engineering should be placed in a hierarchy.

Engineering is not “less scientific” work.

It has a different objective.

Science often asks:

What don’t we understand yet?

Engineering often asks:

Given what we know, how can we make this work?

Science tries to produce explanations, relationships, models, and knowledge that survive beyond a single implementation.

Engineering takes knowledge and turns it into capability under real constraints.

Can we make it faster?

More reliable?

Cheaper?

Scalable?

Observable?

Useful?

Safe?

These are difficult and extremely valuable questions.

But they are different questions.

A system can be a remarkable engineering achievement without producing a remarkable scientific discovery.

Likewise, a scientific discovery may be incredibly important while having no immediate product value.

The problem begins when we stop distinguishing the two.

Software 2.0 Changed What Engineering Looks Like

In 2017, Andrej Karpathy described neural networks as Software 2.0.

His argument was that neural networks represented more than another tool in the machine-learning toolbox. Traditional Software 1.0 consists primarily of explicit instructions written by programmers. In Software 2.0, much of the behavior emerges from optimization over data, with the resulting program encoded in learned weights.[2]

That shift was profound.

We stopped describing every desired behavior explicitly.

Instead, we increasingly defined objectives, collected data, designed architectures, evaluated outputs, and let optimization discover parts of the implementation.

Karpathy also pointed to a limitation that feels even more relevant today: these systems can work extremely well while remaining difficult to understand internally.[2]

Software 2.0 eventually created an enormous new engineering surface around learned systems.

Today we work on:

model selection,

fine-tuning,

prompting,

context engineering,

RAG,

evaluation,

agent orchestration,

model routing,

inference optimization,

observability,

guardrails,

cost optimization,

and production deployment.

I think there is a useful name for much of this work:

AI Software Engineering.

And I mean that positively.

AI Software Engineering is becoming one of the most interesting engineering disciplines we have.

But building a better AI system and understanding something fundamentally new about intelligence, learning, or model behavior are not necessarily the same achievement.

Capability Is Not Understanding

Imagine a paper proposes a new method.

We change the prompting strategy.

Add another model.

Introduce a new orchestration layer.

Use more compute.

Tune a few parameters.

And the benchmark improves by 4.7%.

That’s useful.

Maybe very useful.

But what exactly did we discover?

The engineering question is:

Did we make it work better?

The scientific question goes further:

What did its improvement teach us that we did not know before?

Why did it improve?

Under which conditions does the improvement exist?

When does it disappear?

Does the result generalize?

What alternative explanations could produce the same observation?

Can another team reproduce it?

What evidence would contradict our interpretation?

This is the distinction I keep coming back to:

Capability Improvement

We can make X work better.

versus:

Knowledge Advancement

We now understand something about X that we did not understand before.

Both are valuable.

But they are not interchangeable.

A new capability is not necessarily a new scientific understanding.

A Benchmark Is an Observation, Not an Explanation

This distinction becomes particularly important in AI because experimentation is becoming incredibly cheap.

Models can run thousands of evaluations.

Agents can generate experiments.

Hyperparameters can be searched automatically.

Benchmarks allow approaches to be compared quickly.

That is a huge advantage.

But optimization can discover something that works without explaining why it works.

And if we are not careful, an experimental result can slowly become a general claim.

That is where we should become more cautious.

Reproducibility has already been a recognized concern in machine-learning research. A review of 93 deep-learning studies in software engineering found meaningful replicability and reproducibility challenges, including sensitivity to optimization and sampled data.[3] Nature Computational Science has similarly argued for stronger transparency and reproducibility practices in machine-learning research.[4]

This does not mean the results are useless.

It means the strength of our claim should match the strength of our evidence.

If we demonstrate:

“This architecture achieved better results under these conditions.”

that can be an excellent engineering result.

But if we conclude:

“Therefore, this architecture reveals how intelligence should be structured,”

we have made a very different claim.

That second statement requires much more.

Are We Confusing AI Research With AI Software Engineering?

This is where I think today’s AI ecosystem becomes particularly interesting.

There is clearly genuine scientific research happening in AI.

Work on learning theory, optimization, interpretability, robustness, representation, causality, model behavior, and many other areas is explicitly trying to understand phenomena we do not yet understand.

But there is also a rapidly growing category of work that looks different.

Take existing models.

Combine them differently.

Improve context.

Change prompting.

Add tools.

Introduce an agent.

Build a better evaluator.

Optimize inference.

Improve the benchmark.

These can be brilliant contributions.

They can create products.

They can create companies.

They can dramatically improve what AI systems are capable of doing.

But perhaps we should sometimes describe them for what they are:

excellent AI Software Engineering.

There is nothing wrong with that.

In fact, recognizing the distinction makes both disciplines stronger.

Engineering deserves to be evaluated by engineering standards:

Does it work?

Is it reliable?

Does it scale?

Is it economical?

Can we operate it?

Does it create value?

Science deserves another set of questions:

What did we learn?

Does it generalize?

Can we reproduce it?

What mechanism explains it?

Where are the boundaries?

What new question did it reveal?

Science Should Not Only Solve Problems

There is another part of science that I believe we sometimes underestimate.

Science does not simply close questions.

It creates new ones.

We observe something.

We connect it to something else.

A relationship appears.

That relationship produces an explanation.

The explanation exposes an anomaly.

The anomaly creates a new question.

And suddenly we have discovered something we did not even know we needed to investigate.

The National Academies makes a similar point when describing scientific knowledge as a growing web: evidence strengthens existing connections while also creating new questions.[1]

This is why I like to think of science as something that expands the problem space.

Engineering is often rewarded for closing problems.

Science is often valuable because it opens better ones.

And sometimes the deepest scientific result is not:

“Here is the answer.”

It is:

“We were asking the wrong question.”

Maybe We Need a Better Label

Perhaps the real issue is not whether a particular piece of AI work is “science.”

Maybe the problem is that we try to put too many fundamentally different activities under the word research.

There is a spectrum:

Scientific Discovery

↓

Applied Science

↓

Applied Research

↓

Research Engineering

↓

AI Software Engineering

↓

Product Engineering

These are not rankings.

Moving downward does not mean becoming less valuable.

A new inference technique with little scientific novelty might save a company millions of dollars.

An evaluation framework might dramatically improve the safety of a production AI system.

A new agent architecture might unlock an entirely new product.

Those are real achievements.

But they are achievements of a different kind.

And being precise about that difference helps us evaluate results more honestly.

Better Results or Better Understanding?

So perhaps the question we should ask when we read the next AI paper, benchmark, or architecture proposal is not simply:

Did it perform better?

We should also ask:

Do we understand something better because of it?

If the answer is yes, we may be looking at scientific progress.

If the answer is no—but the system became dramatically more useful, reliable, scalable, or efficient—we may be looking at excellent engineering.

And that’s perfectly fine.

We should not require every AI experiment to be science.

But when we call something science, I think we should expect more than a better result.

We should expect a better understanding.

Because engineering asks whether we can make something work.

Science asks what its working—and its failure—teaches us about what we still don’t understand.

And sometimes, if we’re lucky, that understanding gives us something more valuable than an answer:

a question nobody knew to ask before.

References

[1] National Academies of Sciences, Engineering, and Medicine — Decoding Science: How Does Science Know What It Knows? The National Academies describes science as a collective, cumulative process for developing increasingly reliable knowledge through evidence, while emphasizing that scientific knowledge continues to generate new questions. https://nap.nationalacademies.org/resource/25303/interactive/index.html

[2] Andrej Karpathy — Software 2.0 (2017). Karpathy introduced the Software 2.0 framing, contrasting explicitly written programs with behavior learned through data and optimization, while also noting the difficulty of understanding how high-performing neural networks work internally. https://karpathy.medium.com/software-2-0-a64152b37c35/

[3] Liu et al. — On the Replicability and Reproducibility of Deep Learning in Software Engineering (2020). The authors reviewed 93 deep-learning studies and investigated replicability and reproducibility problems related to optimization stability, convergence, and sampled data. https://arxiv.org/abs/2006.14244

[4] Nature Computational Science — Moving Towards Reproducible Machine Learning (2021). The editorial argues for stronger reporting, transparency, and reproducibility practices in machine-learning research. https://www.nature.com/articles/s43588-021-00152-6

@misc{bilgin2026ai-software-eng,
  title   = {Are We Doing AI Science—or AI Software Engineering?},
  author  = {M. Emre Bilgin},
  year    = {2026},
  url     = {https://mebilgin.com/posts/2026/08/ai-software-engineering/}
}