Why Doesn’t Engineering Already Know This?

September 9, 2026 · 12 min read

Some version of this question gets asked in every company I have worked with, usually by a senior executive, usually in a voice that is trying very hard to stay reasonable:

Why didn’t engineering already know this?

It sounds like a fair question. It is often built on an assumption that falls apart the moment you say it out loud: that a competent engineering organization holds current expert knowledge of every customer’s environment, every network topology, every message format, every vendor quirk, every protocol edge case, every obscure configuration setting, and every failure mode that shows up once every two years.

No organization on earth works that way. Not yours, not your competitor’s, not the vendor you are thinking about switching to.

The Answer Is Usually Tiny

Here is what makes this so tempting. Hard technical problems very often resolve into small answers.

A VPN outage that consumed three days turns out to be one mismatched encryption parameter.
An HL7 interface failure turns out to be one malformed segment, or a field somebody populated differently after an upgrade.
An integration problem turns out to be one port, one expired certificate, one firewall rule, one vendor-specific interpretation of a spec, or one unusual thing a customer did to their own network.

So after somebody finds it, an executive looks at the one-sentence explanation and thinks, understandably: that was it?

Yes. That was it.

But simple after discovery and obvious before discovery are two completely different things, and the entire quality of your judgment about engineering depends on keeping them apart.

Hindsight Compression

There is a predictable thing that happens at the end of an investigation. The team compresses three days of diagnosis into one sentence, because that is what a good writeup does. Management then reads the one sentence and quietly deletes the three days from the story.

The three days were the work. The sentence is the receipt.

The final answer may fit in a Slack message. That does not mean the problem originally arrived in a Slack-message-sized box.

When the problem arrived, it did not say “mismatched encryption parameter.” It said “the connection drops sometimes.” Or “data stopped flowing at the hospital.” Or “it works in test.” Getting from that to the parameter is the profession.

Simple Afterward is not Obvious Beforehand. The fix is one line. Finding out which line took everything to the left of it.

Hard Questions Are Fine. Ask More of Them.

I want to be very clear that I am not asking executives to go easy on engineering. Pressure is healthy. Vague, unfalsifiable pressure is not.

These are all legitimate management questions, and you should ask them without apology:

Why did this take as long as it did?
Did we recognize what kind of problem this was quickly enough?
Did we troubleshoot systematically, or did we guess?
Did we bring in the right specialist early enough?
Have we seen this before, and did we know that we had?
Did we document what we learned?
Could automation, testing, observability, or better documentation catch this next time?
Should this expertise live permanently inside the team?

Every one of those has an answer, and the answer tells you something you can act on.

“Why didn’t they already know everything?” has no answer, tells you nothing, and costs you credibility with the people you most need to be honest with you.

Technical Knowledge Has a Half-Life

There is a real distinction between core competency and perishable specialty knowledge, and most management frustration lives in the gap between them.

Teams should maintain deep, current competency in what they touch continuously. That is non-negotiable. But a lot of genuinely important knowledge is episodic by nature:

Customer-specific networking
VPN and IPsec edge cases
HL7 interface details
Epic and other EHR-specific implementation quirks
Legacy database behavior nobody has provoked in two years
Certificate-chain problems
DNS oddities
Cloud networking interactions across accounts and regions
Vendor-specific APIs that do not match their own documentation
Old protocols that still matter because a large customer still uses them
Infrastructure that gets touched once every eighteen months

Somebody on your team may have known one of these subjects extremely well, and still need thirty minutes, three hours, or a full day to reload the details. That is not decay of competence. That is how expertise works in every field, including yours.

If someone uses Kubernetes every day, the expertise is sharp. If they last configured that particular IPsec tunnel three years ago, they can understand networking perfectly well and still need to look things up.

Do not confuse recall latency with incompetence. You would not conclude that a physician is unqualified because she opened a reference before prescribing something she prescribes twice a decade. You would conclude the opposite.

A Better Test of Competence

If instant recall is a bad measure, what is a good one? Ask whether the team demonstrated the skills that actually transfer between problems:

Did they recognize the class of problem they were in?
Did they know what evidence to collect, and collect it before it disappeared?
Did they know how to narrow the search space instead of guessing repeatedly?
Did they know when they had reached the edge of their expertise?
Did they know whom to bring in, and did they do it without ego?
Did they know how to verify the fix rather than declare it?
Did they capture the learning so the organization owns it now?

That is a much better definition of technical competence than “knew the answer immediately,” and it has the useful property of being observable. You can read an incident timeline and grade all seven.

Rare Skills Economics

The other half of the executive’s frustration is really a staffing question wearing a technical costume. You do not want to hire a full-time expert for every conceivable specialty, and you could not afford to if you did.

So match the model to the frequency:

Need it every day: hire it and keep it internal.
Need it every month: own it internally or share it across teams.
Need it twice a year: fractional or on-call specialist expertise usually wins.
Need it once every three years: your real competency is being able to find the right person quickly.

Skills Should Match Frequency. How often you need an expertise decides whether you hire it, share it, rent it, or route to it.

Notice what this reframes. The goal was never to eliminate unknowns. Unknowns are a permanent feature of connected enterprise systems.

The goal is not to eliminate unknowns. The goal is to become very good at resolving unknowns.

Now a Word to CTOs and Engineering Leads

Everything above is easy to weaponize, so let me close that door.

“It was an esoteric problem” is not an alibi. It cannot become engineering’s version of the dog ate my homework. Business leaders learn very quickly to recognize complexity used as cover, and once they do, you lose the benefit of the doubt on the days you genuinely need it.

The obligations that come with asking for this understanding:

Write runbooks, especially for the things you hope never to see again.
Retain diagnostic evidence instead of restarting the service and losing it.
Document unusual customer configurations while you still remember them.
Automate the repeatable tests that would have caught it.
Maintain a real directory of specialist expertise, internal and external, with names and current contact paths.
Recognize recurring classes of problems, not just recurring tickets.

The first occurrence may be an edge case. By the fifth occurrence, it is your architecture.

If the same rare problem has happened six times, it is not rare and it is not a surprise. It is a decision you keep declining to make.

The Executive Test

Here is a framework you can apply to any incident in about five minutes, without knowing anything about networking.

Frequency. How often should we reasonably expect to encounter this?
Consequence. How damaging is it when we do?
Detectability. Could we have found it faster with better instrumentation or evidence?
Repeatability. Can we make the next occurrence materially easier?

Those four dimensions decide the remediation, and they decide it fairly consistently:

High frequency and high consequence: build permanent internal competency and automate it.
Low frequency and high consequence: maintain playbooks, escalation paths, and guaranteed specialist access.
High frequency and low consequence: automate aggressively and stop discussing it.
Low frequency and low consequence: do not reorganize the company around it.

Most bad post-incident decisions I have watched come from applying the first quadrant’s response to a fourth-quadrant problem, usually because the incident was recent and someone senior was annoyed.

You Are Not Only Operating Your Own Software

This is the part that surprises executives who came up outside enterprise technology. The moment your software connects to a customer environment, the problem stops being entirely yours. It becomes partly about somebody else’s infrastructure, configurations, security rules, network policies, historical decisions, vendors, and people.

You can know your software extremely well and still not know what someone in another company did to a firewall in 2019.

Your team may have complete visibility into your product and near-zero visibility into the environment it has to survive in. They cannot read the customer’s change log from 2017, and often neither can the customer.

This is most acute in healthcare, government, banking, and large enterprises, because integrations there accumulate for decades. Nothing gets removed. A hospital interface engine can carry rules written by people who retired before your company existed. That is not an excuse for anything. It is the terrain, and executives who understand the terrain make better calls about staffing, timelines, and contracts.

What an RCA Is Actually For

Most root cause analyses answer one question: what broke. That is the least useful thing in the document.

A good RCA also answers:

Why was this difficult to diagnose?
Was the knowledge required common or specialized?
How recently had anyone on the team used that knowledge?
Which parts of this were under our control?
Which parts depended on a customer or vendor?
What piece of evidence shortened the diagnosis, and did we get it by luck or by design?
What knowledge should we retain, and where?
What should be automated?
Should this capability be internal, fractional, or external?
What specifically would make this faster next time?

And a reminder that some organizations need more than others: an RCA is not a courtroom. The point is not to identify the person who failed to remember an obscure setting. The point is to identify which organizational capability should exist the next time this situation arrives.

Run RCAs as trials and you will get RCAs written by defendants. The prose will be immaculate and you will learn nothing.

So, to the Executives

You should absolutely expect your engineering organization to be competent. Expect discipline, preparation, documentation, systematic diagnosis, honest ownership, and visible improvement over time. Push hard on all of it.

Just do not confuse competence with omniscience. A strong engineering organization does not know every answer instantly. It knows how to turn unfamiliar problems into understood problems, faster and more reliably, year after year.

The test of a good engineering team is not whether they already know every answer. It is whether the second time is embarrassingly easier than the first.

That is the number to watch. If your team’s second encounter with a problem class is not dramatically cheaper than the first, you have a real issue worth escalating. If it is, you have a good team, and the right response to the one-line fix is not “that was it?” It is “what did we learn, and what did we build with it?”

Try with AI

This one is easy to nod along with and hard to institutionalize. AI is genuinely good at the unglamorous part: converting a messy incident thread into a scorecard, a staffing map, or a runbook that did not exist yesterday. The trick is refusing the generic output. Feed it your real incident, your real customer environments, and your real org chart, and make it argue with you. Paste raw material, not summaries, and strip customer names first.

Score a real incident with the Executive Test
Here is the raw thread and timeline from a recent incident: [paste]. Score it on four dimensions with a number and one sentence of justification each: frequency (how often should we expect this class of problem), consequence, detectability (what evidence would have shortened diagnosis), and repeatability (what would make the next occurrence materially easier). Then place it in one of four quadrants and tell me the remediation that fits the quadrant. If my instinct is to build permanent tooling for something that lands in the low-frequency, low-consequence quadrant, say so bluntly.
Separate what we controlled from what we didn’t
From the same incident material, build a two-column table: what was inside our control versus what depended on the customer or a vendor. For each row, note whether we had visibility into it at the time. Then tell me which of our timeline delays are legitimately attributable to us and which are attributable to waiting on somebody else’s environment. Do not soften this in either direction.
Map skills against frequency, honestly
Here are the specialty areas that showed up in our last twelve months of incidents: [list them]. For each, estimate how often we actually needed the expertise, then recommend one of four models: full-time internal, internal or shared specialist, fractional specialist, or documented escalation path only. Flag any area where we are currently paying for full-time expertise we need twice a year, and any area where we have no path at all to expertise we need monthly.
Turn diagnosis into a runbook that will actually get used
Convert this incident thread into a runbook for the next person who sees these symptoms. Structure it as: symptoms as first reported, the first five checks in order, the evidence to capture before restarting anything, the decision point where this leaves our expertise, and who to escalate to. Keep it under one page. No background narrative. Write it so somebody at 2am can follow it.
Rewrite an RCA that only says what broke
Here is our current RCA document: [paste]. Rewrite it to answer these as well: why was this hard to diagnose, was the required knowledge common or specialized, how recently had anyone used it, which evidence shortened the diagnosis and whether we got it by luck or design, what knowledge should be retained and where, and what capability should be internal, fractional, or external. Remove every sentence that assigns fault to an individual and replace it with the organizational capability that was missing.
Find the “rare” problem that isn’t rare anymore
Here are titles and short descriptions of our last thirty incidents: [paste]. Group them by underlying problem class rather than by symptom or component. Show me any class that appears three or more times, and for each, state plainly whether it should still be treated as an edge case or has become an architectural decision we keep deferring. Then draft the two-paragraph case I would take to my leadership team for the worst offender.
Coach the question, not the answer
I am about to review an incident with my engineering leaders. Given this material [paste], write the eight questions I should ask, ordered so the conversation stays diagnostic rather than defensive. For each, add the answer that would reassure me and the answer that should worry me. Then list three questions I am likely to ask out of frustration that would produce no useful information, and give me a better version of each.