A VP of Engineering at a Canadian fintech told me about a senior ML engineer they had hired through a well-known staffing firm. The candidate had aced every interview: sharp answers, impressive credentials, an offer letter signed in under two weeks. Six months later, the engineer had shipped nothing that worked in production.
Not because they lacked effort. Because they had never actually built a production ML system. They knew how to talk about it — fluently, convincingly — but the recruiter who conducted the technical screen had no way to know the difference.
“We asked the wrong questions,” the VP told me. “Or more accurately, we didn’t know the right questions. And neither did the people doing the screening.”
This story is not unusual. It is, in fact, the norm.
A Market That Has Outrun Its Hiring Infrastructure
The technical talent market — particularly in AI, machine learning, and software engineering — has evolved faster than the hiring infrastructure designed to navigate it. Between 2020 and 2025, demand for ML engineers, data scientists, and AI architects grew at a pace that outstripped supply, created credential inflation, and made the gap between claimed and actual capability wider than at any point in recent memory.
By 2026, organizations across Canada and the US are spending record amounts on technical talent — and still getting it wrong with alarming consistency. Hiring cycles stretch to 90 days. Mismatches are common in the first year. Attrition rates for technical roles remain elevated. And for every team that successfully hires a strong practitioner, two others are managing the fallout of a hire that looked right on paper and failed in practice.
The conventional explanation is supply. There is not enough talent. The pipeline is too thin. Competition from FAANG and hyperscalers is squeezing mid-market companies out. This is partially true. But it misdiagnoses the core problem.
Why the Standard Hiring Model Is Structurally Broken
The prevailing technical hiring model was built for a different era. In most organizations, the process works like this: a business stakeholder defines a job requirement; an internal HR generalist or external recruiter conducts an initial screen; a technical panel — when it exists — evaluates finalists.
The problem is the screen. The recruiter — whether internal or external — is almost always a generalist. They are skilled at sourcing, relationship management, and process coordination. They are not equipped to evaluate whether a candidate actually understands transformer architecture, can explain why a gradient boosting model overfit in production, or has real experience operating ML pipelines at scale.
So they evaluate what they can see: keywords on a resume, years of experience in adjacent roles, confidence in conversation, and the perceived prestige of prior employers. The screen is built around signal that is legible to a non-practitioner.
This is not a criticism of recruiters. It is a structural critique of a process that asks people to evaluate expertise they do not possess. No amount of competency frameworks, interview scorecards, or AI-powered screening tools resolves the fundamental problem: if the person running the evaluation doesn’t understand the domain, they cannot reliably identify domain capability.
AI Roles Have Amplified Every Existing Weakness
The rapid expansion of AI-related hiring has exposed these weaknesses at scale. Organizations across every sector are now hiring for roles that did not exist five years ago: AI product managers, MLOps engineers, LLM fine-tuning specialists, AI ethics leads, prompt engineers, RAG architects. The job descriptions are often written by people who do not fully understand what the role entails. They are screened by recruiters with no background in the domain. And the technical panels — when they exist — are frequently staffed by engineers who work adjacent to AI but not in it.
The result is a market defined by surface-level matching. Candidates learn to speak the language of the job description. Hiring teams confuse fluency with capability. Offers get extended. Roles get filled. And the problems emerge three to six months later, when the gap between claimed expertise and actual performance becomes impossible to ignore.
The compounding factor: AI talent is expensive. A misfire in this category does not just cost a salary. It costs time-to-capability, team morale, delayed deliverables, and in some cases, the credibility of the entire AI initiative.
The Alternative: Evaluation That Matches the Expertise
The organizations consistently making strong technical hires — across AI, engineering, and adjacent domains — share a structural characteristic that rarely appears in job posts or vendor marketing: the people doing the evaluation understand the work.
This is not a novel concept. In medicine, surgeons are credentialed by surgeons. In law, bar exams are designed by lawyers. But in technical hiring, especially at the speed and scale demanded by digital transformation and AI adoption, this principle is frequently abandoned in favor of process efficiency.
Practitioner-led evaluation reintroduces domain expertise at the screening stage — not just the final interview. It means:
Initial technical screening conducted by someone who has built in the domain
Evaluation criteria written by practitioners, not copied from generic competency libraries
Interview questions that probe decision-making under real constraints, not theoretical knowledge
Assessment of how candidates reason about tradeoffs — not just whether they can define acronyms
What Rigorous Technical Evaluation Actually Looks Like
In practice, the difference between a generalist screen and a practitioner screen is not in the volume of questions — it is in the nature of the questions and the evaluator’s capacity to interpret the answers.
For a senior ML engineer, a practitioner evaluator might probe:
Walk me through a model that failed in production. What caused it? How did you diagnose it? What did you change?
How did you decide between approach A and approach B in your last deployment? What did you give up?
Describe a situation where the data told you one thing and the business needed another. How did you navigate that?
These questions have no correct answer that can be memorized. They require genuine experience to answer well, and genuine domain knowledge to evaluate. A generalist recruiter with a scorecard cannot meaningfully assess the responses. A practitioner can — and often can do so in the first fifteen minutes.
The result of rigorous practitioner-led evaluation is not just better hiring accuracy. It is faster hiring — because false positives don’t advance to final rounds, and strong candidates are identified earlier. It is also a better candidate experience: practitioners ask better questions, and strong candidates recognize and appreciate that they are being evaluated by someone who understands their work.
The Evaluation Gap Is Closeable
The talent gap that organizations across North America are struggling with in 2026 is real. But it is not primarily a supply problem. It is an evaluation problem — a structural mismatch between the sophistication of the roles being filled and the sophistication of the process being used to fill them.
Closing this gap does not require larger talent pipelines or bigger sourcing budgets. It requires redesigning who does the evaluation and what they evaluate for. It requires bringing domain expertise into the screening stage, not just the final interview. And it requires treating technical evaluation as a core competency — one that cannot be outsourced to process without losing signal.
The organizations that recognize this shift are already seeing the results: shorter time-to-productivity, lower first-year attrition, stronger team capability, and AI initiatives that deliver what they promised.
The talent you are looking for exists. The question is whether your evaluation process can find them.