The application-forty problem
Michael Brooks
HR Consultant
Latest update
22 min read
IN THIS ARTICLE
The Four Fields Problem
Proxies that survive annoymization
When the rubric itself is the bias
Feedback loops, and why they hide
What monitoring actually catches
A short checklist
10 min remaining
Almost every hiring process reaches the same point: a large pool of applications must be narrowed down to a small group for review. It may seem like the main task is simply finding the strongest candidates. But once the shortlist becomes limited to forty applications, the filtering decisions themselves start shaping the outcome — and that is where the application-forty problem becomes visible.
Each of these decisions affects who reaches the shortlist. Some also expose the rules used to eliminate applications. The two can be difficult to distinguish, because once the pool is narrowed to forty candidates, every earlier filter changes the remaining mix — and that remaining group shapes the next stage of evaluation.
The four fields problem
onsider what remains after the shortlist is reduced to forty: candidates who advanced, applications that were screened out, changes in ranking, rejection rates, reviewer comments, incomplete profiles, and the recurring patterns that show which types of applicants consistently make it through the filter.
Each of these decisions influences who reaches the final group. Some also expose the rules behind the screening process. The two effects can overlap, because when hundreds of applications are narrowed to forty, every filtering choice changes the candidate mix — and that mix determines who receives further attention.
The test we use internally
Take a shortlisted application and give it to two reviewers for independent assessment. Ask each reviewer to explain why they would move the candidate forward or reject them, and whether they would assign the same ranking. If their judgments differ, the screening process may be inconsistent as well — and repeated evaluations can reveal where those differences begin to change the final shortlist.
Proxies that survive anonymisation
These are the factors we see most often influencing which candidates survive the forty-application cutoff, ordered roughly by their potential impact and how easy they are for hiring teams to overlook:
Employment gaps. A prolonged break between roles can affect whether an application reaches the final forty. Screening systems may favour uninterrupted career histories and treat gaps as weaker evidence, even when the candidate has relevant skills and experience. When only a small number of applications can advance, that preference can have a noticeable effect on who remains.
Postcode and commute distance. Where a candidate lives can influence whether they remain within the final forty. A screening process that favours applicants closer to the workplace can quietly introduce location-based preferences, pushing otherwise similar candidates lower in the ranking before their qualifications are fully considered.
Prestige signals in prose. Well-presented CVs can use polished language, quantified results, confident phrasing and carefully structured sections. Those signals may affect which candidates stay within the final forty. If the screening process rewards presentation over substance, applicants with stronger coaching can outrank candidates whose actual experience is equally or more relevant.
Non-linear careers. A candidate who enters a new field later may have fewer years of direct experience than someone who started earlier, but that does not necessarily mean lower capability. When screening focuses heavily on career length, it can mistake a different path for weaker experience and remove qualified candidates before they reach the final forty.
Volunteer and interest lines. Community projects, sports teams, professional groups and personal interests can appear harmless during screening. Yet these details may still influence how an application is perceived, especially when the shortlist is tight. If the system picks up those patterns, seemingly unrelated information can affect which candidates remain in the final forty.
When the rubric itself is the bias
The most revealing result from our shortlist analysis is that much of the measurable variation does not come from the scoring model itself. It comes from the rules used to decide which applications are worth considering in the first place. When those rules narrow the pool too aggressively, the final forty can reflect the filter more than the candidates.
A screening rule that requires “8+ years in a similar role” can remove qualified candidates before the shortlist is formed. Giving extra weight to startup experience can favour one particular career background, while making a specific certification mandatory can exclude people with equivalent training. The model does not need to be biased to create this outcome — it simply applies the criteria consistently, making the filter more efficient at reproducing the same exclusions.
That is why we focus so closely on the rules that determine which applications survive the first filter, rather than treating the scoring model as the only source of risk. We also track changes to those rules over time, so when shortlist rates move, the first question is what changed in the screening criteria — not whether the model itself became unstable.
Feedback loops, and why they hide
The most damaging problem is the one that builds quietly over time. If the shortlist is repeatedly shaped by the same screening rules, those rules can reinforce the profiles that were already being selected. Each round then becomes the basis for the next, gradually narrowing the candidate pool and making the original filtering choices harder to detect or question.
After months of use, the screening process may appear highly effective. It consistently identifies the same types of candidates and produces stable shortlist numbers. But that consistency can hide the real issue: repeating the same filtering decisions does not prove the process is improving outcomes. It may simply show that the system has learned to reproduce its existing selection patterns.
Stage-level shortlist monitoring. If a noticeable change appears during one screening step but disappears later, the issue may be coming from the rule used at that stage rather than the model itself. Tracking each stage separately makes it easier to identify where qualified applications are being removed.
What monitoring actually catches
The two changes were straightforward: instead of requiring an uninterrupted work history, the criteria focused on relevant experience regardless of employment arrangement. A named certification was also replaced with the certification or an equivalent qualification. As a result, more candidates passed the initial filter and reached review — without changing the original applicant pool.
The lesson is not that changing screening criteria solves every problem. It is that the issue became visible at the exact stage where applications were being filtered, while the hiring process was still underway. Waiting for periodic reporting would have revealed the pattern much later, after the shortlist had already shaped real hiring decisions.
A short checklist
Audit the screening rules before reviewing the model. Go through each requirement and ask whether it reflects something the role genuinely needs or simply removes candidates for having different backgrounds. This helps identify unnecessary filters before they determine which forty applications receive further consideration.
Test for consistency. Run the same application through the screening process more than once and compare the results. If the candidate moves significantly between rankings or crosses the shortlist threshold, treat that variation as a warning that the filtering process may not be reliably selecting candidates.
Monitor each screening stage separately, not just the overall shortlist. A combined pass rate can make it difficult to see where candidates are being filtered out, while stage-by-stage tracking shows exactly where the pool changes. That makes it easier to identify unnecessary criteria and correct the process before the final forty are selected.
Never rely on previous hiring decisions as the only training signal. A system that learns from past shortlists may become very good at reproducing the same selection patterns, without showing whether those choices identified the strongest candidates. Historical consistency measures repetition, not whether the process is producing better hiring outcomes.
Version every screening rule. If you cannot identify which criteria were applied when a candidate was evaluated, you will struggle to explain why that application was accepted or rejected later. Keeping a clear record of each version makes shortlist decisions traceable and gives the hiring team something concrete to review when questions arise.
None of this guarantees a perfect shortlist. Nothing guarantees that every screening decision will be consistent. What it does provide is visibility into where applications are being filtered differently, so those differences can be investigated, corrected and explained — rather than allowing the system to make confident decisions that no one can properly examine.
