AI Bias in ATS Screening: What Actually Protects You in 2026

Curriculo ATS · 2026-05-01

What the laws actually require

NYC Local Law 144 took effect in July 2023. It applies to any employer using automated employment decision tools to screen candidates for jobs in NYC, including remote roles tied to an NYC office. The law requires an annual bias audit by an independent auditor (vendors cannot audit their own tools), public disclosure of audit results on the employer's website, candidate notification that AI is being used, and an alternative process for candidates who request one. The Department of Consumer and Worker Protection enforces it, with civil penalties of $500–$1,500 per violation, per day.

The EU AI Act, finalized in 2024, classifies employment AI as "high-risk" under Annex III. High-risk systems must meet technical documentation standards, enable meaningful human oversight, achieve accuracy and robustness benchmarks, and inform workers' representatives before deployment. The full obligations for Annex III systems become enforceable on August 2, 2026.

Both laws share one core requirement: the AI's decision must be explainable and auditable. You need to be able to say, for any specific candidate, why the AI scored them the way it did. That is the standard most legacy ATS platforms cannot meet.

Why legacy ATS keeps failing the explainability test

Most legacy ATS platforms — Greenhouse, Lever, Workable, Ashby, Manatal — use some form of keyword-based matching under the hood. Boolean token extraction, TF-IDF scoring, embedding-based semantic similarity. These methods are computationally straightforward. They are also hard to audit in plain English.

If your ATS tells you Candidate A scored 82 and Candidate B scored 47, neither system can tell you in human language why. The reasoning lives in floating-point vectors and weighted keyword matches. When a candidate asks "why was I screened out," there is no defensible answer. When an auditor asks "show me the decision logic for this protected class," you can only sample statistically.

Three structural failures make legacy ATS audit-fragile:

  • Black-box scores. A number with no narrative is not an explanation. The candidate cannot challenge it; the auditor cannot validate it.
  • Vocabulary drift bias. Keyword matching systematically penalizes candidates whose resumes use different vocabulary than the JD — which correlates with non-traditional backgrounds and underrepresented groups.
  • No audit trail per decision. Most systems log that a score was produced, not what evidence drove it. That gap is the heart of the compliance problem.

What we learned at Amazon about explainable ranking

Before founding CurriculoATS in 2024, our founder Dev built ranking systems at Amazon for search and recommendations — the same problem class as resume screening. The lesson that shaped how we built CurriculoATS came from a research direction Amazon's recommendation teams pursued years before "explainable AI" became a buzzword: a ranker that cannot tell you why it ranked something will eventually rank wrong, and you will not catch it.

The fix is structural, not cosmetic. Every score has to be paired with a reasoning paragraph that a human can read in seconds and verify against the underlying evidence. For hiring, that paragraph names the quantified achievements the model saw, the experience-relevance signals it weighed, the career-trajectory pattern it identified, and the skills-alignment evidence it found. It also says what was missing. A hiring manager reading the paragraph in our Impact Scoring view can override a wrong call in 10 seconds. An auditor reviewing a population can verify each decision was made on legitimate, job-related signals — not on proxies for protected attributes. That is the architecture both NYC Local Law 144 and EU AI Act Annex III actually demand. Most legacy ATS platforms bolted AI on top of keyword pipelines. We built the explainability into the scoring itself.

What an actual NYC Local Law 144 audit costs and contains

Founders ask two questions when they realize they need a bias audit: how much does it cost, and what does the auditor actually look at. Independent bias audits for AEDTs in NYC typically run $5,000-$25,000 depending on the complexity of the tool, the size of the candidate dataset, and whether the auditor needs to ingest custom logs from a legacy ATS.

The audit examines selection rates across protected categories (race, ethnicity, gender) at every stage where the AEDT influenced a decision: initial screening, scoring, advancement to interview. The auditor calculates impact ratios and flags any group whose selection rate falls below 80% of the highest-scoring group — the standard "four-fifths rule" the EEOC has applied to employment decisions since 1978.

The audit report becomes a public document the employer must post on their website, with candidate notice provided before any AEDT is used. Failed audits do not automatically trigger penalties, but they obligate the employer to either remediate the disparate impact or stop using the tool. Most published 2024-2025 audits we have reviewed found impact ratios in the legally acceptable range, but a meaningful subset showed gender-based disparities in technical role screening that vendors did not catch in internal QA.

The structural fix is the same one explainability requires: per-decision logging that lets the auditor sample populations and trace impact to specific scoring inputs. Tools that produce only aggregate scores without per-decision reasoning often cannot complete the analysis at all, which is itself a failure mode.

What founders should do this quarter

If you hire NYC candidates or expect to hire EU candidates after August 2026, treat compliance as a 90-day project, not an annual review item. Five concrete steps:

  • Inventory your AI surface. List every tool that produces a candidate score, ranking, or recommendation: ATS, sourcing tools, assessment platforms, AI interviewers. All of them are in scope.
  • Verify explainability. For each tool, ask the vendor: can you produce, on demand, a human-readable explanation of why a specific candidate was scored a specific way? "It is in the algorithm" is not an answer.
  • Schedule the bias audit. If you use AEDTs in NYC, the law requires an annual audit by an independent auditor. The audit cost is typically $5,000–$25,000 depending on tool complexity and dataset size. Budget it.
  • Add candidate notice and alternative process. The notice can be a single sentence on your application page: "This role uses automated tools to assist with initial review." The alternative process is a contact email candidates can use to request human-only review.
  • Document the human-in-the-loop. Both laws expect meaningful human oversight, not rubber-stamping. Capture the moment a hiring manager either accepts or overrides each AI recommendation. CurriculoATS logs this by default; for legacy ATS, you may need to bolt this in.
FAQ

Frequently asked questions

Does NYC Local Law 144 apply to me if I'm not in NYC?

It applies if you hire candidates for NYC-based jobs or for remote roles tied to an NYC office. A startup headquartered in Austin with a single remote employee whose listed work location is NYC is in scope for that role. The safer posture for any US startup with national hiring is to assume similar laws will reach you within 24 months — California, Illinois, Maryland, and Colorado have already passed adjacent statutes.

Does using a 'human-in-the-loop' exempt me from the EU AI Act?

No. The EU AI Act treats employment AI as high-risk regardless of whether a human approves the output. The human-in-the-loop reduces risk but does not exempt the system. The Annex III obligations on documentation, accuracy, robustness, and worker notification still apply. The August 2026 enforcement date is firm.

What does an explainable AI score actually look like?

A useful explanation is one paragraph, in plain English, that names the evidence: "This candidate scored 78 because their resume documents three quantified product launches in the last 24 months matching the job's core scope; experience relevance is high based on recent SaaS roles at companies under 200 people; career trajectory shows growth from individual contributor to team lead. Skills alignment is moderate — strong on Python, weak on the Kafka requirement listed." If your ATS cannot produce a paragraph like that, it is not explainable in the regulatory sense.

Are bias audits worth it if my candidate volume is small?

The law applies regardless of volume. A startup with 30 NYC applicants per year still needs an annual audit if it uses an AEDT. The good news: small dataset audits cost less. The harder question is whether your tool can produce the data the auditor needs. If your ATS does not log per-decision reasoning, the auditor will tell you they cannot complete the analysis — and you fail the audit by default.

What is the difference between bias-aware and bias-free AI in hiring?

There is no such thing as bias-free AI in hiring; any system trained on historical hiring data inherits the biases in that data. Bias-aware AI is the realistic goal: a system that exposes its reasoning so bias can be detected and corrected, monitors selection rates across protected categories continuously, and makes human override the default rather than the exception. Vendors who claim their AI is bias-free are either marketing or unaware of how their training data was built. The defensible posture under both NYC Local Law 144 and the EU AI Act is bias-aware with documented monitoring, not bias-free.