Extending rule-based accessibility testing with gradient boosting

Automated accessibility testing is, almost without exception, rule-based: a fixed set of checks evaluated against a page - a contrast ratio computed and compared to a threshold, an image checked for an alt attribute, a form field checked for an associated label. Open-source engines built on this model, such as axe-core, are the de facto baseline most tools in this space build on, including earlier iterations of our own scanning.

Rule-based checks are precise for what they're built to detect and cheap to run at scale. Their limitation is structural, not a matter of adding more rules: a static rule can only evaluate what a single page looks like at a single point in time. A meaningful share of real accessibility failures live outside that frame - across steps in a flow, across many instances of one component, or in the difference between a pattern that matches a rule syntactically and one that is actually broken in context. Where that gap sits, and how we've started addressing it with gradient-boosted models (LightGBM, XGBoost), is what this note covers - along with where that choice is a genuine trade-off rather than an obviously correct one.

Why gradient boosting, and where that's a trade-off

The inputs we work with - DOM and ARIA attributes, computed contrast values, focus-order sequences, diffs between DOM states - are structured and can be turned into tabular feature vectors. Gradient boosting is a strong fit for that: it handles mixed, structured input well, trains effectively on comparatively modest labeled datasets, runs fast enough to sit inside a continuous scan, and produces feature-importance output that can be inspected rather than treated as a black box.

That last point matters more than it might seem. A deep-learning or sequence-model approach isn't incapable of this kind of task - for genuinely sequential data, it can be the more natural fit. We chose gradient boosting because it lets us train on the labeled data we actually have (largely drawn from structured manual review), keeps inference cheap enough to run continuously, and keeps every prediction traceable back to specific, explainable signals - not because tree ensembles are the only valid tool for this problem.

Flow-level evaluation

A rule evaluated against a single page has no representation of what happened on the step before it. That's where a class of real failures occurs in multi-step flows - checkout, onboarding, authentication: a dialog opens and traps focus correctly, but the following step doesn't restore it anywhere meaningful; a validation error is inserted into the DOM without a corresponding live-region announcement, in a way that looks unremarkable to a static rule but leaves the update unannounced to assistive technology.

A flow is inherently sequential, which is normally a sequence model's territory. What we do instead is collapse each step transition into a single tabular row and classify that row - a deliberate simplification, not a claim that this captures everything a sequence model would. Per transition, we compute things like: the tag and role of the focused element before and after the action, the tree distance between those two elements, whether the new focus target is visible and interactive, a structural diff of nodes added or removed, and whether an aria-live region's content changed in step with the DOM update. The model is trained on transitions labeled by structured manual review - a transition marked as broken because focus landed somewhere unreachable, or because an error appeared without an announcement - and learns which combinations of those features correlate with a failing transition.

This works because most flows we test are short - a handful of steps, not long open-ended sequences - so most of the useful signal collapses into a per-transition feature set without much loss. It would generalize worse to very long or highly variable flows, where a model that actually reasons over the sequence would likely do better. For the flow lengths we see in practice, the tabular approach has been the more practical choice given the amount of labeled data available.

Component-level evaluation

Components in a design system tend to share structural properties across instances that a per-rule check doesn't capture well: a component can satisfy every individual rule it's checked against and still fail to behave like a correctly implemented version of the pattern it represents. Evaluating across many instances of the same component lets a model pick up on structural correlations between passing and failing implementations, rather than requiring a hand-written rule for every variation a design system can produce.

This is the area most dependent on data volume: it needs enough labeled examples of working and broken implementations of a given pattern to learn from, which is a slower thing to accumulate than the manually reviewed flow transitions above. Where that labeled history is thin, the model leans more heavily on rule-derived signals as features, rather than replacing them outright.

False-positive filtering

Rule matches are syntactic - a rule fires whenever its pattern is present, regardless of whether that pattern constitutes a real problem in its surrounding context. This is a significant source of noise in raw scanner output, where only a fraction of flagged items warrant action. We use a classifier trained on the context around a match, not the match in isolation, to suppress low-value flags before they reach the findings list. Of the three areas here, this is the most straightforward fit for gradient boosting: a well-understood binary classification problem on structured features, with a large, continuously growing set of labeled examples from real scans.

What stays constant

Every finding - rule-derived or model-derived - is still mapped to a specific WCAG success criterion and conformance level. The models extend coverage and improve precision; they do not define conformance on their own, and we treat that grounding as non-negotiable regardless of how the underlying detection is implemented.

This work is ongoing rather than finished, and the trade-offs described here - particularly around flow length and labeled-data volume for component detection - are the parts we expect to revisit as the approach matures.