Meta Deploys AI-Driven Bug Detection System

What happened

Meta has integrated an AI-powered testing framework into its production environment to catch software errors before deployment. The system, known as Catching JiTTests, uses large language models to generate executable tests that detect behavioral differences in code changes.

The system has been operational for five months and is now a core part of Meta's infrastructure. It employs two workflows: one that identifies code changes as potential bugs and another that infers the developer's intent.

Why it matters

The system is designed to address the limitations of traditional testing methods, which are insufficient for Meta's vast codebase. By detecting bugs early, the system reduces the workload on engineers and minimizes the risk of deploying faulty code.

Key details

Catching JiTTests filters false positives using a combination of rule-based methods and an ensemble of state-of-the-art LLMs called LLM-as-Judge. The system also incorporates a rule-based approach named RubFake to provide deterministic judgments.

Engineers at Meta report a true positive rate of 60% from the system, which targets risky code changes by running overnight on spare capacity. The system focuses on diffs most likely to contain severe regressions.