Using AI Agents to Automate Black-Box Audits of Personalization Algorithms at Scale

Paper presents a framework that uses generative AI agents as synthetic behavioral engines to run large-scale black-box audits of personalization algorithms. The authors instantiate 1,120 agents across 14 grounded personas and three counterfactual conditions on X, collecting >200,000 content exposures. They find the algorithmic feed amplifies toxic, polarizing, political, and right-leaning content relative to chronological ordering, with amplification varying sharply by persona ideology; demographic signals produce subgroup-dependent effects in counterfactual tests. Method enables causal, high‑