Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety

This collection of recent CS/SE preprints highlights multiple security-relevant findings around LLMs and developer tooling. SafeAudit introduces an LLM-driven enumerator and a non-semantic metric (“rule-resistance”) to meta-audit LLM agent tool-call safety, finding >20% residual unsafe behaviors across benchmarks and environments and showing coverage improves with more tests. A separate study on LLM-assisted security code review demonstrates strong confirmation bias: framing a change as bug-free reduces vulnerability detection by 16–93%, and adversarial pull requests can reintroduce known CVEs