Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation

Collection of recent papers auditing and proposing methods for tool-using LLM agents, memory governance, benchmark validity, and LLM-driven pentesting. Key findings with security relevance: (1) Tool-calling benchmarks (BFCL, τ2-Bench, LiveMCPBench, MCP-Atlas) show high evaluator misalignment and nondeterministic scoring that can misrepresent agent capability and lead to unsafe deployment decisions; (2) GovMem demonstrates that naive memory promotion from agent traces causes large false-promotion rates, arguing for dependency-aware, review-gated governance to avoid propagating erroneous or copy