Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search

This arXiv CS.LG feed (10 new papers) covers a range of ML research: an automated large-scale search pipeline for heterogeneous 4-expert Mixture-of-Experts (MoE4) that discovered an alphabetical-enumeration coverage bias anchored to the AirNet family and supplies a stratified-random-sampling fix and code in the NNGPT repo; an analysis of weight-space geometry for offline reasoning training showing many offline RL/distillation losses produce nearly colinear weight updates while DPO occupies a near-orthogonal direction and attains higher accuracy under the authors' protocol; a comprehensive taxo