Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

The proposed decoding strategy reuses a preserved pre-adaptation language model to guide a speech-recognition model during inference.

Source: [7], [15]

It uses hidden-state interactions to identify tokens associated with semantic dependence and applies language-model correction selectively to those tokens.

Source: [7]

The work asks whether a speech-recognition language model can improve during inference by reusing its original language model for external guidance.

Source: [7]

The method pairs speech-conditioned hidden states with text-only hidden states from the underlying language model under matched decoded context.

Source: [14], [16]

It derives matched-layer directional similarity and relative representation-magnitude features from those paired trajectories.

Source: [4], [10], [14]

Named-entity tokens are used as a proxy for semantic dependence, while other tokens are treated as a proxy for stronger acoustic grounding.

Source: [6], [12]

Feature calibration uses a speech corpus, a multimodal speech-language model, reference-conditioned decoding, and an entity-recognition tagger.

Source: [12]

Selection is determined from a density-ratio decision region estimated for the token-feature groups, with the threshold controlling how permissive selection is.

Source: [17]

Language-model probability fusion is restricted to targeted tokens, and expanded search is used only while processing targeted words before returning to greedy decoding after beam convergence.

Source: [9], [20]

The reported decoding setup fixes the beam, interpolation, and selection-threshold settings.

Source: [18]

Evaluation uses a multimodal speech-language model across a public speech-recognition benchmark with diverse subsets and an additional speech corpus.

Source: [18]

The evaluation reports overall transcription error and named-entity error.

Source: [18]

The comparison landscape includes prior language-model-assisted speech-recognition correction systems.

Source: [8]

Baselines cover greedy and beam decoding, global correction approaches, a variant without language-model correction, and a feature-free targeted-word variant.

Source: [2], [3], [11]

In the reported comparison with beam decoding, targeted hybrid search improves named-entity recognition while maintaining comparable overall transcription error.

Source: [1]

The reported improvements include subset-specific gains and statistical-test results for both named-entity and overall transcription outcomes.

Source: [1]

Under greedy decoding, the method recovers much of the named-entity benefit associated with beam search while selectively expanding only a limited share of tokens and retaining a lower amortized search cost than full beam decoding.

Source: [11]

Global late fusion degrades overall transcription error, and naive rescoring also degrades that outcome despite possible named-entity benefits.

Source: [3]

The feature-free targeted-word baseline does not retain beam-decoding robustness, supporting the reported need for targeted correction guided by intermediate hidden-state features.

Source: [2], [3]

The approach is described as self-corrective, requiring no added parameters or further training; its demonstrated implementation is for speech recognition, while broader task use remains proposed.

Source: [19]

Feature extraction is reported to add negligible overhead relative to a model forward pass.

Source: [5]

End-to-end correction experiments cover only a single speech-language architecture, so full-method generalization across architectures remains unestablished.

Source: [13]

Feature distributions, layer aggregation, and decision thresholds may require recalibration across models and domains, with challenging settings potentially needing further adaptation.

Source: [13]