候选无关块因果注意力：让决策模型对候选顺序不再敏感

View PDF HTML (experimental)

Abstract:Decision models often score a variable-sized set of candidate actions encoded in a single sequence. This setting is increasingly relevant for System 1 components inside generative systems, where candidates may be proposed or ordered differently across runs. Standard causal cross-encoding is expressive, but it can make a candidate's score depend on serialization order rather than on the underlying decision problem. We introduce candidate-independent block-causal attention, which preserves causal computation within the shared context and each candidate while blocking cross-candidate information flow and resetting candidate positions. We compare this architecture with standard causal attention and complementary invariant baselines across Gemma 3 1B, Qwen3 1.7B, and Qwen3 4B backbones. Candidate-independent attention consistently reduces permutation sensitivity while retaining competitive decision quality; ablations indicate that candidate isolation is the primary source of the effect, with position resetting completing the intended symmetry. A larger Qwen3-4B study further examines the behavior of the proposed architecture with substantially more training data. Code is available at the \href{this https URL}{\textcolor{blue}{project repository}}, and the \href{this https URL}{\textcolor{blue}{Qwen3-4B model artifact}} is available on Hugging Face.

Comments:	Technical Report, will not be submitted to a conference
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.01601 [cs.LG]
	(or arXiv:2610.01601v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.01601 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Guy Amit [view email]
[v1] Thu, 1 Oct 2026 12:47:58 UTC (33 KB)