arXiv：cs.LG· Michael C. Abbott, Benjamin B. Machta·· 11 小时前
全晶体管物理学习 Transformer 可训练完成上下文学习
An all-transistor physical-learning transformer can be trained to perform in-context learning
AI 导读
研究者构建了一种全晶体管物理学习 Transformer，通过引入额外电路元件恢复双向信息流，并利用亚阈值 MOSFET 的指数电流定律实现注意力机制。仿真显示，该电路在用交叉熵损失和 Lion 优化器训练后，可完成简单的上下文学习任务，尽管反向元件只传递近似梯度。
正文
Abstract:Most systems trained by equilibrium propagation or coupled learning are in the same class as multi-layer perceptrons: while they may be universal function approximators, they struggle to learn associative tasks which are easy for transformers. Equilibrium propagation relies on a bidirectional flow of information through the physical system between input and output. The reverse flow is broken by more complex nonlinear systems, such as those performing multiplication which is needed to express dot-product attention. Here we explicitly restore this bidirectional flow by introducing extra circuit elements for this purpose into a double or dyadic circuit. We then use these ideas to construct an attention mechanism exploiting the exponential current law of sub-threshold MOSFET devices. We show that in simulations this circuit can be trained to perform a simple in-context learning task, despite using realistic reverse components which transmit only an approximate gradient. For this training to succeed we need cross-entropy loss (not mean squared error). We generate nudges appropriate for this loss using a novel floating circuit element, based on the same MOSFET sub-circuit used for softmax. We train the resistors in this circuit using the Lion optimiser, which can be implemented using one capacitor per parameter to store local memory of past gradient estimates.
| Comments: | 18 pages, 11 figures, plus appendices |
| Subjects: | Machine Learning (cs.LG); Soft Condensed Matter (cond-mat.soft) |
| Cite as: | arXiv:2610.12571 [cs.LG] |
| (or arXiv:2610.12571v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12571 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Michael Abbott [view email]
[v1]
Thu, 8 Oct 2026 18:01:05 UTC (739 KB)
来源：arXiv：cs.LG · arxiv.org