arXiv：cs.AI· Muhammad Faraz Shoaib, Muhammad Qasim, Raisulhaq Mohammed Rizwan, Rahmatullah Safdar, Muzammil Adnan Shaik, Abdul Aleem Mohammed·· 6 小时前AI 评分34
LOGIC：面向航空航天电气系统意图驱动变更影响的 LLM 基准
LOGIC: An LLM Benchmark for Intent-Grounded Change Impact in Aerospace Electrical Systems
AI 导读
研究者提出 LOGIC，一个用于航空航天电气系统变更影响的受控基准与评测框架，让本地可部署语言模型先将工程请求锚定到确定性候选变更清单，再通过类型化电气追溯图传播。
正文
Abstract:Aerospace electrical-design revisions can contain multiple genuine changes, although an engineering request may authorize only a subset. Propagating every detected difference can therefore produce overly broad impact reports. We present LOGIC, a controlled benchmark and evaluation framework in which locally deployable language models ground a request in a deterministic candidate-change inventory before selected changes are propagated through a typed electrical traceability graph. This separation permits candidate-selection errors to be distinguished from downstream propagation errors. LOGIC contains 168 scenarios, including 144 selection and 24 abstention cases. We evaluate three 7--8B models against intent-agnostic, lexical, and structured-evidence methods, with an oracle-root upper bound. On 96 explicitly anchored selection cases, gate-only structured evidence achieves candidate F1 of 1.0000, compared with 0.9677 for token-lexical matching. On 12 relational-paraphrase cases, token-lexical F1 is 0.1772 and gate-only F1 is 0.0000, compared with 0.5000--0.6400 for the large language models. Model grounding degrades as candidate inventories grow from 4 to 64 changes, while affected-element and typed-path accuracy remain comparatively stable when frozen selections are replayed over graphs of approximately 1K to 100K nodes. Strict evidence gating suppresses false positives but can remove correct semantic selections. An exploratory evidence-empty abstention policy raises strict abstention accuracy to 0.6667 for all three models and reduces unsafe-report rates to 0.1667, while decreasing answerable-case coverage by 16.0--27.1 percentage points. Four of six conflicting requests remain unsafe for each model. These findings support combining literal evidence and language-model reasoning with engineering review when intent cannot be established reliably.
| Comments: | 14 pages, 4 figures, 4 tables |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.07580 [cs.AI] |
| (or arXiv:2610.07580v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07580 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Muhammad Faraz Shoaib [view email]
[v1]
Tue, 6 Oct 2026 01:14:09 UTC (1,952 KB)
来源：arXiv：cs.AI · arxiv.org