arXiv：cs.CL· Amirhossein Vahdat, Thierry Badard, Luc Lamontagne·· 11 小时前
GeoFitBench-LLM：评估大语言模型地理空间数据需求获取能力的基准
GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks
AI 导读
研究团队推出 GeoFitBench-LLM 基准，用于评估 LLM 能否从自然语言地理分析请求中推断所需数据集、支撑资源与约束条件，基于时空关系与语义锚定两个推理维度，并采用与数据提供方无关的固定 schema 编码标准答案。在极简提示下测试七个模型，GPT-5.4 Thinking 表现最佳，66.67% 的请求能完整召回全部资源；随着推理需求增加，准确判断数据适用性仍然更困难。
正文
Abstract:Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. This omission is critical as retrieval oriented geo-analytic tasks cannot be supported reliably without first identifying the necessary datasets and constraints. GeoFitBench-LLM is introduced as a benchmark for assessing whether LLMs can infer the datasets, the supporting resources and the constraints required by geo-analytic requests in natural language. The benchmark is structured around two reasoning dimensions, Spatio-Temporal Relational and Semantic Grounding, reflecting long-standing challenges in geospatial information retrieval and related natural language tasks. Ground truth requirements are encoded in a fixed, provider-independent schema for reproducible evaluation of resource identification and attribute-level specification. Experiments on seven models under minimal prompting indicate that identifying the necessary resources is often achievable, with GPT-5.4 Thinking achieving the best result by recovering the complete set of resources in 66.67% of benchmark requests. Results further indicate that accurate fitness-for-use specification remains more difficult as inferential demands increase.
| Comments: | Presented as an Oral Presentation Paper at GeoAI 2026, Ghent, Belgium. Also available on Zenodo |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.12669 [cs.CL] |
| (or arXiv:2610.12669v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12669 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Proceedings of the 1st International Conference on Geospatial Artificial Intelligence (GeoAI 2026) |
| Related DOI: | https://doi.org/10.5281/zenodo.20129011
DOI(s) linking to related resources |
Submission history
From: Thierry Badard [view email]
[v1]
Thu, 8 Oct 2026 19:45:40 UTC (912 KB)
来源：arXiv：cs.CL · arxiv.org