arXiv：cs.CL· Malte Ostendorff·· 11 小时前
面向 LLM 预训练数据的主机感知网页抓取
Host-Aware Web Crawling for LLM Pretraining Data
AI 导读
研究发现，LLM 预训练数据抓取中约一半的质量分类器分数差异发生在主机之间，而 Common Crawl 用于指导抓取分配的调和中心性指标相比随机排序收益有限、对部分分类器甚至更差。
正文
Abstract:Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data. Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl's per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation. Finally, Common Crawl's observed per-host coverage has little correlation with the evaluated quality signals. We estimate that tens of millions of high-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts.
| Comments: | Accepted at WaC-13 @ EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13321 [cs.CL] |
| (or arXiv:2610.13321v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13321 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Malte Ostendorff [view email]
[v1]
Fri, 9 Oct 2026 10:03:04 UTC (163 KB)
来源：arXiv：cs.CL · arxiv.org