Thomas Wolf· @Thom_Wolf · X·· 2 小时前AI 评分45
AI 导读
@microagi 关于传输 PB 级机器人数据的文章里有个有趣的数据: 他们向 Hugging Face 重新上传了一个 239 GB 的数据集,只做了小改动。网络上只传输了 2.6 MB,大约一分钟就完成了(!) 而且这在规模上持续见效,因为机器人录制数据高度重复(相同场景、相同设备)。然后 @Cloudflare workers 将其分发到任何云,超过 1 PB 仅需约 500 美元 团队写得很好:https://www.microagi.ai/articles/delivery-platform
正文 · 原文
fun stat from @microagi’s write-up on shipping petabytes of robot data:
they re-uploaded a 239 GB dataset with small changes to Hugging Face. only 2.6 MB went over the network, done in about a minute (!)
and it keeps paying off at scale, because robot recordings are super repetitive (same scenes, same rigs). then @Cloudflare workers fan it all out to any cloud, >1 PB for ~$500
great write-up by the team: https://www.microagi.ai/articles/delivery-platform
As our customers scale to training robot models using millions of hours of our recordings, we needed to build a new kind of data infrastructure. Terabytes of data delivered in minutes, and the capacity to deliver over a Petabyte per day to each of our customers, to multiple customers at the same time. That’s more than 10 Gigabytes a second, delivered into any cloud, wherever our customers need it. Using our customer portal, customers select precisely the data they want and it starts arriving in minutes at their destination, including AWS, GCP, Cloudflare, Hugging Face and any cloud provider who can handle the ingest speed. We’ve written about how we worked with @huggingface and @Cloudflare to build this secure, scalable infrastructure. Link in comments.在 X 查看被引用的帖子
来源:Thomas Wolf · x.com