XTX Markets 作为处理每日交易量达 2500 亿美元的最大算法交易公司之一,其价格预测和其他算法交易数据依赖约 650+拍字节的存储空间,现已开源其 Linux 文件系统。XTX 在超越了原始 NFS 使用和其他文件系统替代方案后,开发了 TernFS 用于分布式存储。
TernFS 是一种分布式文件系统,用于处理读写大型不可变文件——在创建后通常不会被修改的不可变性,以及至少几兆字节的大型存储需求。TernFS 专为 XTX 数据中心设计,可实现约 10EB 逻辑文件存储、约一万亿个文件和 1000 亿个目录,并支持约一百万客户端同时运行,所有这些均基于通用硬件和以太网网络。
TernFS 还能够在多个地理区域中扩展,通过冗余存储数据以防止驱动器故障,没有单个元数据故障点,以及满足 XTX Markets 对苛刻需求的其他特性。
TernFS 的开发历时超过三年,在过去的一年里已被所有 XTX 机器学习项目使用。XTX Markets 报告称:“到目前为止,我们在 TernFS 上从未丢失过一个字节的数据。”
那些希望了解更多关于这款原生 Linux 的 TernFS 文件系统的信息(该系统现已开源)的人,可以通过 XTX Markets 博客获取。TernFS 在 GPLv2+和 Apache 2.0 许可证下可用。TernFS 文件系统已在 GitHub 上开源。
A Major Trading Firm Has Open-Sourced The Latest Linux File-System: TernFS
XTX Markets as one of the largest algorithmic trading firms that handles $250 billion in daily traded volume and relies on around 650+ petabytes of storage for its price forecasts and other algorithmic trading data has open-sourced its Linux file-system. XTX developed TernFS for distributed storage after they outgrew their original NFS usage and other file-system alternatives.
TernFS is a distributed file-system for dealing with reading and writing large immutable files -- immutable in the context that they are typically not modified after being created. And large in the context of being at least a few megabytes. TernFS is designed for XTX data center needs of maxing out at around 10EB of logical file storage, around one trillion files and 100 billion directories with around one million clients. All running atop commodity hardware and Ethernet networking.
TernFS is also able to span across multiple geographic regions, store data redundantly to protect against drive failures, no single point of failure for metadata, and other features for the demanding needs of XTX Markets.
TernFS has been in development for over three years and for over the past year is being used by all of the XTX machine learning efforts. XTX Markets reports that "to this day, we haven't lost a single byte" on TernFS.
TernFS logo
Those wishing to learn more about this Linux-native TernFS file-system that is now open-source can do so via the XTX Markets blog. TernFS is available under GPLv2+ and Apache 2.0 licenses. The TernFS file-system is open-source on GitHub.
来源:Phoronix
--
FROM 171.221.52.*