File Storage vs. S3 for Archival Data at Scale

When comparing on-premise file storage (either DFS – distributed filesystem or NAS – Network-attached storage) to cloud-style object storage (S3) for huge cold archives, several myths must be dispelled. In particular, tape’s cost advantage is often overstated, S3’s architectural role is misunderstood, and both DFS and S3 can be equally complex at scale. Below we update each point with recent insights and benchmarks.

Tape vs. HDD: Real-World Costs and Energy

Historically, tape (e.g. LTO) has claimed the lowest cost per TB and nearly zero idle power consumption. However, when factoring in total cost of ownership – hardware, software, redundancy, lack of compression – HDD systems using ICE-like technologies (e.g., SaunaFS with spin-down capabilities) significantly close the gap. For active archives, this energy difference is negligible. HDDs offer faster access and still achieve long-term cost-efficiency with technologies like host-managed SMR and dynamic spin-down.

S3’s True Architectural Role

S3’s primary strength is its ability to serve data over the internet using HTTPS, making it ideal for CDN and public cloud integration. However, on-premises use cases – especially where bandwidth and latency matter – gain no such benefit. S3’s RESTful interface introduces overhead not present in DFS protocols like NFS or SMB, and it is not inherently simpler or more scalable than DFS at large scale.

Operational Complexity and Scalability

At petabyte and exabyte scale, both S3 and DFS require substantial engineering. DFS platforms like SaunaFS and Hammerspace decouple metadata and data paths, enabling highly scalable deployments with clustered metadata. Operational complexity is implementation-specific – not inherent to the protocol. Modern DFS supports exabyte-scale deployments as effectively as object storage when properly architected.

DFS Scalability – Metadata and Data Separation

Modern distributed file systems like SaunaFS and Hammerspace achieve separation of metadata and data paths. SaunaFS uses custom master and chunk servers. Hammerspace leverages NFSv4.2 (pNFS) and FlexFiles, decoupling metadata control (Anvil nodes) from data I/O (DSX nodes). Clients connect directly to storage nodes, keeping metadata services out of the data path and enabling parallel throughput without bottlenecks – all while using open standards.

The Small-File Challenge

Both S3 and DFS struggle with small files, but S3’s flat, unstructured metadata makes managing billions of small objects more challenging. DFS benefits from hierarchical POSIX attributes, caching, and potentially storing small files within metadata nodes (as in HopsFS). These features simplify management and improve performance at scale.

Latency Considerations

Latency is not critical for deep archive but matters for active archives. DFS with spin-down HDDs (Pin3, ICE) typically outperforms S3 HDD-backed systems in latency, as the latter keep disks spinning without smart power management. DFS protocols avoid the additional overhead introduced by HTTPS and object API access paths, providing better responsiveness for analytics and occasional access workloads.

Summary

S3 is ideal for internet-scale distribution, but on-premise environments – especially for large-scale data tiers – benefit from distributed file systems like SaunaFS. These modern DFS architectures scale metadata and decouple control from data using either own protocols or standards like NFSv4.2. They offer superior performance, lower latency, and comparable TCO to tape-backed object storage, especially when paired with SMR HDDs and intelligent power-saving technologies.

Author: David Gerstein
David Gerstein is the CTO of Leil, an Estonia-based storage infrastructure company building HDD-native software for the exabyte era. Leil is in active co-engineering partnership with Western Digital, validated across the WD, Seagate, and Toshiba drive ecosystems.