Choosing the Right Storage on AWS

Most AWS storage advice starts in the wrong place. It starts with service names.

S3 for objects. EBS for block. EFS for files. DynamoDB for key-value. FSx for high-performance file systems. That is technically true, and still not enough to design a system that behaves well under pressure.

The storage decision that matters is not "which box on the AWS diagram should hold the data?" The better question is: what does the application believe about the data?

Does it need to overwrite small ranges in place, or does it write whole objects? Does it need file locks? Can readers tolerate a second copy of the truth? Is the write path synchronous with a user request? Does the system need sub-10 ms reads, or does it need cheap durability for data that may be read next quarter? Who owns backup, restore, compaction, schema changes, hot partitions, lifecycle rules, and tenant isolation?

Those questions decide the storage service more reliably than any comparison table.

AWS has a solid storage decision guide, and the Well-Architected guidance says the same thing in plainer architecture language: choose storage from access patterns, not from habit. The hard part is translating that into decisions when a real workload has mixed reads, old code, cost pressure, and one team that says "just mount a filesystem" while another says "just put it in S3."

This is the way I would reason about it.

Start With The Contract

Every storage choice is a contract between the application and the data plane.

S3 gives you an object API. You put and get objects by key. You do not get a POSIX filesystem, block-level mutation, or cheap random writes into the middle of a file. That constraint is not a flaw. It is why S3 works so well for blobs, logs, data lakes, backups, static assets, model artifacts, and durable exchange between systems.

EBS gives one EC2 instance a low-latency block device in one Availability Zone. It is a good fit when the application expects a disk: databases, queues, search nodes, self-managed filesystems, and old software that wants to control its own I/O. It is also a trap if you treat it as shared storage or forget that the volume lives in one AZ.

EFS gives many clients a shared NFS filesystem across AZs. That is useful when the application really needs shared file semantics: home directories, content management systems, shared plugin directories, build artifacts, and container workloads that need ReadWriteMany access. It is rarely the cheapest place to dump large immutable data.

FSx gives managed file systems with stronger opinions. FSx for Windows File Server is for SMB and Windows semantics. FSx for Lustre is for high-throughput compute against file data, often next to S3. FSx for NetApp ONTAP and OpenZFS exist because some enterprise workloads are not just "files", they are snapshots, clones, protocols, quotas, and migration constraints wrapped into years of operational expectation.

DynamoDB is not usually described as "storage" in the same breath as S3 or EBS, but in application architecture it often is the storage layer. It is where you put state that is small, keyed, latency-sensitive, and shaped by known access patterns. The AWS Database Blog's piece on single-table versus multi-table design is worth reading because it forces the right discipline: write down the access patterns before you pick the key design.

The first decision is the interface. Object, block, file, database, cache, stream, archive. Once you choose the interface, you inherit its operational model.

Latency Is A Shape, Not A Single Number

Teams often ask whether a service is "fast." That hides the useful detail.

Fast for what size request? From how many clients? Against what consistency requirement? With what tail-latency tolerance? Reading one 4 KB item for an API response is not the same problem as scanning 40 TB for analytics, even if both read data.

S3 can deliver very high aggregate throughput. AWS documents that S3 request performance scales per prefix, and the old trick of randomizing prefixes is no longer required for normal performance scaling. That matters for data lakes, media, logs, backups, and parallel readers. It does not mean S3 is the right primary store for a user-facing lookup that must complete in 15 ms at p99.

EBS is closer to the instance and exposes IOPS, throughput, queue depth, volume type, and instance limits. The EBS volume type documentation is useful because it separates the workload shapes. gp3 is often the boring default because capacity, IOPS, and throughput can be provisioned independently. io2 Block Express is for the workloads where outlier latency and high IOPS justify the bill. Even then, the EC2 instance can cap you before the volume does. A 3,000 IOPS volume attached to an instance that cannot push the traffic is a paper configuration.

EFS has its own performance shape. It can serve many clients and scale throughput, but it is still a network filesystem. AWS calls out Elastic Throughput for spiky or hard-to-forecast workloads in the EFS performance docs. That is helpful, but it does not remove the cost and latency profile of NFS. A workload with thousands of tiny metadata operations can behave very differently from one streaming large files.

FSx for Lustre is a good example of why "fast storage" is too vague. It is fast for high-performance computing, ML, simulations, and analytics workloads that need a parallel filesystem. AWS emphasizes its native S3 integration, which is the real design pattern: keep durable source data in S3, stage hot compute access through Lustre, then write results back. That is very different from using Lustre as the permanent system of record for a web application.

Latency choices become cleaner when you state the read path in full. "The API reads one customer record by ID and must return in 80 ms" points toward DynamoDB, Aurora, ElastiCache, or a search index, not S3. "The batch job reads 20 TB of Parquet with 1,000 workers" points toward S3. "The simulation reads millions of shared files from many compute nodes" may point toward FSx for Lustre. Same cloud, different physics.

Shared Filesystems Are Often A Migration Smell

EFS and FSx are legitimate services. The mistake is using "we need a shared filesystem" as a shortcut for "we do not want to change the application."

Sometimes that is a valid trade. If you are moving a WordPress fleet, a legacy CMS, a shared reports directory, or a commercial package that expects POSIX paths and file locks, EFS can save months. AWS's own EFS comparison page frames it clearly: EFS gives a traditional filesystem interface, shared access, and a hierarchical permission model. That is exactly what some applications expect.

But new cloud-native systems should be suspicious of shared mutable files. A shared filesystem creates hidden coupling. Writers assume immediate visibility. Readers assume directory listings are truth. Deployments start depending on files that were placed by another process. Permissions become application logic. Backups get murky because the filesystem is now a database that nobody calls a database.

For new systems, I prefer this split:

Put durable blobs in S3.
Put authoritative metadata in DynamoDB, Aurora, PostgreSQL, or another database that matches the query model.
Use EFS only when the application needs real file semantics.
Use FSx when the protocol or workload shape is specific enough to justify it.

Dropbox's engineering post on Object Store is a useful reminder that large systems often separate blob storage from metadata and indexing. The object store holds bytes. Another system owns object identity, placement, batching, lookup, and lifecycle. That pattern shows up everywhere because blobs and metadata have different scaling problems.

S3 plus a metadata store is not free. You have to handle two writes, retries, orphan cleanup, idempotency, and race conditions. For many systems, that is still cleaner than pretending a shared directory is a database.

Durability And Availability Are Not The Same Thing

AWS service pages usually quote durability numbers, and they matter. They just do not answer the whole question.

Durability means the data is unlikely to be lost. Availability means the system can serve reads and writes when you need it. Recovery means you can get back to a known good state after a bad deploy, bad delete, bad migration, bad customer import, or compromised credential. Those are separate properties.

S3 Standard is a strong default for durable object storage because it stores data across multiple Availability Zones and gives you lifecycle, versioning, replication, object lock, inventory, batch operations, and event integration. But S3 does not save you from deleting the wrong prefix unless you configured versioning, retention, permissions, and recovery workflows before the incident.

EBS snapshots are stored in S3 under the hood, but a running EBS volume is zonal. If the application cannot tolerate AZ loss, you need database replication, application-level failover, or a managed service that handles the problem. Taking snapshots is not the same as having a recovery point objective you can meet.

EFS stores data redundantly across multiple AZs in its regional classes, which is useful for shared access. But the application can still corrupt data instantly from every mounted client. Multi-AZ storage does not protect you from a bad process with write permission.

Multi-Region is another place where architecture gets slippery. S3 replication, EFS replication, DynamoDB global tables, Aurora Global Database, and backup copy all solve different problems. The AWS Architecture Blog's multi-Region data article notes that DynamoDB global tables support active-active multi-Region writes with conflict handling. That is not the same as saying every workload should use active-active writes. Most systems need a boring primary region, tested restore, and a clear failover plan before they need conflict resolution.

The storage choice should include the failure you are willing to rehearse. If nobody has restored it, it is not a backup. It is a hope with an ARN.

Cost Is Mostly About The Operations Around The Bytes

Storage pricing starts with GB-month, but the bill rarely ends there.

S3 looks cheap until request volume, lifecycle transitions, retrieval fees, replication, KMS calls, inventory, analytics scans, and cross-region transfer show up. For data lakes, the waste is often not storage. It is bad partitioning, tiny files, repeated full scans, and no lifecycle policy.

EBS costs more per GB than S3 because it is block storage attached to compute. The hidden cost is idle provisioned performance, oversized volumes created to chase gp2 burst behavior, and databases that keep old indexes forever. gp3 helped because you no longer need to grow capacity just to get baseline IOPS, but you still need to measure read and write size, queue depth, and instance limits.

EFS can get expensive when teams use it as a dump for large cold files. Lifecycle policies and Infrequent Access can help, but access patterns matter. A workload that constantly recalls cold data can pay more than expected. A workload with heavy metadata chatter can also surprise you.

FSx is usually a conscious performance or compatibility spend. That is fine when it replaces a real operational burden: Windows file shares with AD integration, Lustre for compute, NetApp features during migration, OpenZFS semantics for applications that need them. It is a weak choice when the team only wants "a folder everyone can mount."

DynamoDB cost is tied to reads, writes, storage, indexes, streams, global tables, and design discipline. A bad access pattern can turn into a hot partition or a new GSI that doubles write cost. A good one can run with little operational work for years.

The cost model should be attached to the access model. Write it down before picking the service:

Average object size or item size.
Reads and writes per second.
Request fan-out per user action.
Retention period.
Percentage of data read after 1 day, 30 days, and 1 year.
Required p95 and p99 latency.
Backup and restore frequency.
Cross-AZ and cross-Region traffic.
KMS request volume.
Operational owner.

If those numbers are unknown, choose the service that gives you the most room to learn without painting the application into a corner. That often means S3 for blobs, gp3 for single-host block storage, DynamoDB or Aurora for live state, and avoiding shared filesystems until the need is real.

A Practical Decision Path

I would not start with a giant matrix. I would ask these questions in order.

Is This Data Authoritative Application State?

If the data drives user-visible behavior, permissions, money, workflow state, inventory, or business rules, treat it as application state. It probably belongs in a database, not just a bucket.

Use DynamoDB when access patterns are known, key-based, high-volume, and latency-sensitive. Use Aurora or RDS when you need relational constraints, joins, transactions across flexible predicates, ad hoc reporting, or easier operational familiarity. Use OpenSearch when the primary operation is search. Use ElastiCache when the data can be rebuilt or is a derived latency layer.

S3 can still hold the large payload. The database can hold the pointer, status, checksum, owner, tenant, version, retention policy, and processing state. That split works well for uploads, documents, images, exports, model artifacts, and event payloads.

Does The Application Need Filesystem Semantics?

If it needs POSIX paths, file locks, append behavior, directory traversal, or a library that only speaks local files, use file or block storage.

Use EBS when one instance owns the disk and the workload needs low-latency block access. A database on EC2 is the common case. Use EFS when multiple Linux clients need shared NFS access across AZs. Use FSx for Windows when SMB and Windows features matter. Use FSx for Lustre when many compute nodes need high-throughput shared file access, especially when S3 is the durable backing store.

Do not mount S3 through a filesystem adapter and assume you now have a normal filesystem. That can be fine for narrow tooling and migration tasks. It is a risky primary design when the application expects rename, locking, append, or metadata behavior to work like local disk.

Is The Data Mostly Immutable?

If data is written once and read many times, S3 becomes attractive fast.

That includes media, logs, backups, exports, analytics datasets, ML training data, software artifacts, invoices, generated reports, and customer uploads. Add lifecycle rules early. Add object versioning when overwrite or delete mistakes are realistic. Add object lock only when the compliance model needs retention that even admins should not bypass.

For analytics, think about file format and layout before you think about the bucket. Parquet, partitioning, compaction, object size, and catalog design will matter more than the fact that S3 can hold an absurd amount of data.

Is The Hot Path Small And Keyed?

Use a database or cache. S3 is not the first choice for "give me the current state for user 123 on every request." DynamoDB is usually the AWS-native answer if the keys are clear and the item model is stable. Aurora or PostgreSQL is often better if the access patterns are still changing or the product needs relational queries.

The mistake is choosing DynamoDB because it scales, then discovering the product needs query flexibility that was never modeled. The opposite mistake is putting a simple key-value workload into a relational database and carrying connection pools, vacuum, index maintenance, failover tuning, and migration work for no reason.

Is The Workload Temporary And Compute-Heavy?

For scratch space, intermediate compute data, sort spill, ML preprocessing, and HPC jobs, durable storage may not be the right first concern. You may want instance store, EBS, or FSx for Lustre during the job, with S3 as the durable input and output.

This pattern keeps the permanent data cheap and durable while giving the compute job storage that matches its I/O shape. It also keeps failed jobs from becoming permanent data architecture.

The Choices I Reach For First

For a typical web application, my default is boring:

S3 for uploaded files, exports, static assets, logs, backups, and data lake inputs.
A database for authoritative state.
EBS under the database when I am running it myself, though I would rather use a managed database if possible.
EFS only for shared filesystem needs I can name.
FSx only when the protocol or performance case is specific.
ElastiCache for derived hot reads, never as the only durable copy.

For Kubernetes, I am more cautious:

EBS is fine for stateful workloads pinned through the CSI driver, but it is still AZ-bound and ReadWriteOnce for normal use.
EFS solves ReadWriteMany, but can become a shared mutable dumping ground.
S3 is usually better for application-level blobs and artifacts.
Stateful services in EKS still need backup, restore, disruption budgets, and node/AZ failure testing. A PersistentVolume is not a data strategy.

For analytics and ML:

S3 is the durable lake.
Glue Data Catalog or another metastore owns table meaning.
Athena, EMR, Redshift Spectrum, Spark, or another engine reads the data.
FSx for Lustre is worth considering when the compute pattern needs a parallel filesystem.
EBS or instance store can be useful for local shuffle, scratch, and engine internals.

For migrations:

Start by matching the old interface so the system moves safely.
Then remove storage assumptions one by one.
EFS and FSx are often bridge technologies, not always the final design.
S3 plus metadata is usually the destination for durable file-like content once the application can tolerate object semantics.

Anti-Patterns Worth Calling Out

The first is the "S3 as a filesystem" design. S3 is excellent object storage. It is not local disk. If the application treats it like local disk, test rename, locking, append, directory listing assumptions, metadata operations, and failure behavior under load before committing.

The second is "EFS as a database." If multiple writers are coordinating through files, you may have built a weak database with fewer tools. Sometimes it is acceptable for legacy software. For new software, use a real coordination store.

The third is "EBS as shared storage." EBS Multi-Attach exists for specific clustered filesystems and workloads, not as a casual way to share a disk. Most teams should not start there.

The fourth is "DynamoDB because scale." DynamoDB is excellent when the access patterns are known. It is uncomfortable when the product needs arbitrary querying, rich filtering, and relational exploration. Scaling the wrong model does not make it less wrong.

The fifth is "one bucket for everything." S3 bucket boundaries often become security, lifecycle, ownership, replication, and blast-radius boundaries. A single bucket with a pile of prefixes can work, but only if access control, inventory, lifecycle, and tenant isolation are designed intentionally.

The Short Version

Pick storage by the contract the application needs.

Use S3 when data is object-shaped, durable, cheap to keep, and usually read or processed by key or prefix. Use EBS when one compute node needs a real disk with low-latency block access. Use EFS when many Linux clients need shared filesystem semantics. Use FSx when the file protocol, enterprise feature set, or high-performance compute pattern is the point. Use DynamoDB, Aurora, RDS, OpenSearch, or ElastiCache when the data is live application state with query and latency requirements.

Most bad storage decisions come from hiding the access pattern. Someone says "files" when they mean customer documents. Someone says "objects" when they need transactions. Someone says "database" when they only need a durable blob and a status row. Someone says "shared storage" when the application has accidental coupling.

Write down the reads, writes, latency target, failure mode, recovery path, owner, and lifecycle. The AWS service usually becomes obvious after that. If it does not, build the smallest benchmark that exercises the ugly part of the workload, not the happy path.

Sources Worth Reading

Choosing an AWS storage service, AWS Decision Guide.
Choose data storage based on access patterns, AWS Well-Architected Framework.
When to choose Amazon EFS, AWS.
Amazon EBS volume types, AWS.
Amazon EFS performance specifications, AWS Docs.
Using data repositories with Amazon FSx for Lustre, AWS Docs.
Single-table vs. multi-table design in Amazon DynamoDB, AWS Database Blog.
Everything in its write place, Dropbox Engineering.