AWS DataSync & Transfer Family
Key concepts
DataSync for migration
Transfer Family protocols
Bandwidth throttling
Task scheduling
Data validation
Overview
AWS provides two managed data-movement services that solve different ingestion problems. AWS DataSync is an online transfer service that copies large volumes of file and object data between storage systems, accelerating moves from on-premises NFS, SMB, HDFS, and object stores into AWS (and between AWS storage services). AWS Transfer Family is a fully managed service that exposes standard file-transfer protocols (SFTP, FTPS, FTP, and AS2) in front of Amazon S3 or Amazon EFS, so existing partners and applications keep using their familiar clients while the data lands in AWS storage.
This topic appears on the SAA-C03 exam through "how do I get data into AWS" scenarios. DataSync is the answer when you need to migrate or periodically sync bulk file or object data efficiently and with integrity checking. Transfer Family is the answer when business partners or legacy software must keep speaking SFTP, FTPS, FTP, or AS2 but you want the data stored in S3 or EFS without running and patching your own transfer servers. You should be able to separate these two use cases quickly and recognize features like bandwidth throttling, task scheduling, and data validation.
Two Jobs: Bulk Sync and Protocol Endpoints
Use DataSync to move and synchronize large amounts of file or object data into AWS with automatic integrity validation and built-in scheduling. Use Transfer Family to put a managed SFTP, FTPS, FTP, or AS2 endpoint in front of S3 or EFS so external parties keep their existing transfer workflows while you avoid operating transfer servers.
Match the service to the trigger word. Bulk migration, periodic file or object sync, NFS or SMB or HDFS source, integrity validation: DataSync. Partners need SFTP, FTPS, FTP, or AS2 into S3 or EFS without managed servers: Transfer Family. Remember DataSync supports bandwidth throttling and scheduling, and that an on-premises DataSync agent is required for on-premises sources.
Key Concepts
DataSync for Migration
Accelerated Bulk Transfer
AWS DataSync is an online data-movement service that copies file and object data quickly and securely. For on-premises sources you deploy a DataSync agent (a VM or EC2 instance) that reads from NFS, SMB, HDFS, or self-managed object storage and sends data over an encrypted channel to AWS. A transfer is defined by a task, which pairs a source location with a destination location (for example S3, EFS, or FSx) and the options that control the copy. DataSync uses a purpose-built protocol and parallelism to run much faster than open-source tools over the same link, and it can run one time for a migration or on a schedule for ongoing sync.
Where DataSync Moves Data
DataSync handles three movement patterns: on-premises into AWS, AWS back to on-premises, and AWS service to AWS service (for example Amazon S3 to Amazon EFS, or between AWS Regions). Supported AWS destinations include Amazon S3 (all storage classes), Amazon EFS, Amazon FSx for Windows File Server, Amazon FSx for Lustre, and other FSx file systems. An agent is only needed for the on-premises or other-cloud side; transfers fully inside AWS do not require an agent.
DataSync vs Transfer Family
| Dimension | AWS DataSync | AWS Transfer Family |
|---|---|---|
| Primary job | Bulk migration and sync of files or objects | Standard protocol endpoint into AWS storage |
| Driven by | AWS-managed task and agent | External clients using SFTP, FTPS, FTP, or AS2 |
| Typical source | NFS, SMB, HDFS, object stores | Partner or application file-transfer clients |
| Destinations | S3, EFS, FSx | S3 or EFS |
| Best for | Large one-time or recurring transfers | Keeping existing transfer workflows |
Transfer Family Protocols
Managed SFTP, FTPS, FTP, and AS2
AWS Transfer Family runs fully managed file-transfer endpoints that store data directly in Amazon S3 or Amazon EFS. It supports four protocols. SFTP (SSH File Transfer Protocol) is encrypted file transfer over SSH and is the most common choice. FTPS (FTP over TLS) adds TLS encryption to classic FTP. FTP (File Transfer Protocol) is unencrypted and is intended only for internal networks. AS2 (Applicability Statement 2) is a protocol for secure, signed business-to-business message exchange, common in supply-chain and EDI workflows. You can place a Transfer Family server on a public endpoint or inside a VPC, and authenticate users with a service-managed user store, AWS Directory Service, or a custom identity provider backed by API Gateway or Lambda.
Transfer Family Protocols
| Protocol | Encryption | Typical use |
|---|---|---|
| SFTP | SSH | Default secure transfer with external partners |
| FTPS | TLS | Legacy clients that require FTP over TLS |
| FTP | None | Internal VPC-only transfers where the network is trusted |
| AS2 | Signed and encrypted messages | B2B and EDI document exchange |
Bandwidth Throttling
Protecting the Network Link
Bandwidth throttling caps the network rate a DataSync task may consume so a migration does not starve production traffic on a shared link. You set a bandwidth limit (in MiB/s or as unlimited) in the task options, and you can adjust it on a running task. This matters when a Direct Connect connection or VPN is shared between the data transfer and live applications: throttling keeps the copy from saturating the pipe during business hours. Combined with scheduling, you can run unthrottled overnight and capped during the day.
# Limit a DataSync task to about 100 MiB/s (value is in bytes per second)
aws datasync update-task \
--task-arn arn:aws:datasync:us-east-1:123456789012:task/task-0abc \
--options BytesPerSecond=104857600
# Remove the cap by setting the value to -1 (unlimited)
aws datasync update-task \
--task-arn arn:aws:datasync:us-east-1:123456789012:task/task-0abc \
--options BytesPerSecond=-1Task Scheduling
Recurring, Automated Transfers
A DataSync task schedule runs a task automatically on a recurring basis without an external trigger. Schedules are expressed with a rate or cron-style expression (for example, every hour or every day at 02:00 UTC), so you can keep a destination continuously synchronized with a changing source. Because DataSync detects and copies only changed data on each run, a scheduled task is an efficient way to maintain an ongoing replica during a phased migration or for periodic ingestion. Schedules pair naturally with bandwidth throttling to confine heavy transfers to off-peak windows.
# Run an existing task every day at 02:00 UTC using a cron schedule
aws datasync update-task \
--task-arn arn:aws:datasync:us-east-1:123456789012:task/task-0abc \
--schedule ScheduleExpression="cron(0 2 * * ? *)"
# Or run it once per hour with a rate expression
aws datasync update-task \
--task-arn arn:aws:datasync:us-east-1:123456789012:task/task-0abc \
--schedule ScheduleExpression="rate(1 hour)"Data Validation
Verifying Integrity After Transfer
Data validation confirms that what arrived at the destination matches the source. DataSync always runs checksum integrity checks during the transfer itself, and at the end of a transfer it offers three verification modes: verify only the data it transferred in the current run (ONLY_FILES_TRANSFERRED, the recommended option), verify the entire destination against the source (POINT_IN_TIME_CONSISTENT), or skip the end-of-transfer check (NONE). The recommended option, verifying transferred data, catches corruption introduced in transit while keeping verification scoped to what changed. Verification adds time and resource use, so for very large datasets you weigh the verification mode against the migration window. This automatic integrity checking is a key reason to choose DataSync for migrations over hand-rolled copy scripts.
DataSync Verification Modes
| Mode | What it checks | Trade-off |
|---|---|---|
| Verify transferred data (recommended) | Checksums of files copied in this run | Balanced integrity and speed |
| Verify all data | Entire destination against the source | Highest assurance, slowest |
| No verification | No end-of-transfer check | Fastest, no post-transfer guarantee |
Best Practices
1. Pick the service from the trigger
├── Bulk file or object migration and sync: DataSync
└── Partners need SFTP, FTPS, FTP, or AS2 into S3 or EFS: Transfer Family
2. Protect shared links during business hours
├── Apply bandwidth throttling on DataSync tasks
└── Schedule heavy transfers for off-peak windows
3. Keep destinations current with schedules
└── Use rate or cron schedules; DataSync copies only changed data
4. Keep data validation on for migrations
└── Recommended transferred-data verification catches in-transit corruption
5. Secure Transfer Family by protocol and placement
├── Prefer SFTP or FTPS; restrict plain FTP to VPC-internal use
└── Use a VPC endpoint and custom identity provider for tighter controlCommon Pitfalls
Pitfall 1: Using Transfer Family for a Bulk Migration
Mistake: Choosing AWS Transfer Family to migrate terabytes of files from an on-premises NFS share into S3.
Why it fails: Transfer Family exposes protocol endpoints for clients to push and pull individual files; it is not built to drive an accelerated, validated bulk copy with scheduling.
Correct Approach: Use AWS DataSync with an on-premises agent for high-throughput migration with automatic integrity validation.
Pitfall 2: Forgetting the On-Premises DataSync Agent
Mistake: Designing a DataSync transfer from an on-premises NFS or SMB share without deploying the DataSync agent.
Why it fails: On-premises and other-cloud sources require an agent VM (or EC2 instance) to read the source and connect to AWS; without it the task cannot reach the data.
Correct Approach: Deploy the DataSync agent on premises (transfers wholly inside AWS, such as S3 to EFS, need no agent).
Pitfall 3: Saturating a Shared Link with No Throttle
Mistake: Running a large DataSync task at full speed over a Direct Connect or VPN link shared with production traffic.
Why it fails: The transfer can consume all available bandwidth and degrade live application performance.
Correct Approach: Set a bandwidth limit on the task and schedule heavy transfers for off-peak hours.
Pitfall 4: Exposing Plain FTP to the Internet
Mistake: Standing up a public Transfer Family endpoint using FTP for external partners.
Why it fails: FTP transmits credentials and data without encryption, exposing them on the public internet.
Correct Approach: Use SFTP or FTPS for external transfers, and keep plain FTP confined to trusted VPC-internal networks.
Test Your Knowledge
A company must migrate 40 TB of files from an on-premises NFS file server into Amazon S3 and then keep S3 synchronized nightly until cutover, with verification that copied data is intact. Which approach fits best?
External trading partners exchange documents with a company using SFTP clients today. The company wants to keep partner workflows unchanged while storing the files in Amazon S3 and avoid running file-transfer servers. Which service should the architect choose?
A DataSync migration shares a Direct Connect link with production applications. Daytime transfers slow the applications, but the migration can run freely overnight. What should the architect configure?
Related Services
Quick Reference
Service Facts
Key Facts and Limits
| Topic | Detail |
|---|---|
| DataSync source types | NFS, SMB, HDFS, self-managed object storage, AWS storage |
| DataSync AWS destinations | S3, EFS, FSx for Windows, FSx for Lustre, other FSx |
| DataSync agent | Required for on-premises or other-cloud endpoints |
| DataSync verification (recommended) | Verify data transferred in the current run |
| DataSync scheduling | Rate or cron expressions on a task |
| Transfer Family protocols | SFTP, FTPS, FTP, AS2 |
| Transfer Family storage backends | Amazon S3 and Amazon EFS |
| Transfer Family endpoint types | Public or VPC-hosted |
Common CLI Commands
# Create a DataSync task between two existing locations
aws datasync create-task \
--source-location-arn arn:aws:datasync:us-east-1:123456789012:location/loc-src \
--destination-location-arn arn:aws:datasync:us-east-1:123456789012:location/loc-dst \
--name nfs-to-s3
# Start a one-time execution of the task
aws datasync start-task-execution \
--task-arn arn:aws:datasync:us-east-1:123456789012:task/task-0abc
# Create a managed SFTP server backed by S3
aws transfer create-server \
--protocols SFTP \
--endpoint-type PUBLIC \
--identity-provider-type SERVICE_MANAGED
# Add a user mapped to an S3 home directory
aws transfer create-user \
--server-id s-1234567890abcdef0 \
--user-name partner1 \
--role arn:aws:iam::123456789012:role/transferS3AccessRole \
--home-directory /my-bucket/partner1