Data Lakes & Analytics
Key concepts
Lake Formation
S3 as data lake storage
Data catalog
QuickSight for visualization
Analytics architecture
Overview
A data lake is a centralized repository that stores structured, semi-structured, and unstructured data at any scale in its raw form, so you can run many kinds of analytics on the same data. On AWS, the foundation is Amazon S3 (Simple Storage Service), which provides durable and inexpensive object storage. Around that core you add a data catalog to track what data exists and where, AWS Lake Formation to set up and secure the lake, query and processing engines such as Amazon Athena and AWS Glue, and Amazon QuickSight to turn results into dashboards. Together these services form a serverless analytics architecture that decouples storage from compute.
This topic is low-weight on the SAA-C03 exam, but it appears in scenario questions about choosing the right storage for analytics, securing access to lake data, and assembling a serverless reporting pipeline. You should recognize S3 as the storage layer, understand what Lake Formation and the AWS Glue Data Catalog do, know where Athena and Glue fit, and know that QuickSight is the managed business intelligence service for visualization.
S3 Is the Lake, Everything Else Plugs In
Store raw data once in S3, register and govern it with Lake Formation and the Glue Data Catalog, query it in place with Athena, transform it with Glue, and visualize it with QuickSight. Storage and compute scale independently, and you pay only for what you query and process.
When a scenario asks for cheap, durable analytics storage, the answer is S3. When it asks to query S3 data with standard SQL and no servers, the answer is Athena. When it asks for centralized, fine-grained permissions across the lake, the answer is Lake Formation. When it asks for dashboards, the answer is QuickSight.
Key Concepts
S3 as Data Lake Storage
The Storage Foundation
Amazon S3 is the standard storage layer for an AWS data lake because it offers eleven nines (99.999999999%) of durability, virtually unlimited capacity, and low per-gigabyte cost. You decouple storage from compute: data sits in S3 while separate engines (Athena, Glue, Amazon EMR, Amazon Redshift Spectrum) read it on demand. A common layout uses prefixes for raw, cleaned, and curated zones, partitions data by date or region for faster queries, and uses columnar formats such as Apache Parquet to reduce scan cost. S3 lifecycle policies automatically move older data to cheaper tiers such as S3 Glacier.
Data Lake vs Data Warehouse
| Dimension | Data Lake (S3) | Data Warehouse (Redshift) |
|---|---|---|
| Data shape | Raw, any format | Structured, modeled |
| Schema | Schema on read | Schema on write |
| Cost model | Cheap object storage | Provisioned or serverless compute |
| Typical use | Store everything, explore later | Fast repeated BI queries |
| Best for | Diverse data and ad hoc analytics | Curated reporting at scale |
Data Catalog
Knowing What Data You Have
A data catalog is a metadata store that records the databases, tables, schemas, and S3 locations of your data so query engines know how to read it. On AWS this is the AWS Glue Data Catalog, a single Apache Hive compatible metastore shared across Athena, Glue, EMR, and Redshift Spectrum. A Glue crawler scans S3 paths, infers schema and partitions, and creates or updates catalog tables automatically. Without a catalog entry, an engine like Athena has no table definition to query against.
AWS Lake Formation
Setting Up and Governing the Lake
AWS Lake Formation is a managed service that simplifies building, securing, and governing a data lake on S3. It registers S3 locations as lake storage, helps load and catalog data, and adds a centralized permissions layer on top of the Glue Data Catalog. Lake Formation grants fine-grained access at the database, table, column, row, and cell level using simple grant statements, instead of writing many S3 bucket policies and IAM (Identity and Access Management) policies by hand. It also supports tag-based access control (LF-Tags) to manage permissions at scale and integrates with AWS Identity Center for user federation.
Access Control: IAM/S3 vs Lake Formation
| Dimension | IAM and S3 Policies | Lake Formation |
|---|---|---|
| Granularity | Bucket, prefix, object | Database, table, column, row, cell |
| Management | Many policies to maintain | Central grants on catalog resources |
| Model | Resource permissions | Database-style grant and revoke |
| Best for | Coarse storage access | Fine-grained analytics governance |
QuickSight for Visualization
Dashboards and Business Intelligence
Amazon QuickSight is a fully managed, serverless business intelligence service that builds interactive dashboards and reports. It connects to many sources including Athena, S3, Amazon RDS, and Amazon Redshift. Its in-memory engine, SPICE (Super-fast Parallel In-memory Calculation Engine), caches imported data so dashboards load quickly without re-querying the source each time. QuickSight uses per-session or per-user pricing, supports embedding dashboards into applications, and includes Amazon Q natural-language querying for asking questions in plain English.
Analytics Architecture
A Serverless Analytics Pipeline
A typical serverless analytics flow ingests data into S3 (for example through Amazon Kinesis Data Firehose or AWS Database Migration Service), catalogs it with a Glue crawler into the Glue Data Catalog, governs access with Lake Formation, transforms or enriches it with AWS Glue ETL (Extract, Transform, Load) jobs, queries it in place with Athena using standard SQL, and visualizes the results in QuickSight. Because each layer is serverless, you scale storage and compute independently and pay per use, with no clusters to manage.
Analytics Layers and Services
| Layer | Service | Role |
|---|---|---|
| Ingest | Kinesis Data Firehose, DMS | Land raw data into S3 |
| Store | Amazon S3 | Durable lake storage |
| Catalog | Glue Data Catalog and crawlers | Track schema and locations |
| Govern | Lake Formation | Fine-grained permissions |
| Transform | AWS Glue ETL | Clean and reshape data |
| Query | Amazon Athena | Serverless SQL on S3 |
| Visualize | Amazon QuickSight | Dashboards and reports |
Best Practices
1. Use S3 as the single source of truth
├── Organize raw, cleaned, and curated zones by prefix
└── Apply lifecycle policies to tier old data to Glacier
2. Store data in columnar, partitioned formats
├── Use Parquet or ORC to cut Athena scan cost
└── Partition by date or region to scan less data
3. Catalog with Glue crawlers
└── Let crawlers infer schema and keep partitions current
4. Govern centrally with Lake Formation
├── Grant column, row, and cell-level access
└── Use LF-Tags to manage permissions at scale
5. Query in place with Athena before moving data
└── Reserve Redshift for repeated high-concurrency BI
6. Cache dashboards in QuickSight SPICE
└── Refresh SPICE on a schedule to balance cost and freshnessCommon Pitfalls
Pitfall 1: Treating a Data Lake Like a Data Warehouse
Mistake: Loading everything into Amazon Redshift first to run ad hoc exploration on raw data.
Why it fails: Provisioned warehouse storage and compute are more expensive than S3 for raw data that is queried occasionally, and rigid schemas slow down exploration.
Correct Approach: Land raw data in S3, query it with Athena for exploration, and promote only curated, frequently queried data into Redshift.
Pitfall 2: Querying Without a Catalog
Mistake: Expecting Athena to query S3 objects directly with no table definition.
Why it fails: Athena needs a table in the Glue Data Catalog that describes the schema, location, and format before it can read the data.
Correct Approach: Run a Glue crawler or create the table with DDL so the Glue Data Catalog has the metadata Athena requires.
Pitfall 3: Securing the Lake With Bucket Policies Alone
Mistake: Trying to enforce column-level or row-level access using only S3 bucket policies and IAM.
Why it fails: S3 policies grant access at the object or prefix level and cannot restrict individual columns, rows, or cells within a table.
Correct Approach: Use Lake Formation to grant fine-grained, catalog-aware permissions across the analytics services that read the lake.
Pitfall 4: Scanning Uncompressed Data With Athena
Mistake: Storing data as raw CSV or JSON and querying it repeatedly with Athena.
Why it fails: Athena charges per terabyte scanned, so reading large uncompressed row-based files makes every query slow and costly.
Correct Approach: Convert data to compressed columnar formats such as Parquet and partition it so Athena scans only the relevant data.
Test Your Knowledge
A company wants to store petabytes of raw clickstream, log, and image data cheaply and run several different analytics tools against it later. Which service is the foundation of this data lake?
A security team must give analysts access to a sales table but hide the customer email and credit-limit columns, while controlling this centrally across Athena and Redshift Spectrum. What should the architect use?
An analyst needs to run standard SQL queries directly against JSON and Parquet files in S3 without managing any servers, then build a dashboard on the results. Which pair of services fits best?
Related Services
Quick Reference
Service Roles at a Glance
Quick Decisions
| If you need... | Choose |
|---|---|
| Cheap durable storage for any data | Amazon S3 |
| Centralized fine-grained lake permissions | Lake Formation |
| A shared metadata catalog | Glue Data Catalog |
| Schema discovery from S3 | Glue crawler |
| Serverless SQL on S3 data | Amazon Athena |
| Serverless ETL transformations | AWS Glue |
| Dashboards and reports | Amazon QuickSight |
Common CLI Commands
# Register an S3 location as Lake Formation managed storage
aws lakeformation register-resource \
--resource-arn arn:aws:s3:::my-data-lake \
--use-service-linked-role
# Create a Glue crawler that catalogs an S3 path
aws glue create-crawler \
--name sales-crawler \
--role AWSGlueServiceRole \
--database-name sales_db \
--targets '{"S3Targets":[{"Path":"s3://my-data-lake/sales/"}]}'
# Start the crawler to populate the Glue Data Catalog
aws glue start-crawler --name sales-crawler
# Grant column-level access with Lake Formation
aws lakeformation grant-permissions \
--principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/Analyst \
--resource '{"TableWithColumns":{"DatabaseName":"sales_db","Name":"orders","ColumnNames":["order_id","amount"]}}' \
--permissions SELECT
# Run a query in Athena
aws athena start-query-execution \
--query-string "SELECT region, SUM(amount) FROM sales_db.orders GROUP BY region" \
--result-configuration OutputLocation=s3://my-athena-results/