AWS Glue (ETL)
- 01
Glue Data Catalog
- 02
Glue crawlers
- 03
ETL jobs
- 04
Glue Studio
- 05
Schema discovery
Overview
AWS Glue is a serverless data integration service that discovers, catalogs, cleans, and transforms data so it can be analyzed. ETL stands for Extract, Transform, Load, which means reading data from a source, reshaping it, and writing it to a destination. Glue runs the underlying Apache Spark (a distributed data processing engine) for you, so there are no clusters or servers to provision. Its core pieces are the Glue Data Catalog (a central metadata store), Glue crawlers (jobs that scan data and infer its schema), Glue ETL jobs (the transformation scripts), and Glue Studio (a visual authoring tool).
This topic appears on the SAA-C03 exam in "how do I prepare and query data without managing servers" scenarios. You should recognize that Glue is serverless and pay-per-use, that the Data Catalog acts as a shared metadata layer for Athena, Amazon Redshift Spectrum, and Amazon EMR, that crawlers automate schema discovery, and that Glue handles batch ETL while real-time streaming aggregation is Kinesis territory.
Serverless Catalog Plus Serverless ETL
Glue gives you a managed metadata catalog and serverless Spark transformations in one service. Crawlers populate the Data Catalog automatically, ETL jobs reshape the data, and downstream services such as Athena query that same catalog without you running any infrastructure.
On the exam, map keywords to Glue: serverless ETL, central data catalog, schema discovery, and prepare data in S3 for Athena. Glue is batch-oriented, the Data Catalog is shared across Athena, Redshift Spectrum, and EMR, and crawlers infer schema and partitions automatically. If a question stresses real-time streaming, look at Kinesis instead.
Key Concepts
Glue Data Catalog
The Central Metadata Store
The Glue Data Catalog is a persistent, managed metadata store that holds table definitions: column names, data types, partitions, and the location of the underlying files (for example an Amazon S3 path). It does not store the data itself, only the schema and pointers to it. The catalog is organized into databases (logical groupings) and tables. It is region-scoped and acts as the single source of truth for Athena, Redshift Spectrum, and Amazon EMR, so one crawler run can make data queryable across all of them. The catalog is also a drop-in replacement for the Apache Hive Metastore.
Glue Crawlers
Automated Schema Discovery
A Glue crawler connects to a data store (such as S3, Amazon RDS, or Amazon DynamoDB), scans a sample of the data, infers the schema using built-in or custom classifiers (rules that recognize formats like JSON, CSV, Parquet, or Avro), and writes or updates the resulting table definitions in the Data Catalog. Crawlers also detect partitions (for example, year and month folders in S3) and can run on demand or on a schedule. When new files arrive or columns change, a re-run keeps the catalog current. This is how schema discovery is automated rather than hand-typed.
ETL Jobs
The Transformation Engine
A Glue ETL job is the script that does the actual extract, transform, and load work. Jobs run on serverless Apache Spark and are measured in DPUs (Data Processing Units, a bundle of compute and memory) billed per second. Glue introduces the DynamicFrame, a Spark abstraction that tolerates messy, evolving schemas better than a plain DataFrame. Jobs can be authored in Python (PySpark) or Scala, support job bookmarks (state that tracks already-processed data so re-runs skip it), and can be triggered on a schedule, on demand, or by events through Glue triggers and workflows.
import sys
from awsglue.context import GlueContext
from awsglue.transforms import ApplyMapping
from pyspark.context import SparkContext
glueContext = GlueContext(SparkContext.getOrCreate())
# Read from a Data Catalog table populated by a crawler
source = glueContext.create_dynamic_frame.from_catalog(
database="sales_raw",
table_name="orders",
transformation_ctx="source" # enables job bookmarks
)
# Transform: rename and cast columns
mapped = ApplyMapping.apply(frame=source, mappings=[
("order_id", "string", "id", "string"),
("amount", "string", "amount", "double")
])
# Load: write partitioned Parquet back to S3
glueContext.write_dynamic_frame.from_options(
frame=mapped,
connection_type="s3",
connection_options={"path": "s3://my-bucket/curated/orders/", "partitionKeys": ["dt"]},
format="parquet"
)Glue Studio
Visual Job Authoring
Glue Studio is a visual, drag-and-drop interface for building ETL jobs as a graph of nodes (sources, transforms, and targets) without writing Spark code by hand. It generates the underlying PySpark or Scala script, which you can edit directly, and includes a job-run dashboard for monitoring. It lowers the skill bar for teams that want ETL without deep Spark expertise. AWS Glue DataBrew is a related no-code tool aimed at data analysts for visual data cleaning and profiling.
Schema Discovery
From Raw Files to Queryable Tables
Schema discovery is the process of turning raw, untyped files into structured table definitions. A crawler reads the data, a classifier identifies the format and column types, and the inferred schema is stored in the Data Catalog. Glue supports schema evolution, so when columns are added or types change, a re-crawl updates the table version while keeping history. Once discovered, the schema is immediately usable by Athena and other engines, which removes the need to define tables manually.
| Component | Purpose | Output |
|---|---|---|
| Data Catalog | Store metadata centrally | Databases and tables |
| Crawler | Discover schema and partitions | Catalog table definitions |
| ETL job | Extract, transform, load data | Curated data in S3 or a database |
| Glue Studio | Author jobs visually | Generated PySpark or Scala script |
| Need | Use Glue | Use Something Else |
|---|---|---|
| Serverless batch ETL | Yes, Glue ETL jobs | EMR for large custom Spark or Hadoop |
| Central metadata catalog | Yes, Data Catalog | Self-managed Hive Metastore |
| Query S3 with SQL | Catalog feeds Athena | Athena runs the queries |
| Real-time stream processing | No | Kinesis Data Analytics or Managed Flink |
Best Practices
1. Let crawlers maintain the catalog
├── Schedule crawlers so new partitions are picked up automatically
└── Use one catalog as the shared metadata layer for Athena and Redshift Spectrum
2. Write curated output in columnar Parquet
├── Compress and partition by query keys (for example date)
└── Smaller scans lower Athena and downstream costs
3. Enable job bookmarks for incremental loads
└── Re-runs process only new data and skip what was already handled
4. Right-size DPUs and use Flex execution for non-urgent jobs
└── Flex uses spare capacity at lower cost for jobs that tolerate variable start times
5. Secure data and metadata
├── Encrypt catalog and job output with AWS KMS
└── Control access with IAM and Lake Formation permissionsCommon Pitfalls
Pitfall 1: Reaching for Glue to Do Real-Time Streaming
Mistake: Choosing Glue ETL jobs to aggregate a high-velocity event stream with sub-second latency.
Why it fails: Glue is batch-oriented and starts Spark jobs that take time to spin up, so it cannot deliver true real-time stream aggregation.
Correct Approach: Use Amazon Kinesis Data Analytics or Amazon Managed Service for Apache Flink for streaming, and reserve Glue for batch transformation and cataloging.
Pitfall 2: Defining Tables by Hand Instead of Crawling
Mistake: Manually creating and updating Data Catalog tables every time new files or partitions land in S3.
Why it fails: Manual definitions drift out of sync, miss new partitions, and break queries when the schema evolves.
Correct Approach: Run a scheduled crawler to automate schema discovery and partition detection so the catalog stays current.
Pitfall 3: Skipping Job Bookmarks on Recurring Jobs
Mistake: Running the same ETL job on a schedule without enabling job bookmarks.
Why it fails: Each run reprocesses the entire dataset, which wastes DPU time, raises cost, and can create duplicate output.
Correct Approach: Enable job bookmarks and set a transformation context so each run handles only newly arrived data.
Pitfall 4: Treating the Catalog as Data Storage
Mistake: Assuming the Data Catalog holds the actual records.
Why it fails: The catalog stores only metadata (schema and file locations); the data still lives in S3 or another store, so deleting source files breaks queries even though the table definition remains.
Correct Approach: Manage the underlying data lifecycle (for example S3 lifecycle policies) separately from the catalog metadata.
Test Your Knowledge
A company stores raw CSV and JSON log files in Amazon S3 and wants analysts to query them with SQL through Athena, with table definitions kept current as new daily partitions arrive. What is the most operationally efficient approach?
A team needs to transform several terabytes of nightly batch data in S3 into partitioned Parquet without provisioning or managing any Spark clusters. Which service fits best?
An organization wants its data engineers and less Spark-experienced analysts to build the same ETL pipelines, with metadata reused by Athena, Redshift Spectrum, and EMR. Which combination of Glue features supports this?
Related Services
Quick Reference
Decision Summary
| If you need... | Choose |
|---|---|
| A central, shared metadata store | Glue Data Catalog |
| Automated schema and partition discovery | Glue crawler |
| Serverless batch ETL on Spark | Glue ETL job |
| Visual no-Spark job authoring | Glue Studio |
| Incremental processing of new data only | Job bookmarks |
| Real-time stream processing | Kinesis or Managed Flink |
Common CLI Commands
# Create a Data Catalog database
aws glue create-database --database-input '{"Name":"sales_raw"}'
# Create a crawler that scans an S3 path into the catalog
aws glue create-crawler \
--name orders-crawler \
--role AWSGlueServiceRole-orders \
--database-name sales_raw \
--targets '{"S3Targets":[{"Path":"s3://my-bucket/raw/orders/"}]}'
# Run the crawler to discover schema and partitions
aws glue start-crawler --name orders-crawler
# Start an ETL job run
aws glue start-job-run --job-name orders-etl
# List tables the crawler created
aws glue get-tables --database-name sales_raw