AWS for Data Work

Most AWS lists try to cover everything and help with nothing. This post does the opposite. It covers five services, S3, Glue, Athena, Redshift and CloudWatch, and shows how they work together on one common job: working with data.

That pattern shows up in many places. Sales files, order data, website events, inventory lists or survey results all follow the same shape. Files arrive, they are messy, they need cleaning and checking, someone needs to query them today, and the same reports need to run again tomorrow, on time, with proof that they ran.

If your work looks like that, these five services are enough to build the whole pipeline.

The data job this is about

Picture one daily cycle:

1. Data files land from a source system, late, in mixed formats.
2. The files are cleaned, standardised and checked for missing fields.
3. Analysts query the clean data to answer questions and spot errors.
4. A fixed set of reports runs for the team.
5. Someone has to know, quickly, if any step failed.

Each service below owns one part of that cycle. None of them does the whole job alone, and that is the point.

How the five services fit together

The flow is simple: S3 first, Glue second, then Athena for one-off questions and Redshift for repeated reporting, with CloudWatch watching all of it.

S3 is where every file lands first

Amazon Simple Storage Service, or S3, is object storage. For data work, treat it as your landing zone and your archive, not as a dumping ground.

Use three prefixes, and keep them separate:

data-bucket/
  raw/        files exactly as they arrived, never edited
  clean/      cleaned data, partitioned by date
  output/     finished reports, ready to share

Three rules make this work:

• Never edit the raw files. If cleaning goes wrong, you need the original to run it again.
• Partition clean data by date, for example clean/year=2026/month=11/day=08/. Athena and Glue both use those partitions to read only the data a query needs.
• Convert clean data to a columnar format such as Parquet. It is smaller in S3 and much cheaper to query than raw CSV, because Athena charges by the amount of data it scans.

Put a lifecycle rule on the raw prefix if your retention needs allow it, so old raw files move to cheaper storage instead of sitting at full price forever. Decide how long to keep files first, and set the rule to match, not the other way round.

Glue does the cleaning and keeps the catalogue honest

AWS Glue has three parts, and data teams use all three.

• Crawlers scan files in S3 and work out their schema: column names, types and partitions.
• Data Catalog stores that schema. It is the shared table definition that Athena and Redshift Spectrum can both read.
• ETL jobs do the actual cleaning in Python or Scala, using Apache Spark under the hood.

A typical Glue job for a data feed does unglamorous, essential work:

• Standardise date and amount formats from different source systems.
• Reject or flag rows with missing required fields, instead of passing them through silently.
• Write the good rows to the clean/ prefix as Parquet, partitioned by date.
• Write the rejected rows to a separate prefix, with a reason, so they can be counted and fixed.

Schedule the crawler after the ETL job, or define the target table yourself and update partitions in the job. If the catalogue and the files drift apart, every query after that is wrong in a way that is hard to see.

Athena is for questions you ask once

Amazon Athena runs SQL directly over data in S3, using the tables in the Glue Data Catalog. There is no server to start, patch or size. You write a query, it scans the files, you get a result.

That makes Athena the right tool for the questions data work throws up every day:

• Did today's file arrive, and how many rows does it have?
• How many records were rejected, and for what reason?
• Does this total match what the source system said it sent?

SELECT reject_reason, COUNT(*) AS rejected
FROM clean.orders
WHERE year = '2026' AND month = '11' AND day = '08'
GROUP BY reject_reason
ORDER BY rejected DESC;

Two habits keep Athena useful and cheap:

• Always filter on the partition columns, here year, month and day. Without that filter, Athena scans the whole table.
• Save the query when it turns out to be useful. A question asked three times is the start of a report, and repeated reports belong in Redshift, not in a pile of saved Athena queries.

Athena is not a warehouse. It is slow for complex joins over huge data, and every user running full scans at once will feel it. Use it to explore and to check, not to serve the whole business.

Redshift is for questions you ask every day

Amazon Redshift is a data warehouse. You load clean data into it, or query S3 data through Redshift Spectrum, and it serves fast SQL to reporting tools and dashboards.

Move a report to Redshift when:

• The same query runs daily or hourly, for many users.
• The report has a deadline, and "Athena was slow today" is not an acceptable answer.
• The joins and aggregations are heavy, and they are the same every time.

A sensible split for a data team:

Load Redshift from the clean/ prefix in S3, on a schedule, after the Glue job finishes. Do not load it from raw files. If Redshift and Athena read the same clean data and the same catalogue, their numbers should agree, and when they do not, you know the load is the problem.

CloudWatch is how you know it failed before the business does

Amazon CloudWatch collects logs, metrics and alarms from the other four services. In a data pipeline, its job is simple: a silent failure is the worst failure.

Set up three things:

• Logs: Send Glue job logs and your loading scripts to CloudWatch Logs, with the date and file name in each run's log lines. When a result is questioned weeks later, the log is your evidence of what ran.
• Metrics: Watch Glue job failures, Athena query failures and data scanned, and Redshift disk use and query queue time.
• Alarms: Raise an alarm when a Glue job fails, when the daily Redshift load does not finish by its deadline, or when rejected rows jump well above the usual level.

An alarm that only says "something failed" is not enough. Name the job, the date and the step in the alarm, so the person reading it at 7 AM knows where to look first.

Work through a failure in this order: CloudWatch log for the failed step, rejected rows in S3, source file in the raw/ prefix, then a checking query in Athena. Fix the cause, run that step again from the raw file, and check the totals before the report goes out.

A day in this pipeline

• Early morning: The source files land in raw/ in S3. A check confirms the expected files arrived.
• Cleaning: A Glue ETL job standardises the files, writes clean Parquet to clean/, and writes rejected rows out with reasons. The catalogue is updated.
• Checking: Athena queries count rows in, rows out and rows rejected. The three numbers must add up.
• Reporting: Clean data loads into Redshift, and the daily reports and dashboards run from there.
• Watching: CloudWatch logs every step. An alarm fires if a job fails or a deadline is missed, not when a user notices a missing report.

Nothing in that day needs a server you manage. The work that remains is the work that matters: field rules, checking logic and report definitions.

What each service costs you if you leave it on

You pay in different ways, so the traps are different.

• S3: Cheap per file, expensive as an archive of everything forever. Use lifecycle rules and delete test data.
• Glue: You pay for crawlers and ETL jobs while they run. A crawler set to run every hour over a huge bucket is a quiet, steady cost.
• Athena: You pay for data scanned. Unpartitioned CSV tables are the classic way to turn a small query into a large bill.
• Redshift: You pay for the cluster while it exists. Size it for reporting load, and pause or scale it down outside reporting hours if your pattern allows it.
• CloudWatch: Logs add up. Set a retention period on log groups instead of keeping every log forever by default.

Checklist

• Land files in S3 raw/, and never edit them there.
• Clean with Glue, write Parquet to clean/, partitioned by date.
• Keep the Glue Data Catalog in step with the files.
• Use Athena for checks and one-off questions, always filtered by partition.
• Use Redshift for the reports that run every day, loaded from clean data only.
• Alarm in CloudWatch on failure and on missed deadlines, not just on errors.
• Check that rows in, rows out and rows rejected add up before any report leaves.

Start with S3, Glue and Athena. That trio alone handles landing, cleaning and querying. Add Redshift when the same reports start running every day, and add CloudWatch alarms from day one, because a data pipeline nobody watches will fail quietly.