Data Ingestion and Transformation at Scale

Streaming ingestion splits into two AWS services with different jobs. Amazon Kinesis Data Streams is the raw, real-time pipe: producers write records into shards, and you write custom consumer applications (on EC2, Lambda, or Kinesis Data Analytics) that read and process them, with records staying available for a retention period — 24 hours by default, extendable up to 365 days — so more than one consumer can replay the same data independently. Amazon Data Firehose takes the opposite tradeoff: it's a fully managed delivery pipe with no persistent storage of its own, automatically scaling to load streaming data directly into a destination like Amazon S3, Amazon Redshift, or OpenSearch, in near real time — and it can convert incoming JSON records into Parquet or ORC on the way in, using the schema from an AWS Glue Data Catalog table.

Reach for Data Streams when you need custom, low-latency processing logic and possibly more than one independent consumer of the same data. Reach for Firehose when the job is "get this stream of records into a destination, reliably and with minimal setup."

aws kinesis create-stream \
  --stream-name clickstream \
  --stream-mode-details StreamMode=ON_DEMAND

Once raw data lands in S3, three services turn it into something queryable. AWS Glue is serverless ETL: a crawler connects to a data source, infers its schema, and populates the Glue Data Catalog — a central metadata store of databases and tables that other services read from. A Glue job then runs the actual transformation (commonly on Apache Spark) — for example, converting a folder of CSV files into compressed, columnar Parquet, which is both cheaper to store and faster to query.

Amazon Athena is a serverless SQL query engine that reads directly from S3, using table definitions from the Glue Data Catalog, and bills per query rather than per running server — there's no cluster to provision or manage, which makes it the natural choice for ad hoc analysis over data already sitting in a data lake.

AWS Lake Formation sits above both: it centralizes fine-grained permissions (down to the table, column, or row) across everyone and everything that touches the data lake — Athena queries, Glue jobs, and more — instead of managing a separate access policy per service.

Getting data into that lake in the first place is a separate question from streaming: sometimes it's a one-time or scheduled bulk transfer, not a continuous stream. AWS DataSync automates exactly that: an online transfer service that moves large datasets between on-premises NFS or SMB storage and Amazon S3, EFS, or FSx (or between AWS storage services), with built-in scheduling, retries, and data-integrity verification.

AWS Storage Gateway solves a different problem: ongoing, hybrid access rather than a one-time move. A File Gateway lets on-premises applications read and write S3 objects using NFS or SMB; a Volume Gateway presents cloud-backed iSCSI block storage; a Tape Gateway presents a virtual tape library for backup software that still expects one. The distinction to hold onto: DataSync moves data; Storage Gateway gives ongoing local access to data that lives in AWS.