AWS Patterns

https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-migration/patterns.html

AWS Prescriptive Guidance patterns provide tools, architectures, and step-by-step guidance for implementing the methodologies for the migration strategy. You'll also find patterns that support other initiatives, such as cloud-native or security projects. You can view all patterns on the AWS Prescriptive Guidance website, use the filtering options, or type in a search query to find patterns of interest.

The workflow illustrated in the diagram consists of these high-level steps(Use Case):

The user uploads a CSV file into the source folder in Amazon S3.

An Amazon S3 notification event initiates an AWS Lambda function that starts the Step Functions state machine.

The Lambda function validates the schema and data type of the raw CSV file.

Depending on the validation results:

If validation of the source file succeeds, the file moves to the stage folder for further processing.

If validation fails, the file moves to the error folder, and an error notification is sent through Amazon Simple Notification Service (Amazon SNS).

An AWS Glue crawler creates the schema of the raw file from the stage folder in Amazon S3.

An AWS Glue job transforms, compresses, and partitions the raw file into Parquet format.

The AWS Glue job also moves the file to the transform folder in Amazon S3.

The AWS Glue crawler creates the schema from the transformed file. The resulting schema can be used by any analytics job. You can also use Amazon Athena to run ad-hoc queries.

If the pipeline completes without errors, the schema file is moved to the archive folder. If any errors are encountered, the file is moved to the error folder instead.

Amazon SNS sends a notification that indicates success or failure based on the pipeline completion status.

All the AWS resources used in this pattern are serverless. There are no servers to manage.