AWS and a proposed financial crime cloud architecture
Explain where data lives, where it is processed, who can access it and how an output is traced back to a source. This diagram is a learning proposal for discussion, not any organisation's actual architecture.
01Proposed topology
Follow the flow from source to consumer. The governance plane (identity, catalogue and audit) spans every layer rather than being one box at the end.
- 01Source systems send data through controlled ingestion, which records source, event identifier, schema version, extraction time and load batch.
- 02The original payload lands in restricted raw storage (S3); invalid records go to quarantine with a reason and an owner.
- 03Validation and transformation check required fields, key scope and effective time before records enter the canonical model.
- 04Curated history retains the relevant business versions in Snowflake.
- 05Governed views and data products expose only the fields and populations authorised for each consuming team.
- 06Approved analytics and financial crime consumers receive complete lineage back to the source event.
02Follow one record
A source sends customer-risk event E123. Ingestion records the source, event identifier, schema version, extraction time and load batch. The original payload lands in restricted raw storage. Validation checks required fields, key scope and effective time. Valid records enter the canonical model; invalid ones go to quarantine with a reason and owner. Curated history retains the relevant business versions. A view exposes only the fields and population authorised for the consuming team. The output retains enough lineage to trace E123 and the transformation version.
“Raw retained” does not mean retained forever. Retention, legal holds, deletion and access must follow the approved policy.
03Service vocabulary
Service selection is a design decision. An AWS service is not mandatory simply because it appears here. Snowflake has its own permissions; a Lake Formation rule does not automatically govern every copy loaded into Snowflake. Map and test each access boundary separately.
| Service or concept | Purpose in this proposed design |
|---|---|
| Amazon S3 | Object storage for landing files and approved retained source data |
| AWS Glue / EMR | Candidate transformation engines; choose based on existing workloads |
| Glue Data Catalog | Metadata for supported lake consumers |
| Lake Formation | Fine-grained governance for supported lake access paths |
| IAM | Identities and permissions for AWS resources |
| KMS | Managed cryptographic key controls |
| VPC and private connectivity | Network boundaries and approved communication paths |
| CloudWatch / CloudTrail | Operational monitoring / AWS activity audit |
| Snowflake | Analytical processing and governed data products in this proposal |
04Questions before drawing more boxes
- What data volumes and peak arrival rates exist?
- What latency matters to the consumer? Is the workload batch or genuinely real time?
- Which regions are approved? Who owns encryption keys?
- Is cross-account access required?
- What are the recovery point objective (acceptable data loss) and recovery time objective (acceptable outage)?
- What is the retention classification? Which platform is authoritative for each dataset?
05Reliability and cost
Separate development, test and production. Use version-controlled deployment definitions and reviewed changes. Attach ownership and cost tags. Set budgets and operational alerts. Avoid running the same transformation in multiple platforms without a clear reason. Include storage, compute, data transfer, logging and duplicated migration environments in cost estimates.
Exercise
Draw a flow for daily customer-risk updates and a transaction consumer requiring yesterday's complete data by 07:00. Mark a late-file alert, a quarantine path, access boundaries and the lineage fields.
Expected outcome
A valid solution must define what happens when the source arrives at 08:00; “the job succeeded” is not sufficient evidence of timely data.