
Custom Data Lake with API Orchestration Layer
Cloud-native data lake on AWS S3 and Azure ADLS Gen2 with API-driven workflow orchestration, Spark/Glue ETL, unified metadata governance, and centralized observability.
AWS + Azure (Multi-cloud)
Cloud Providers
Spark, Glue, ADF, Databricks
ETL Engines
Step Functions + Durable Functions
Orchestration
Eliminated legacy pipelines, improved reliability
Result
Designed and implemented a custom data lake to centralize structured, semi-structured, and unstructured data across multiple business domains using AWS S3 and Azure Data Lake Storage Gen2. Standardized REST endpoints via Azure API Management and AWS API Gateway enabled consistent, governed data intake.
The orchestration layer coordinated ingestion, validation, transformation, and publishing using Azure Durable Functions, AWS Step Functions, and event-driven architecture design patterns (SNS/SQS, EventBridge, Azure Event Grid). AWS Glue Data Catalog, Azure Purview, and custom metadata services tracked lineage, schema, and transformation history. Spark (Databricks), AWS Glue ETL, and Azure Data Factory handled batch and near-real-time workloads. Fine-grained access controls (IAM, Azure RBAC), encryption, and centralized structured logging practices via CloudWatch, Azure Monitor, and Datadog provided full observability without excess.
Architecture Diagrams

Explore the Full Portfolio
This is the complete portfolio of Sufi Khan Sulaiman, a technology leader specialising in B2B commerce and digital automation. Start from the Home page for the overview, then move through two decades of career experience across FLIR Systems, Lorex Technology, and 1c Platform, and the full catalogue of project case studies spanning headless commerce migrations, AI recommendation engines, and multi-channel fulfilment systems.
The skills and certifications page maps the technical and leadership capabilities behind the work, while the articles and the knowledge base break down the thinking into actionable frameworks. For hands-on learning, the tutorials and applications sections cover practical builds from front-end fundamentals to full-stack web apps.
For consulting engagement, the expertise page outlines service offerings, the ecommerce hub covers platform architecture and automation strategy, and the ecommerce guide (PDF) is a downloadable 55-page field manual. When you are ready to talk, the contact page is the direct line.