Data Engineer – Architect

KEY RESPONSIBILITIES

• System Architecture: Design scalable, resilient, secure, and cost-effective data lakehouses and warehouses using modern patterns such as Medallion Architecture (Bronze/Silver/Gold).
• Technical Leadership: Set engineering standards, conduct architecture and code reviews, establish reusable frameworks, and mentor data engineers on best practices.
• Data Enablement: Deliver reliable, high-performance data platforms that support BI, data science, machine learning, and operational analytics.
• Cloud Architecture: Design end-to-end Azure/cloud infrastructure blueprints covering storage, compute, RBAC, security, network isolation, scalability, and cost optimization.
• Big Data & Processing: Architect and optimize Spark/Databricks/Snowflake workloads, including Delta Lake design, cluster sizing, partitioning, Z-Ordering, Vacuuming, clustering, and performance tuning.
• Data Pipelines & Ingestion: Build and standardize production-grade PySpark, Python, and SQL pipelines supporting batch, micro-batch, streaming, and CDC ingestion patterns.
• Data Modeling: Establish organizational data-modeling standards and design Star/Snowflake schemas, Kimball dimensional models, Data Vault patterns, and Lakehouse storage layers.
• Orchestration: Architect workflow dependencies, retries, alerts, monitoring, and backfilling using Azure Data Factory, Airflow, or Databricks Workflows.
• DevOps & CI/CD: Implement Git-based development standards and automated CI/CD pipelines using Azure DevOps or GitHub Actions for reliable data-platform deployments.
• Infrastructure-as-Code: Design and implement infrastructure provisioning using Terraform, Bicep, or equivalent IaC frameworks.
• Governance & Security: Implement metadata catalogs, lineage, data quality controls, access policies, and regulatory compliance practices using technologies such as Unity Catalog and Microsoft Purview.
• Performance & FinOps: Benchmark workloads, resolve bottlenecks, optimize memory and compute configurations, monitor cloud consumption, and drive cost-efficiency across data platforms.

MINIMUM REQUIREMENTS

• Overall Experience: 10 to 12+ years of IT/software engineering experience, with at least 5 years in Azure data architecture, cloud platform design, or equivalent data-platform architecture.
• Distributed Computing: 10+ years of experience working with distributed data processing or modern cloud data platforms, with strong hands-on exposure to Databricks, Spark, Snowflake, or equivalent technologies.
• Data Architecture: Strong understanding of Lakehouse/Warehouse architecture, Medallion Architecture, scalability, resiliency, security, and cost optimization.
• Data Modelling: Strong mastery of Kimball dimensional modelling, Star/Snowflake schemas, Data Vault, and analytical data design.
• Programming: Strong hands-on expertise in Python/PySpark and SQL, including production-grade transformations and performance optimization.
• Orchestration: Strong experience with Airflow, Azure Data Factory, Databricks Workflows, or equivalent orchestration platforms.
• DevOps: Hands-on experience with Git, CI/CD, Infrastructure-as-Code, Terraform/Bicep, and automated deployment practices.
• Cloud & Security: Strong understanding of Azure services, RBAC, identity, network isolation, security controls, monitoring, and enterprise governance.
• Communication & Leadership: Ability to communicate architecture decisions clearly, lead technical discussions, mentor engineers, and work effectively with cross-functional stakeholders.

GOOD TO HAVE

• Governance: Experience with Unity Catalog, Microsoft Purview, metadata management, data lineage, data quality, GDPR, HIPAA, or other regulated-data environments.
• Monitoring: Experience with Azure Monitor, Log Analytics, operational alerting, and platform observability.
• Cloud Breadth: Exposure to AWS or GCP data platforms in addition to Azure.
• Modern Data Engineering: Experience with CDC, streaming architectures, reusable ingestion frameworks, and platform engineering practices.