Key Responsibilities: ● Architect and design end-to-end data platform solutions using Azure Databricks as the primary compute engine. ● Define data lake strategies and structures leveraging Azure Data Lake Storage Gen2 for raw, curated, and consumer zones. ● Design and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Databricks notebooks/jobs. ● Lead platform-level security design by integrating with Azure Active Directory (AAD) and applying role-based access control (RBAC), Unity Catalog (if applicable), and data masking policies. ● Define standards for Delta Lake implementation, partitioning, schema evolution, and cost optimization in Spark jobs. ● Guide teams on the use of CI/CD, infrastructure-as-code, and automated deployment patterns for data workloads. ● Ensure data solutions meet governance, quality, and compliance requirements across environments. ● Collaborate with business, data governance, and engineering teams to align architecture with enterprise data strategies. ● Provide architecture oversight for performance tuning, scalability, disaster recovery, and observability setup. Must-Have Skills:● 10+ years of experience in data engineering or architecture, with at least: 3–5 years of hands-on experience with Azure Databricks and Apache Spark. Strong exposure to Azure Data Lake Storage Gen2. Proficiency in Azure Data Factory for pipeline orchestration. Good understanding of Azure Active Directory, AAD groups, service principals, and access control models. ● Expertise in designing multi-zone lakehouse architectures and distributed processing solutions. ● Strong in Python (PySpark), SQL, and Spark job performance tuning. ● Experience with Delta Lake, schema evolution, and versioning. ● Proven ability to lead and mentor engineering teams on architecture and technical decisions. Nice-to-Have Skills: ● Knowledge of Unity Catalog for data governance and metadata management. ● Experience with Azure Synapse Analytics, Event Hubs, or Purview. ● Familiarity with data mesh, domain-driven data ownership, or MDM integration. ● Understanding of data quality frameworks and observability tooling. Preferred Certifications: ● Databricks Certified Data Engineer – Professional ● Azure Data Engineer Associate
Job Title
Technical Architect - Data