Job Title

"KPI Partners is looking for hands-on Azure Data Platform Architect with strong expertise in Databricks, DevOps (CI/CD), and IaC (Terraform/Bicep).This is a platform-focused architect role, not a data engineering or infrastructure admin role.The ideal candidate must have strong hands-on expertise across Azure, Databricks, IaC, and DevOps, with the ability to design, automate, and optimize the data platform end-to-end. This is a platform-focused architectural role where the candidate acts as a Databricks Platform Owner, guiding teams, enforcing best practices, and driving automation, governance, and scalability across the environment.The architect will own the Databricks platform setup, governance, automation, cost optimization, and best practices, while also enabling CI/CD and ADF monitoring.Key Responsibilities1. Databricks Platform Architecture- Set up and configure Databricks workspaces, clusters, schemas, RBAC, and security. - Define and enforce best practices: - Cluster policies - Workspace standards - Cost control - Job orchestration rules - Automate Databricks (clusters, jobs, configurations) using Terraform/Bicep. - Implement observability: - Failed job alerts - Cost and usage insights - Performance reporting2. ADF onitoring & Automation- Review and assess pipeline quality (not develop pipelines). - Automate extraction of ADF metrics (failures, slow pipelines, cost insights). - Configure integration runtimes and automate ADF operations when required.3. DevOps (CI/CD) Enablement- Implement CI/CD for Databricks and ADF using Azure Repos/GitHub + Azure DevOps. - Ensure controlled promotion from DEV → TEST → PROD. - Not responsible for full project management pipelines—only deployment pipelines.4. Infrastructure-as-Code (IaC)- Use Terraform/Bicep to automate deployments of: - Data product resource groups - ADF - Key Vault - Databricks - Basic networking components (deep expertise not required)5. Data Platform Governance (Long Term)- Help define platform policies and governance strategy: - Cataloging - Sensitivity labels - Data mesh / data product standards6. Cost Optimization- Generate Databricks usage and billing reports. - Identify expensive or inefficient workloads. - Recommend cost optimization strategies.Required Skills- Strong experience in Azure Databricks administration & platform design. - Hands-on experience with Terraform/Bicep for Databricks/Azure automation. - Good understanding of Azure DevOps or GitHub CI/CD. - Familiarity with Azure services (Key Vault, ADF, Storage, Functions, etc.). - Good scripting skills (Python/PowerShell). - Ability to design best practices, policies, automation, and cost optimization.Nice-to-Have- Experience with data observability practices. - Exposure to data governance frameworks."