Job Title

We are looking for an experienced Azure Databricks Engineer with strong hands-on expertise in Python, SQL, and Apache Spark to design, build, and optimize scalable data pipelines and analytics solutions on the Azure cloud platform. The ideal candidate should have experience working with large datasets, distributed data processing, analytics use cases, and modern data engineering practices. ResponsibilitiesDesign, develop, and maintain scalable data pipelines using Azure DatabricksImplement ETL/ELT workflows using PySpark, Spark SQL, and PythonImplement pipelines for data ingestion using Azure Data FactoryOptimize Spark jobs for performance, cost, and scalabilityWork with structured and semi-structured data (Parquet, Delta, JSON, CSV)Build and manage Delta Lake tables (ACID, time travel, schema evolution)Integrate Databricks with Azure Data Lake Storage (ADLS Gen2)Develop complex queries and transformations using SQLCollaborate with Data Science teams to prepare data for modelling use cases, ensuring appropriate transformations, feature generation, and storage.Follow best practices for security, access control, and governance in AzureEnsure data quality, validation, and monitoring using testing toolsDeployment of solutions to Production environmentsQualifications4+ years of experience in Data Engineering, ideally supporting POS and SKU datasets.Handling high volume transactional datasetsStrong hands-on experience with Azure DatabricksUnderstanding of the Medallion Architecture and implementing it within DatabricksGood understanding of data modelling techniques Proficiency in Python for data processingStrong knowledge of SQL (joins, window functions, performance tuning)Hands-on experience with Apache Spark / PySparkExperience working with Delta LakeKnowledge of Azure Data Lake Storage (ADLS Gen2)Understanding of distributed computing conceptsExperience with Git version controlUnderstanding of ML use cases and data considerations for model developmentGood to HaveExperience working with Retail/CPG dataData governance awareness for customer data (PII, tokenisation, Unity Catalog)Exposure to CI/CD pipelines (Azure DevOps, GitHub Actions)Familiarity with cloud security and RBAC in AzureExperience supporting data science workflows with feature stores or ML pipeline tooling.