Job Title

Azure Databricks Overview We are seeking a highly skilled and motivated Senior Technical Lead to join our team and drive the development and deployment of Azure Databricksbased solutions This role will require expertise in Python programming Databricks administration CICD pipelines for Databricks artifacts and handson experience with Delta Live Tables Auto Loader Unity Catalog and Databricks Asset Bundles The ideal candidate will have a strong technical background proven experience in leading teams and a passion for building highquality scalable solutionsKey Responsibilities Python Programming Lead the development of robust scalable and efficient Python code for data processing transformation and analysis within the Databricks environment Ensure adherence to coding standards quality and performance best practices Azure Databricks Administration Oversee Databricks platform configuration resource management cluster optimization and monitoring to ensure high availability and performance Implement best practices for managing Databricks workspaces libraries and notebooks CICD for Databricks Artifacts Design and implement CICD pipelines for automating the deployment of Databricks artifacts such as notebooks libraries jobs and Delta tables Use tools such as Azure DevOps GitHub Actions or Jenkins to streamline the deployment process Delta Live Tables Design and manage Delta Live Tables pipelines for realtime data processing ensuring data quality reliability and optimal performance in the processing layers Auto Loader Implement and optimize Auto Loader for scalable faulttolerant ingestion of data from cloud storage sources into Databricks Unity Catalog Manage and configure Unity Catalog for centralized governance of data across workspaces ensuring proper access controls data lineage and compliance Databricks Asset Bundles Leverage Databricks Asset Bundles to manage and share reusable components including notebooks libraries and ML models across teams or projects Team Leadership Mentor and guide junior team members fostering a collaborative environment that encourages knowledge sharing innovation and personal growth Provide technical leadership and support to ensure the success of the teams initiatives Collaboration and Communication Work closely with stakeholders including data engineers data scientists and business analysts to understand requirements and deliver endtoend solutions Communicate technical challenges and solutions clearly to both technical and nontechnical stakeholdersRequired Qualifications Experience in Python Programming o Minimum of 5 years of experience in Python programming with a focus on data engineering and data science workflows o Expertise in designing and building scalable Pythonbased ETL pipelines using libraries such as Pandas NumPy PySpark and requests o Proficiency in debugging optimizing and maintaining Python code to ensure high performance and reliability in a distributed environment Extensive Experience with Azure Databricks o Minimum of 3 years of handson experience working with Azure Databricks in a production environment o Strong knowledge of Databricks clusters workspaces and the Databricks runtime o Ability to optimize and troubleshoot Sparkbased jobs and notebooks ensuring performance and costefficiency in cloud environments CICD for Databricks Artifacts o Proven experience in setting up and managing Continuous IntegrationContinuous Deployment CICD pipelines for Databricks artifacts notebooks libraries and jobs o Familiarity with version control systems Git and experience with tools such as Azure DevOps GitHub Actions or Jenkins for automated deployment o Experience with integrating Databricks with external version control repositories eg GitHub Azure Repos for collaboration and artifact management Delta Live Tables o Strong experience in designing managing and optimizing Delta Live Tables DLT pipelines to support realtime reliable and scalable data processing o Ability to handle complex data transformations ensure data quality and monitor pipeline performance within the Delta Lake architecture o Expertise in implementing automated data quality checks and validation within Delta Live Tables pipelines Auto Loader o Practical experience with the Auto Loader feature in Databricks for realtime faulttolerant and scalable data ingestion from cloud storage eg Azure Blob Storage ADLS Gen2 o Knowledge of best practices for optimizing Auto Loader performance and ensuring minimal latency in largescale data ingestion tasks Unity Catalog for Data Governance o Indepth experience in implementing and managing Unity Catalog for central data governance in Databricks o Proven track record in setting up and enforcing finegrained access control and managing permissions for users groups and roles o UnderPlease share your details in below Link.