Job Title

Job Description:- Designing and Implementing Data Pipelines: - Developing and maintaining scalable data pipelines using Databricks and related technologies like Azure Data Factory, Spark, and Python. - ETL Processes: - Creating and optimizing ETL (Extract, Transform, Load) workflows for efficient data extraction, transformation, and loading into Databricks. - Data Modeling and Warehousing: - Designing and implementing data models and data warehouses within the Databricks environment, including the Lakehouse Architecture. - Data Quality and Governance: - Ensuring data accuracy, consistency, and reliability through data quality checks, validation, and governance policies. - Collaboration and Communication: - Working with data scientists, analysts, and other stakeholders to understand requirements and deliver effective data solutions. - Performance Optimization: - Monitoring and optimizing data system performance, ensuring scalability, reliability, and cost-effectiveness within Databricks. - Troubleshooting and Support: - Identifying and resolving issues related to Databricks performance, data pipelines, and overall system functionality. - Staying Updated: - Keeping up-to-date with the latest Databricks features, best practices, and emerging technologies in the data engineering field. - Required Skills and Experience: - Databricks Proficiency: - Strong hands-on experience with Databricks, including Delta Lake, clusters, notebooks, jobs, and workspaces. - Programming Languages: - Proficiency in Python and SQL, with experience in PySpark for Spark-based data processing. - Data Engineering Concepts: - Solid understanding of data warehousing concepts, data modeling, ETL processes, and data quality principles. - Cloud Technologies: - Experience with cloud platforms like Azure (Azure Data Factory, Azure Data Lake, Azure Synapse) or AWS. - Data Integration: - Experience integrating with various data sources, including databases, cloud storage (Azure Blob Storage, ADLS), and APIs. - Agile Development: - Experience working within Agile development methodologies and DevOps practices. - Communication and Collaboration - Excellent communication and collaboration skills to work effectively with diverse teams.Skills:Pyspark advanced, Python Basics and SQL and Databricks AdvancedAirflow