Job Title

Key Responsibilities: Data Pipeline Development: o Develop and maintain data pipelines for ingesting, transforming, and loading data from various sources (e.g., APIs, databases, files) into the Azure Databricks platform. o Implement data extraction, transformation, and loading (ETL) processes. o Ensure data pipelines are efficient, reliable, and scalable. Data Transformation & Processing: o Implement data transformations using Spark (PySpark or Scala) and other relevant tools. o Perform data cleaning, validation, and enrichment. o Ensure data quality and consistency. Azure Databricks Implementation: o Work with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other related services. o Follow best practices for Databricks development and deployment. o Contribute to optimising Databricks workloads. o Need to program using the languages such as SQL, Python, R, YAML and JavaScript Data Integration: o Integrate data from various sources, including relational databases, APIs, and file systems. o Work with different data formats (e.g., CSV, JSON, Parquet, Delta). o Ensure data is readily available for analysis and modelling. o Data should be accessed from downstream to build dashboards and interactive reports. Data Quality & Monitoring: o Hands on experience to use Azure Purview for data quality and data governance o Implement basic data quality checks and monitoring. o Identify and resolve data quality issues. o Contribute to improving data quality processes. Collaboration & Communication: o Collaborate with other data engineers, data scientists, and other team members. o Communicate technical concepts clearly. o Participate in team meetings and code reviews. Automation & DevOps: o Build CI/CD pipelines, for environmental deployments. o Contribute to automating data pipeline deployments and other data engineering tasks. o Learn and apply DevOps principles.