Job Title

Senior Data Engineer Location - London Hybrid - 3 days in the office SC Clearance - ESSENTIAL Rate 550-600 per day Experience - 8+ years We are seeking a highly skilled and experienced SC Cleared Senior Data Engineer to join our client and play a key role in developing and maintaining their Azure Databricks platform for economic data . This platform underpins critical Monetary Analysis, Forecasting, and Modelling activities, making this position essential to their mission. Key Responsibilities Data Pipeline Development & Optimisation Design, develop, and maintain robust, scalable data pipelines for ingesting, transforming, and loading data from diverse sources (APIs, databases, financial data providers) into Azure Databricks . Optimise pipelines for performance, efficiency, and cost-effectiveness . Implement data quality checks and validation rules within pipelines. Data Transformation & Processing Implement complex transformations using Spark (PySpark or Scala) and related technologies. Develop logic for data cleaning, enrichment, and aggregation . Ensure data consistency and accuracy throughout the lifecycle. Azure Databricks Implementation Work extensively with Unity Catalog, Delta Lake, Spark SQL , and other Databricks services. Apply best practices for development and deployment . Optimise workloads for performance and cost . Program in SQL, Python, R, YAML, and JavaScript . Data Integration Integrate data from relational databases, APIs, and streaming sources . Implement data integration patterns and best practices . Collaborate with API developers for seamless data exchange. Data Quality & Governance Hands-on experience with Azure Purview for governance and quality. Implement monitoring, alerting, lineage tracking, and metadata management . Ensure compliance with data governance policies and standards . Collaboration & Communication Work closely with data scientists, economists, and technical teams to translate requirements into solutions. Communicate technical concepts to technical and non-technical audiences . Participate in code reviews and knowledge-sharing sessions . Automation & DevOps Automate pipeline deployments and data engineering tasks . Build CI/CD pipelines with DevOps teams. Promote and implement DevOps best practices . Essential Skills & Experience 10+ years in data engineering, with 3+ years hands-on Azure Databricks experience. Strong proficiency in Python and Spark (PySpark or Scala) . Deep understanding of data warehousing, modelling, and integration patterns . Extensive experience with Azure services (Data Factory, Blob Storage, SQL Database). Expertise in large datasets and complex pipelines . Proven Databricks implementation experience and certifications . Skilled in SQL and NoSQL databases . Experience with data quality, governance, and version control (Git) . Familiarity with Agile methodologies . Excellent communication and problem-solving skills . Experience with streaming technologies (Kafka, Azure Event Hubs). Knowledge of visualisation tools (Tableau, Power BI). Experience with DevOps tools (Azure DevOps, Jenkins, Docker, Kubernetes). Background in financial services or economic data environments . Relevant Azure certifications (e.g., Azure Data Engineer Associate). TPBN1_UKTJ