What will your role look like- You will be responsible for testing end-to-end data pipelines built in Azure Data Factory and Databricks, ensuring data flows accurately from source to sink. - Using PySpark and Python, you will validate complex business logic and data transformations created by the engineering team. - You will engage in pipeline reviews, treating "test as code" and ensuring that the engineering concepts applied are robust and scalable. - You will manage and execute tests within Azure DevOps environments, ensuring that data quality is maintained throughout the continuous integration and delivery lifecycle. - While the organizational title is "Tester," your role is that of a hands-on technical contributor. You will take ownership of the validation strategy for your specific domain.Why you will love this role- This isn't traditional manual or API testing. You will be writing code and working with the same "heavy-weight" tech stack as the developers. - You are the final gatekeeper for data integrity, ensuring that the insights derived from our platforms are accurate and reliable. - While we are looking for individual contributors, we are positioning this role for future scaling. You will have the opportunity to influence how our data QA processes evolve. - We have stripped away the overhead—you won't be bogged down by data generation, lineage tracing, or monitoring. You get to focus entirely on the logic and the code.We would like you to bring along- 12+ years of total experience, with at least 8 years of focused experience in data-centric technical roles. - Strong Python and PySpark skills are non-negotiable. You must be able to write scripts to validate large datasets. - Hands-on experience with Azure Data Factory and Databricks. - Advanced SQL knowledge for data manipulation, comparison, and validation. - Practical experience with Azure DevOps for managing pipelines and version control. - A solid understanding of data engineering principles—you must be able to speak the same language as the developers you are testing for.
Job Title
SDET Lead - Data Engineer