Job Title

A Data Engineer job description typically focuses on building and maintaining data systems, while an SDET (Software Development Engineer in Test)focuses on automating software testing and ensuring quality. Data Engineers work with data pipelines, data storage, and data processing, while SDETs focus on writing code to automate tests and improve the overall testability of software. SDET (Software Development Engineer in Test)Responsibilities: ● Design, develop, and maintain automated test frameworks for software quality and reliability.● Write high-quality code for test automation scripts and tools to enhance testing efficiency and repeatability.● Perform thorough ETL testing to validate data transformation logic and ensure accurate source-to-target mapping. ● Implement schema and contract testing to proactively detect schema drift and validate pipeline stage contracts.● Support data pipeline integration testing for seamless end-to-end data flow and component compatibility.● Collaborate with software developers to integrate quality assurance practices and improve testability.● Design and implement comprehensive test plans and test cases.● Conduct performance and security testing.Required Skills:● Strong proficiency in Python and SQL.● Solid understanding of software testing methodologies and best practices.● 5+ years of hands-on experience with test automation frameworks and tools.Data Engineer role qualifications (addition to SDET role) Responsibilities: ● Develop data pipelines for data ingestion, transformation, and loading from diverse sources into data storage solutions.● Develop and maintain optimal and reliable data storage solutions, including databases and data warehouses.● Implement comprehensive data quality and reliability measures ensuring accuracy, consistency, and integrity.● Collaborate with data scientists, data analysts, and cross-functional teams to understand business needs and build efficient data systems.● Implement and enforce robust data security measures, including access controls and data protection protocols. Required Skills:● Strong proficiency in Python and SQL.● Solid understanding of data modeling principles and database design.● Experience building and maintaining data pipelines and ETL processes.● Exposure to AWS services (Athena, Glue, DynamoDB, Lambda, S3 etc)Good to Have:● Exposure to relevant AWS services for testing.● Experience testing data pipelines and ETL processes.● Familiarity with data quality tools (Deequ, DQ).● Experience with Databricks, Airflow, PagerDuty (On-call), Amplitude● Effective collaboration with data scientists, analysts, and technical stakeholders.