Job Title

Responsibilities:- Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes. - Create the data integration and data diagram documentation. - Lead the data validation, UAT and regression test for new data asset creation. - Create and maintain data models, including schema design and optimization. - Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.Qualifications and Skills:- Strong knowledge on Python and Pyspark - Expectation is to have ability to write Pyspark scripts for developing data workflows. - Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum - Expectation is to write SQL to query metadata and tables from different data management system such as, Oracle, Hive, Databricks and Greenplum. - Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks. - Expectation is to use Hue and run Hive SQL queries, schedule Apache Oozie jobs to automate the data workflows. - Good working experience of communicating with the stakeholders and collaborate effectively with the business team for data testing. - Expectation is to have strong problem-solving and troubleshooting skills. - Expectation is to establish comprehensive data quality test cases, procedures and implement automated data validation processes. - Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience. - 3-7 years of experience in Data Engineer. - Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL. - Experience in GCP computing platform, such as developing ETL processes - Strong communication, problem solving and analytical skills with the ability to do time management and multi-tasking with attention to detail and accuracy.