Job Title

Job TitleLead Data Engineer Python, PySpark & SQLLocationCanadaJob TypeFull time contractResponsibilitiesBuild scalable data ingestion and transformation pipelines using Python, PySpark, and SQL.Process raw CSV/text files from AWS S3, including validating headers, schema checks, and malformed file detection.Convert raw data into structured DataFrames and implement reusable data quality checks.Develop advanced transformations using SQL/PySpark (Window functions, LAG(), grouping logic, date gap detection, etc.).Deploy and tune PySpark applications on AWS EMR, optimizing executor memory, cores, shuffle behavior, and cluster performance.Work with AWS services such as S3, EMR, Glue, Lambda, IAM.Debug performance issues (OOM errors, shuffle spill, GC problems) and improve pipeline reliability.Lead design discussions, code reviews, and mentor junior engineers.Required Skills8+ years of experience in Data Engineering.Expert Python (file processing, scripting, validation automation).Strong PySpark (DataFrames, job tuning, distributed processing).Advanced SQL (analytical functions, performance tuning).Handson with AWS data stack: S3, EMR, Glue, Lambda.Strong understanding of Spark memory allocation, YARN container usage, and EMR resource tuning.Excellent debugging, communication, and problemsolving skills.Nice to HaveAirflow or Databricks experience.Terraform or CloudFormation.Experience with data lake formats (Delta, Iceberg, Hudi).Seniority levelMid-Senior levelEmployment typeContractJob functionInformation Technology #J-18808-Ljbffr