Direct message the job poster from TL Consulting GroupAccounts And Talent Lead at TL Consulting GroupAbout UsTL Consulting Group is a professional services organisation specialized in driving transformational uplift through offering intelligent automated solutions & services across the infrastructure, application, and data layer. Our transformational model & technology partners are cutting edge that provide a unique value proposition for organizations that are looking to move to a future state model being Cloud Native, driven by Microservices, DevOps/CICD and Containerization. Please refer to our website which highlights our broad client base now have an opportunity for a Senior Data Engineer to join our growing Professional Services team. This role requires someone that is willing to go above and beyond and embrace the challenge to take our organisation into the next chapter. You will be exposed to some of the most cutting-edge technologies and work on building state-of-the-art solutions. Your expertise in data analysis, data preparation, machine learning, and cloud technologies will be essential in unlocking the potential of our data assets.This role will be responsible for planning and executing full and incremental data loads, ensuring data transformation, cleansing, and validation, handling exception scenarios, and optimizing performance. Additionally, you will work closely with cross-functional teams, including data reporting, developers, and business stakeholders, to ensure smooth data migration and high data quality.Key Responsibilities:Design, develop, and manage Azure Data Factory (ADF) pipelines to facilitate end-to-end data migration.Implement full load and incremental load strategies for migrating data from various sources to the target system using dynamic pipeline.Handle pipeline scheduling, monitoring, and execution ensuring reliability and minimal downtime.Optimize data pipelines for performance and scalability.2. Data Transformation & CleansingEnsure data integrity, accuracy, and consistency through transformation and cleansing techniques.Develop ETL logic to map source-to-target transformations efficiently.Implement data validation checks to identify and handle discrepancies during migration.Identify, diagnose, and resolve issues related to data processing, transformation failures, or connectivity problems.Maintain detailed logs and proactively monitor data flow issues.Develop complex SQL queries, stored procedures, functions, and scripts for data extraction, transformation, and loading (ETL).Work with Azure SQL Server databases to design and optimize data models for efficient migration.Perform data profiling, indexing, and partitioning for enhanced query performance.Support data warehouse development and maintenance.Work extensively with Azure Storage solutions (Blob & ADLS Gen2) to manage and process large datasets.Ensure secure and optimized data transfer between environments.Manage access permissions and security compliance for Azure data storage.6. Testing & ValidationConduct pre-migration and post-migration data validation to ensure completeness and accuracy.Create test cases and scripts to verify data integrity across multiple environments.Work with QA teams and business users to validate data consistency after migration.Work closely with business analysts, solution architects, and other stakeholders to understand data migration requirements.Document ETL processes, data mappings, pipeline configurations, and troubleshooting steps for knowledge sharing.Provide technical support and guidance to the team regarding data engineering best practices.Required Qualifications & Skills:1. Experience & Background - Essential7+ years of hands-on experience in data engineering, data migration, and ETL development.Strong background in Azure Data Factory (ADF) with expertise in:Pipeline planning, designing, execution, and monitoringHandling full and incremental loadsException handling and performance tuningData transformation and cleansingExperience with SQL Server, including database design, writing complex stored procedures, and performance optimization.2. Technical SkillsStrong expertise in Azure Data Factory (ADF) and its integration with other Azure services.Proficiency in SQL Server and data warehousing concepts.Hands-on experience in Azure Storage solutions (Blob, ADLS Gen2) for handling structured and unstructured data - Essential.Good understanding of ETL frameworks and data transformation techniques - Essential.Experience with pipeline scheduling, exception handling, and monitoring.Knowledge of Power BI or other data visualization tools is an added advantage.Familiarity with Microsoft Fabric is a plus.3. Problem-Solving & Analytical ThinkingStrong problem-solving skills to troubleshoot migration challenges.Ability to analyze large datasets and identify inconsistencies or performance bottlenecks.Experience in optimizing SQL queries and ADF pipelines for better performance.Ability to communicate effectively with technical and non-technical stakeholders.Experience working in cross-functional teams to align data migration activities with business goals.Strong documentation skills to maintain clear and comprehensive project records.Preferred Qualifications (Good to Have):Experience with Microsoft Fabric for data integration and processing.Proficiency in Power BI or similar BI tools for reporting and visualization.Knowledge of data governance and compliance standards in cloud environments.Familiarity with other Azure cloud services related to data engineering.This role will be based in Melbourne VIC and will be hybrid in nature. No agencies please. Candidates must have valid visa with full work rights in Australia and must be based in Victoria. Visas will not be transferred or sponsored.TL Consulting Group understands the importance of fostering a diverse and inclusive workplace. We are committed to ensuring that everyone, no matter their gender, identity, sexual orientation, race, ethnicity, religion, disability, or age, has equal access to opportunities. We warmly welcome applications from all individuals, including Aboriginal and Torres Strait Islander peoples, members of the LGBTQI+ community, and individuals with disabilities.Seniority levelSeniority levelMid-Senior levelEmployment typeEmployment typeFull-timeJob functionJob functionInformation TechnologyIndustriesIT Services and IT ConsultingReferrals increase your chances of interviewing at TL Consulting Group by 2xGet notified about new Data Engineer jobs in Melbourne, Victoria, Australia.Melbourne, Victoria, Australia 2 months agoMelbourne, Victoria, Australia 17 hours agoMelbourne, Victoria, Australia 2 weeks agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 2 weeks agoMelbourne, Victoria, Australia 5 days agoMelbourne, Victoria, Australia 3 weeks agoMelbourne, Victoria, Australia 1 week agoSoftware Engineer (Python) - Data Platform (Open to remote across ANZ)Melbourne, Victoria, Australia 8 hours agoMelbourne, Victoria, Australia 3 weeks agoMelbourne, Victoria, Australia A$120,000.00-A$140,000.00 4 weeks agoMelbourne, Victoria, Australia 15 hours agoMelbourne, Victoria, Australia 3 weeks agoMelbourne, Victoria, Australia 2 weeks agoMelbourne, Victoria, Australia 5 hours agoParkville, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 2 weeks agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 4 days agoMelbourne, Victoria, Australia 5 days agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 2 weeks agoMelbourne, Victoria, Australia 3 weeks agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 1 week agoMelbourne, Victoria, Australia 2 weeks agoDocklands, Victoria, Australia 4 days agoMelbourne, Victoria, Australia 2 weeks agoMelbourne, Victoria, Australia 2 weeks agoMelbourne, Victoria, Australia 1 week agoWere unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI. #J-18808-Ljbffr
Job Title
Senior Data Engineer