Job Title

Company DescriptionThreatXIntel is a startup dedicated to delivering tailored, cost-effective cybersecurity solutions, empowering businesses to protect their digital assets. Specializing in cloud security, web and mobile security testing, DevSecOps, and more, ThreatXIntel focuses on the unique challenges faced by organizations of all sizes. By proactively identifying vulnerabilities through continuous monitoring and testing, the company ensures the safety and reliability of digital environments. ThreatXIntel's mission is to provide reliable security services that enable businesses to grow with confidence, offering peace of mind in the ever-evolving landscape of cybersecurity.Role DescriptionWe are seeking an experienced Azure Databricks Data Engineer to architect and implement secure, scalable lakehouse solutions using the medallion architecture (bronze, silver, gold layers).The ideal candidate will have strong hands-on expertise in Azure Databricks, Apache Spark, Delta Lake, and large-scale data migration strategies. This role involves designing high-performance ETL/ELT pipelines, implementing governance controls, and delivering production-grade data products for business consumption.Key ResponsibilitiesArchitect and implement lakehouse solutions using medallion architecture on Azure DatabricksDesign and develop ETL/ELT pipelines to migrate data from transactional systems to Azure lakehouse environmentsBuild and optimize Spark jobs for large-scale batch and streaming data processingCreate curated gold-layer datasets ready for analytics and reportingImplement schema evolution and ingestion using Databricks Autoloader and Delta LakeEnable and manage Delta Lake features such as ACID transactions and time travelEnsure security, governance, and compliance using Unity Catalog, Azure Key Vault, and RBACCollaborate with data scientists, analysts, and stakeholders to deliver data productsMonitor, troubleshoot, and optimize workloads for performance and cost efficiencyMentor junior engineers and contribute to architectural best practicesRequired Technical Skills5+ years of experience in data engineering2+ years of hands-on experience in Azure Databricks and Apache SparkStrong expertise in Delta Lake, Databricks Autoloader, and Structured StreamingProficiency in PySpark and SQLStrong understanding of medallion architecture and lakehouse designExperience with Azure Data Lake Storage Gen2, Azure Synapse, Event Hub, and Azure FunctionsExperience implementing CI/CD using Azure DevOps, GitHub Actions, or TerraformKnowledge of Unity Catalog, MLflow, and Delta Live TablesExperience in schema design, data modeling, and data quality frameworksExperience in data migration strategies including CDC and streaming ingestionPreferred QualificationsAzure certifications such as DP-203 or Azure Solutions Architect ExpertExperience with data mesh or domain-driven designFamiliarity with Databricks REST APIs and job orchestrationExperience in regulated industries with strong data governance practices