Job Title

Company Description ThreatXIntel is a startup cyber security company that specializes in protecting businesses and organizations from cyber threats. Our team offers services in cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. We provide customized and affordable solutions to meet the specific needs of our clients, focusing on continuous monitoring and testing to identify vulnerabilities proactively.Role Description We are looking for a highly experienced Freelance Senior Data Engineer with8+ yearsof expertise in building, optimizing, and maintaining large-scale data pipelines using Databricks and Apache Spark. You will play a key role in architecting data solutions that drive business insights and support advanced analytics initiatives. Key Responsibilities Architect, develop, and maintain scalable and efficient ETL/ELT pipelines using Databricks and Spark Lead optimization efforts for complex data workflows and Spark job performance tuning Collaborate closely with data scientists, analysts, and architects to deliver high-quality data solutions Design and implement robust data ingestion processes across diverse structured and unstructured data sources Implement data governance, security, and compliance measures within data pipelines Develop automation frameworks for workflow orchestration using Databricks jobs and/or tools like Airflow Mentor junior engineers and provide technical guidance on best practices and architecture decisions Support cloud migration, integration, and modernization projects involving Databricks on AWS, Azure, or GCP Ensure high data quality, reliability, and observability across all pipelines Required Skills & Experience 8+ yearsof experience in Data Engineering or related roles Proven expertise with Databricks and Apache Spark (PySpark, Scala, SQL) at an enterprise scale Deep knowledge of cloud data platforms: AWS, Azure, or Google Cloud Platform Extensive experience designing and optimizing large-scale ETL/ELT pipelines Strong skills in SQL and Python scripting Hands-on experience with data warehousing solutions such as Delta Lake, Snowflake, or Redshift Familiarity with orchestration tools like Airflow, Prefect, or native Databricks workflows Solid understanding of data modeling, data governance, and best practices for secure data management Excellent problem-solving skills and ability to work independently in a remote environment Strong communication and collaboration skills with cross-functional teams Preferred Qualifications Experience with streaming data platforms (Kafka, Kinesis) and real-time processing Knowledge of CI/CD processes for data engineering pipelines Exposure to machine learning workflows and MLOps on Databricks Previous experience working in freelance or contract roles with global clients