Senior Data Engineer / GCP Big Data Intelligence SpecialistAbout the ProjectWe are launching a strategic initiative to modernize our data architecture and unlock deep business intelligence from our 10+ million customer records. This project, "GCP for Big Data Intelligence," centers on establishing a scalable, secure, and intelligent data platform using Google Cloud Platform (GCP). The core goal is to transition from raw data storage to AI-driven customer segmentation, directly impacting marketing campaign effectiveness and customer lifetime value (LTV).The Role: Senior Data Engineer / GCP SpecialistWe are seeking a highly experienced Senior Data Engineer with deep expertise in the Google Cloud Platform, particularly with BigQuery and the Vertex AI ecosystem. You will be responsible for the end-to-end implementation of the data pipeline, from secure ingestion of 10M+ records to the deployment of predictive machine learning models for marketing intelligence.️ Key ResponsibilitiesPhase 1: Data Ingestion and Security- Design and implement a secure, scalable data transfer strategy for over 10 million records from source systems (Google Drive/Local) into Cloud Storage (GCS). - Configure and manage IAM & Admin roles to ensure least-privilege access for data teams. - Utilize the Storage Transfer Service for reliable, automated data ingestion into GCS. - Define the schema and execute high-speed data loading into BigQuery for the raw dataset.Phase 2: Data Cleaning and Standardization- Develop complex BigQuery SQL and use Dataform to standardize and clean the raw dataset (e.g., field standardization, data quality checks). - Implement advanced deduplication logic using window functions (ROW_NUMBER() OVER (PARTITION BY...)) to maintain the most accurate customer record. - Create and maintain a production-ready, partitioned "clean table" (nu_skin_clean_contacts) as the single source of truth for AI/ML tasks.Phase 3: AI Intelligence and Segmentation- Integrate and leverage Vertex AI services to generate marketing intelligence features. - Execute Sentiment Analysis using the Natural Language API on unstructured text data. - Develop and train a Vertex AI AutoML model to predict customer behaviors (e.g., Propensity to Attend a Seminar). - Use model outputs in BigQuery to create hyper-segmented, targeted lists (e.g., Hyderabad_Top_50K_Attendees). - Integrate AI-generated features (e.g., Propensity_Score, Predicted_Value) back into the master contact table. - Explore and implement advanced AI models for Next Best Action and Lifetime Value (LTV) prediction.Phase 4: Integration and Deployment- Manage the export of final, segmented lists from BigQuery to GCS. - Design and implement the integration pipeline using tools like Zoho Flow or native connectors to ensure seamless data flow into downstream CRM/Marketing Automation systems (Zoho Marketing Automation).✅ Required Qualifications- 5+ years of experience in Data Engineering, with a focus on building high-volume data pipelines. - 3+ years of hands-on experience with Google Cloud Platform (GCP) services, including: - BigQuery (Advanced SQL, Partitioning, Clustering) - Cloud Storage (GCS) - IAM & Admin - Demonstrated experience in Vertex AI for model training, deployment, and using pre-trained APIs (e.g., Natural Language API, AutoML). - Strong proficiency in scripting/programming (e.g., Python, SQL) for data manipulation and automation.Preferred Qualifications- GCP Professional Data Engineer or Cloud Architect certification. - Experience with Dataform or similar tools for managing BigQuery code. - Experience integrating GCP with marketing automation/CRM platforms (e.g., Zoho, Salesforce, HubSpot). - Understanding of fundamental machine learning concepts and feature engineering.
Job Title
SeniorData Engineer