Skip to Main Content

Job Title


Kafka Data Architect(Streaming And Payment)


Company : IBU


Location : Greater London, England


Created : 2026-01-12


Job Type : Full Time


Job Description

We are seeking a Hands-On Data Architect to design, build, and operate a high-scale, event-driven data platform supporting payment and channel operations. This role combines strong data architecture fundamentals, deep streaming expertise, and hands-on engineering in a regulated, high-throughput environment.You will lead the evolution from legacy data ingestion patterns to a modern AWS-based lakehouse and streaming architecture, handling tens of millions of events per day, while applying domain-driven design (DDD) and data-as-a-product principles.This is a builder role, not a documentation-only architect position.Key ResponsibilitiesData Products & ArchitectureDesign and deliver core data products including:Channel Operations Warehouse (high-performance, ~30 days retention)Channel Analytics Lake (long-term retention, 7+ years)Define and expose data APIs and status/statement services with clear SLAs.Architect an AWS lakehouse using S3, Glue, Athena, Iceberg, with Redshift for BI and operational analytics.Enable dashboards and reporting using Amazon QuickSight (or equivalent BI tools).Streaming & Event-Driven ArchitectureDesign and implement real-time streaming pipelines using:Kafka (Confluent or AWS MSK)AWS Kinesis / Kinesis FirehoseEventBridge for AWS-native event routingDefine patterns for:Ordering, replay, retention, and idempotencyAt-least-once and exactly-once processingDead-letter queues (DLQs) and failure recoveryImplement CDC pipelines from Aurora PostgreSQL into Kafka and the lakehouse.Event Contracts & Schema ManagementDefine and govern event contracts using Avro or Protobuf.Manage schema evolution through Schema Registry, including:Compatibility rulesVersioning strategiesBackward and forward compatibilityAlign domain events with Kafka topics and analytical storage models.Migration & ModernizationAssess existing “as-is” ingestion mechanisms (APIs, files, SWIFT feeds, Kafka, relational stores).Design and execute migration waves, cutover strategies, and rollback runbooks.Ensure minimal disruption during platform transitions.Governance, Quality & SecurityApply data-as-a-product and data mesh principles:Clear ownershipQuality SLAsAccess controlsRetention and lineageImplement security best practices:Data classificationKMS-based encryptionTokenization where requiredLeast-privilege IAMImmutable audit loggingObservability, Reliability & FinOpsBuild observability for streaming and data platforms using:CloudWatch, Prometheus, GrafanaTrack operational KPIs:Throughput (TPS)Processing lagSuccess/error ratesCost per million eventsDefine actionable alerts, dashboards, and operational runbooks.Design for high availability with multi-AZ / multi-region patterns, meeting defined RPO/RTO targets.Hands-On EngineeringWrite and review production-grade code using:Python, Scala, SQLSpark / AWS GlueAWS Lambda & Step FunctionsBuild infrastructure using Terraform (IaC).Implement CI/CD pipelines (GitLab, Jenkins).Enforce automated testing, performance profiling, and secure coding practices.Required Skills & ExperienceStreaming & Event-Driven SystemsStrong experience with Kafka (Confluent) and/or AWS MSKExperience with AWS Kinesis / FirehoseDeep understanding of:Event ordering and replayDelivery semanticsOutbox and CDC patternsPractical experience using EventBridge for event routing and filteringAWS Data PlatformHands-on experience with:S3, Glue, AthenaRedshiftStep Functions and LambdaFamiliarity with Iceberg-based lakehouse architecturesExperience building streaming pipelines into S3 and GluePayments & Financial MessagingExperience with payments data and flowsKnowledge of ISO 20022 messages:PAIN, PACS, CAMTUnderstanding of payment lifecycle, reconciliation, and statementsExposure to API, file-based, and SWIFT-based integration channelsData Architecture Fundamentals (Must-Have)Logical data modeling (ER diagrams, normalization up to 3NF/BCNF)Physical data modeling:Partitioning strategiesIndexingSCD typesStrong understanding of:Transactional vs analytical schemasStar schema, Data Vault, and 3NF trade-offsPractical experience with:CQRS and event sourcingEvent-driven architectureDomain-driven design (bounded contexts, aggregates, domain events)