Payments · Event-Driven Architecture · Java 21
On-Behalf Clearing
A near-real-time payment clearing solution designed to reduce dependency on end-of-day batch processing by transforming authorization events into an event-driven clearing workflow.
The Context
Authorization happens in seconds. Settlement does not.
In a traditional card-present transaction, an ISO 8583 authorization request travels from the point of sale through the payment network to the issuing bank.
The issuing bank performs checks such as card validity, available balance, account status and other authorization controls before returning an approval response.
The merchant can then complete the sale and the customer can leave with their goods within seconds.
However, transaction authorization and the actual clearing and settlement of funds are separate processes.
The Challenge
Approval was real-time. Clearing remained batch-driven.
Traditionally, merchants aggregated approved transactions and generated a batch file at the end of the day for clearing and settlement.
This introduced a delay between the time a transaction was approved and the time the clearing process was initiated. Funds could therefore reach the merchant only after the subsequent settlement cycle.
The batch-based model also created operational dependencies around end-of-day processing, reconciliation complexity and limited visibility into the processing status of individual transactions.
The Approach
Initiating clearing from the transaction event
The idea behind On-Behalf Clearing was to move the initiation of clearing closer to the authorization event rather than waiting for the merchant's end-of-day batch process.
Transaction events were recorded in the Journal, which acted as the single source of truth and published relevant events to Kafka.
OBC consumed the authorization-response events, transformed them into the canonical Nexus message format and initiated downstream clearing processing through an event-driven workflow.
In effect, the platform performed part of the clearing initiation on behalf of the merchant, reducing dependency on merchant-side batch aggregation.
How It Worked
Event-driven clearing architecture
The solution combined transaction journaling, Kafka-based event processing, message transformation, temporary persistence and acknowledgement-based recovery into a resilient processing flow.
System Flow
From authorization event to clearing acknowledgement
OBC transformed the traditional batch-oriented clearing process into an event-driven flow. Once an authorization response was recorded at the switching layer, the relevant transaction event could enter the clearing pipeline without waiting for an end-of-day batch.
- Journal and Event PublicationAuthorization-response transaction events were recorded in the Journal, which acted as the single source of truth for transaction activity. The Journal published the relevant 0110 authorization-response events to Kafka for downstream processing.
- OBC Consumes the Authorization EventThe OBC application consumed the relevant authorization-response events from Kafka and initiated downstream clearing processing without waiting for the merchant's traditional end-of-day batch cycle.
- Transform to Nexus Canonical FormatOBC transformed the ISO 8583 authorization-response message into the Nexus canonical message format required by the downstream clearing component, mapping the relevant fields according to the defined ISO and Nexus message mappings.
- Durable Staging in PostgreSQLBefore sending the transformed Nexus message downstream, OBC temporarily persisted it in PostgreSQL. This provided a durability checkpoint and recovery point for messages that had been created but were still awaiting successful downstream processing.
- Publish to the Nexus Processing PipelineOBC published the transformed Nexus message to a Kafka topic, where it was consumed by the downstream Nexus component for clearing-related processing.
- Process and AcknowledgeAfter processing the message, Nexus published a success or failure acknowledgement to a dedicated Kafka acknowledgement topic. OBC consumed this acknowledgement to determine whether the downstream processing had completed successfully.
- Confirm, Clean Up or RetryWhen a successful acknowledgement was received, OBC removed the corresponding pending message from PostgreSQL. If an acknowledgement was not received within the defined timeout window of 10 seconds, OBC's background reconciliation process identified the pending message and automatically re-published it for processing.This self-healing retry mechanism provided resilience against transient failures involving downstream processing or message delivery, without requiring manual intervention.
Reliability By Design
Persistence, acknowledgement and recovery
Publishing a message to a downstream component was not treated as the end of the processing lifecycle. OBC maintained temporary state in PostgreSQL until a corresponding acknowledgement was received.
When a successful acknowledgement arrived, the corresponding record was removed from PostgreSQL.
OBC also performed a background sweep to identify messages that were still awaiting acknowledgement after the configured acknowledgement window.
Messages without a timely acknowledgement were automatically re-published for processing, allowing the system to recover from transient failures without manual intervention.
My Contribution
Technical ownership from architecture through delivery
As Technical Manager, I owned the end-to-end technical design and helped drive the project from architecture through implementation and delivery.
- Architected the full message flow — from switch-layer interception through Kafka to the Nexus clearing component.
- Defined Kafka topology, including broker considerations, topic partitioning strategy and consumer group design to support throughput and fault isolation.
- Designed the resiliency model using PostgreSQL staging, acknowledgement timeout and automatic retry mechanisms to create a self-recovering processing flow.
- Led technical alignment across junior and senior engineers on a business-critical, low-tolerance-for-error payment processing system.
- Drove execution from technical design through implementation, testing and delivery.
Result
From batch dependency to event-driven processing
The solution enabled the clearing workflow to be initiated from transaction events rather than relying entirely on the merchant's end-of-day batch aggregation process.
The event-driven architecture introduced greater visibility into transaction processing and reduced dependency on a single batch processing cycle.
The combination of Kafka-based processing, temporary persistence, acknowledgement tracking and automated retry provided a resilient mechanism for handling transactions awaiting downstream confirmation.
What I Learned
Designing for recovery, not just the happy path
This project strengthened my understanding of distributed, event-driven architecture and the importance of designing systems around failure scenarios rather than assuming successful delivery.
The technical challenge was not simply transforming one payment message into another. It was ensuring that messages could be tracked, recovered and retried when downstream acknowledgement was delayed or unavailable.
It also reinforced the importance of early architectural decisions around Kafka topics, consumer groups, persistence and recovery mechanisms, as these decisions have a significant impact on throughput, resilience and operational behaviour.
Above all, the project helped shape my approach to solution architecture: understanding the complete business flow, identifying failure points and designing a system that can recover safely when things do not follow the expected path.