On Building a Collection Pipeline That Must Not Be Wrong
Scheduled auto-debits, partner systems you do not control, and no tolerance for a wrong number. What we built, what we got wrong, and what we learned.
In 2023, a mandate went out twice. Same instalment, same account, same day. Two debits. The technical cause was not mysterious once we found it: the presentation layer had no idempotency check, and a retry after a transient network error processed the message a second time. The fix took a morning. What it exposed took much longer to fully absorb: we had built a system that was correct on the happy path and had not yet asked hard enough questions about what happens when things go sideways.
This post is about how we rebuilt the thinking, not just the code.
What Makes This Hard
Jodo collects fee instalments on behalf of education institutes and businesses. We handle the full cycle: scheduling reminders and debits, fetching the current due amount from a partner’s system where that integration exists, and then presenting the mandate or sending the communication within the permitted window.
We work with close to 4,000 institutes and 12,000 other businesses. Not all of them have a live fee fetch integration; a significant portion do, and those are what this pipeline primarily handles. This is not a uniform set of integrations: partner systems range from well-maintained fintech platforms to systems built in-house with varying levels of engineering support and inconsistent data integrity guarantees. Each has its own reliability profile, its own latency, and its own approach to data consistency.
The amount due in a partner’s system is a live value. Manual collections, fee revisions, discounts, late fees, scholarship adjustments. Any of these can change the number between when we scheduled the debit and when we present it. We are not the system of record. Our job is to be current at the moment we act.
Five things make this engineering-difficult:
Time constraints with no slack. Nothing can go out after 7PM: not reminders, not mandate presentations. We set this cutoff ourselves, as a buffer — we cannot afford to be the reason a customer was not notified before a debit, or the reason a collection was delayed. These are not SLOs. They are hard walls. Miss them and the action either does not happen or we have failed the partner.
External systems we do not control. The partner’s API is the source of truth for the amount. Response times vary enormously, from a few milliseconds on fast systems to 15 seconds on slow or loaded ones. The same partner can respond in 200ms on a quiet Tuesday and timeout on a Friday morning when their own traffic spikes.
Load we cannot always predict. On any given day, the volume of instalments we need to process can vary by as much as 60x, from around a thousand on quiet days to 60,000 and beyond on peak days. Some of this variance is predictable: semester cycles, renewal periods, large batch schedules. Some is not. A bulk reschedule decision taken by one large institute after our daily predictions have run can quietly double the day’s count with no advance notice.
Fresh data, every time. We cannot cache the amount we pulled yesterday and present it today. Any caching strategy reintroduces the staleness problem we are trying to solve. If a payment came in this morning, caching means we present the wrong amount this afternoon. Every fetch must be live.
Two payment rails with different mechanics. eNACH mandate presentation and UPI Autopay behave differently. Both rails fetch the amount in Stage 2. The difference is what that fetch feeds into: Present Mandate for eNACH, Pre-Debit Notification for UPI Autopay.
The Architecture
The pipeline has three stages. They run in sequence. Each stage has a time budget. Fail to stay inside it and the stage downstream is at risk.
Booking the work before it can be lost
EventBridge and a Scheduler fire a Lambda for each scheduled job. This covers three distinct types of work: reminder communications, mandate presentations, and pre-debit notifications. All three follow the same pipeline pattern — the difference is when they are triggered, not how they are processed.
The Lambda is deliberately thin. It calls a Scheduling API, which creates a job for a worker. The worker performs the actual scheduling work: identifying target instalments, writing them to a MySQL outbox table, and publishing a Kafka message per instalment. No calls to partner systems at trigger time. No heavy computation in the Lambda. The function’s only job is to initiate the work and hand it off.
Scheduling is more complex than it sounds. A single debit event can have multiple reminders configured before it. A partner might set a reminder three, five, and seven days prior, with another on the morning of the debit itself. Each of these is a separate scheduled job. The scheduler has to handle all of them correctly, in the right order, without collision. For the mandate presentation itself, the job runs the day before the actual debit date; there is a window the presentation must land in, and the scheduler is responsible for hitting it.
The load also overlaps across debits. The day that reminders go out for a debit three days away is also the day that presentations run for tomorrow’s debit. A partner with three reminder intervals configured will have multiple batch types active simultaneously on certain days. The scheduler handles all of this without collision, but it means the volume on any given day is not just one batch — it is the sum of whatever reminder and presentation batches happen to align on that date.
The outbox as a recovery anchor. Think of it as writing something down before making a phone call. If the call drops, you still have the note. The MySQL outbox is that note. We use the transactional outbox pattern: write to MySQL atomically, then publish to Kafka. The sequence matters. If the Kafka publish fails after the outbox write, the record exists and the scheduler can retry from a known state. If the worker crashes partway through a large batch, the outbox records what was committed. There is no ambiguity about what work is pending.
We applied this thinking to scheduling before we applied it to presentation. That asymmetry cost us. Early on, the presentation layer had no idempotency check. Kafka delivers at-least-once: a transient failure or consumer rebalance can result in the same message being processed more than once. Without a guard, the same mandate went out twice. The outbox protected us from losing work. We needed the complementary safeguard at presentation: protecting against the same work being executed twice. The fix was to record a confirmed presentation as a terminal state before proceeding. We just had not carried the idea far enough the first time.
Alerts fire if a scheduler job fails to complete. This is the right place for them. By the time the fetch window opens, ops has a narrow time budget to investigate and recover. Catching a scheduling failure before the fetch window begins is what makes recovery possible.
The Kafka topic is instalment-level. Each message is one instalment. This isolation is intentional: a failure on one instalment, whether a timeout, a bad response, or a validation rejection, should not require retrying the entire batch. Consumers pick up from Kafka and scale independently of the scheduler.
The hour that everything depends on
The amount in a partner’s system is not a fixed value; it is a reading, and readings go stale. A payment that came in at 10AM changes the number. A fee revision pushed at noon changes it again. Yesterday’s fetch is fiction.
Before we can present a mandate or send a reminder, we need the number as it stands right now. That means making an API call to the partner’s system for each instalment in the batch. Manual collections happen constantly; someone walks in and pays cash. Fee structures change: a scholarship is applied, a late fee is added, an error is corrected. We only know the current due if we ask. The requirement for fresh data is not a design preference; it is the business reality.
The 1hr SLA. From day zero, our internal target has been to complete the full fetch stage within one hour, regardless of instalment count. We call this a survival SLA; it defines failure, not ambition. The framing matters: this is not a goal we try to meet. It is a condition we design against. If we cannot finish fetching within that window, the downstream stages, presentation and communication, are at risk of missing their hard deadlines.
Meeting the 1hr target is arithmetic at the system level. 60,000 is a single batch count, not a daily aggregate — a single presentation or reminder run can reach this on a peak day. At 60,000 instalments and a mean processing time per instalment of roughly 5–6 seconds (accounting for the range from sub-second responses on fast partners to 15-second timeouts on slow ones, plus retries), the worker-second requirement is around 300,000–360,000. Spread across a 3,600-second window, you need around 90 concurrent workers at peak. Task scaling provisions this automatically: as consumer lag rises, consumers scale out to meet it. The math holds.
What the math cannot control. What does not scale the same way is the other side of the network call. We bring significant concurrency to partner APIs. Not every partner system is built to absorb that. A system that handles its own normal traffic fine can saturate when it starts receiving 90 concurrent inbound requests from Jodo during peak fetch time. This is not a hypothetical. We have seen it.
This is where our predictions mechanism matters. We run a routine check of instalments due in the next 30 days. When a high-volume period is flagged (a large institute’s renewal cycle, a semester changeover, a cohort moving to a new fee tier), the signal goes to our internal ops team, who communicate proactively with high-volume partners. The ask is simple: know that a heavy day is coming and prepare your systems accordingly.
This process is not automated end-to-end, and that is deliberate. The communication often involves nuance that a system signal alone cannot carry. When a bulk reschedule hits after predictions have run, however, that proactive window is gone. The spike arrives unannounced, ops coordinates reactively, and the consumer layer absorbs what it can while the partner side adjusts.
I do not have a clean answer to whether proactive partner communication is a sustainable substitute for per-partner rate limiting at higher scale. It is what worked first, and we have not yet needed to replace it. Currently we apply the same concurrency and retry configuration across all partners; per-partner rate limiting is a natural next step, held back only by operational complexity that the failure rate has not yet justified.
What the data taught us about trust. I went into this assuming we could treat all partners as reliable data sources. I came out of the first year with a much clearer picture of the variance.
Specific problems we encountered: some systems allow fee amounts to be updated without enforcing logical limits on the change, a configuration that lets any amount be set, including multiples of the previous value. Some do not validate whether a new amount is internally consistent with prior payment history. Some return technically valid responses that represent a data entry error: an accountant miskeying an amount, a system bug that wrote the wrong value to a field. A record showing ₹50,000 due where the previous amount was ₹5,000 might be a legitimate revision or it might be a decimal error. Without knowing which, we cannot safely present the mandate.
An overdebit is not a minor inconvenience. In the eNACH rail, a debit that goes out for the wrong amount triggers a reversal process that is operationally expensive for everyone involved. Trust is damaged in a context where it matters: these are school fees and business payments, not SaaS subscriptions.
Over time we built a validation layer that sits between the fetch result and the presentation step: allowed amount ranges, maximum percentage increase thresholds per cycle, consistency checks against prior payment history, flags for amounts that change beyond a defined magnitude between fetches. None of these rules came from a whiteboard session. Each one was written in response to a specific incident. The 2023 version of the system had none of them.
I learned that you cannot design as if external systems are reliable sources of truth. You have to design as if they are usually right, and build the validation layer that catches the cases when they are not. Jodo is the last line of validation before money moves. That is not a slogan. It is the design constraint that every part of this system is built around.
Where the system meets the outside world
This is where the work reaches the outside world. It is also where Jodo is not the only system that needs to be ready.
eNACH. The amount fetched in Stage 2 feeds directly into mandate presentation. We present the mandate to NPCI gateway partner. The bank processes the debit. The eNACH rail has its own presentation deadline for the next debit cycle; miss it and the debit does not happen. Our 7PM cutoff is set to give us buffer to land within that window, not to match it exactly. There is no “try again tomorrow” for a missed presentation window. The debit cycle resets.
UPI Autopay. The fetch happens in Stage 2 on this rail too. The difference is what follows: instead of going directly to mandate presentation, the fetched amount feeds into a Pre-Debit Notification, the PDN. The PDN communicates the upcoming debit to the customer and locks the amount. After PDN, the debit follows. Both rails share the same Stage 2 fetch; they diverge at Stage 3.
Both the PDN and the eNACH mandate presentation happen the day before the actual debit. The amount is locked at that point. Partners who run auto-debit collections are made aware that manual collections should not take place during this window. The period between presentation and debit is treated as a lock — the amount is committed, and the debit will follow.
Reminders. Communications go out informing customers of an upcoming debit. The 7PM cutoff is ours. We set it as a buffer against worst-case delays, not because the rails require it, but because we cannot be the reason a customer was debited without being told. We enforce it at the system level. If a job runs late and a batch would go out after 7PM, that batch does not go. It reschedules or gets flagged for ops review. There is no override.
The downstream services also have to scale. The pipeline’s output is a set of signals handed to two separate downstream service groups: gateway services that handle mandate presentation to the payment rail, and communication services that dispatch the actual SMS or notifications. At peak, both of these receive 60,000 signals in a compressed window.
The communication services have their own rate considerations: SMS gateways impose limits, vendor-level throughput caps exist, and all of it has to stay inside the 7PM cutoff. The gateway services face the volume of presentations simultaneously, and they need to be sized to handle the concurrency the pipeline generates.
This is a coordination problem that the pipeline’s design cannot solve alone. The same predictions mechanism that tells partners to prepare for a heavy day also informs the teams running these downstream services. When we know volume will spike, the communication layer and gateway layer need to know too; their scaling decisions are upstream of the actual execution window.
What failure actually looks like
Partner system unavailable during fetch. Transient unavailability is handled by retries with backoff. For extended unavailability, the instalment times out and moves to a dead-letter state. It does not silently skip to presentation. Ops is alerted. Resolution requires explicit action before the mandate attempt goes out. The dead-letter state is a gate, not a bypass.
Duplicate presentation. Kafka delivers at-least-once: a transient failure or consumer rebalance can result in the same message being processed more than once. The presentation layer handles this as an idempotent consumer: a confirmed presentation is recorded as a terminal state. Any subsequent attempt for the same instalment checks this state and returns success without making the call again.
The more subtle risk is a gateway timeout: the mandate may have been accepted by the payment rail but the response never reached us. A retry in that state could mean a second presentation without a first confirmation on record. We address this in two ways. First, we create the payment record before calling the gateway and pass a stable idempotency key with the request; if the gateway already accepted the presentation, a retry with the same key gets a rejection rather than a second acceptance. Second, webhook events from the gateway carry that same key, so even when the sync response is lost, the incoming event allows us to reconcile the state and populate the missing confirmation. A reconciliation job handles any remaining unresolved records where neither the sync response nor the webhook arrived within the expected window.
Abnormal fee changes. When a fetched amount exceeds the allowed threshold relative to the previous amount for that instalment, in absolute terms or as a percentage, the instalment is flagged and held before presentation. The mandate does not go out. Ops is notified and reviews the case. The validation fires before money moves, not after.
Unexpected volume spikes. Consumer task scaling handles the system-level problem: more instalments in the queue means more consumers. The 1hr SLA math holds as long as we can provision workers fast enough, which task scaling enables. What does not scale on the same curve is the partner coordination side. If a bulk reschedule doubles the day’s volume after predictions have run, our consumers will process the additional instalments, but the partner’s system may not have been told to expect 2x the inbound call volume. That is where the operational stress lives. The engineering handles its half automatically; the partner side requires the human layer.
Operational Visibility
After each processing cycle completes, reports go out. Internal ops gets a view of everything that ran: what was processed, what was retried, what failed and why. Partner-facing reports go to the businesses themselves, showing which instalments were processed, which were flagged, and the reason for any flag.
The partner reports serve a function beyond transparency. They are how data errors on the partner’s side surface to the people who can fix them. When an amount comes back flagged for an abnormal increase, the partner’s team may genuinely not know there is an error in their system. The report is what tells them. This creates a feedback loop that gradually improves data quality on the partner side, not because we forced it, but because the report makes the problem visible.
Alerts on the scheduler are the early-warning layer. A scheduling failure before the fetch window opens is recoverable. A scheduling failure discovered after the presentation window has closed is not. The alert placement is deliberate: catch problems at the stage where the most options still exist.
The question we ask before writing any service
Before we write any service at Jodo, we name the specific outcomes that constitute failure: not performance targets, not SLOs, but the conditions under which the service has simply not done its job. We call this the survival instinct for a service. You write down the two or three things that, if they happen, mean the service has failed at its purpose. Everything else is noise.
For this pipeline, those conditions are:
- Completing the wrong financial action is failure: an incorrect amount or a duplicate debit.
- Missing the collection window is failure: valid scheduled work must reach its outcome within the deadline.
- Acting outside the permitted window is failure: no communication or presentation after 7PM.
Everything else (instalments retrying, ops getting paged for a bulk reschedule spike, latency within acceptable bounds, slow partners) is expected noise that the system handles or escalates. The 1hr fetch SLA is a derived constraint from these conditions, not an arbitrary target: if we cannot complete the fetch within that window, the downstream stages cannot reliably meet the 7PM cutoff.
When these conditions conflict, correctness wins. We would rather miss a collection than present a debit for an amount we cannot validate.
This framing changes how you build. You do not design for average load with a margin. You design for the failure conditions and reason backwards. The outbox: because a scheduling failure should not mean lost work. The idempotency at presentation: because a network ambiguity or a retry should not mean a duplicate debit. The validation layer before money moves: because we cannot trust that the amount returned by an external system is correct. The task scaling: because the 1hr SLA has to hold at 1,000 instalments and at 60,000 without us manually intervening between them.
Each of those mechanisms is in the system because one of those three failure conditions was at risk without it.
The pipeline looks different today than it did when we first shipped it in 2023. The architecture was mostly right from the start. The validation layer, the specific alert placements, the idempotency guarantees: those evolved from what actually happened. That is a less satisfying story to tell than “we designed it correctly from first principles.” But it is a more honest account of how systems like this actually get built.
You cannot design for all the ways a network of external systems will surprise you. You can design a system that learns from each surprise and builds in a guard. That is what this pipeline is: an accumulation of things that happened once and were made impossible to happen again. If you are building anything where a mistake means real money moves incorrectly, I think that is the most honest description of what the engineering job actually is: not designing for correctness from first principles, but building the memory into the system.
Thanks to Jaychandra and Rishav for their significant contributions to this pipeline, and to Rudransh for improvements made in a later phase.
Originally published at jodo.in.
