Why we can't overlook non-technical requirements

Building a platform that can scale up to serve a billion global customers for any client isn’t easy. And when those clients are reputable banks, and the platform is designed to handle real money, it’s near impossible to do so without focusing on customer security. 

• We are dealing with a real-life person 
• There are no illegal activities 
• Occupation-related risks are mitigated (i.e. bribery prevention) 

This is managed by building a Screening Engine – a system that reacts to events read from a Kafka Topic. And for each event, it conducts mandatory security checks. The goal is to ensure the process is smooth and transparent, but also set up to react in a timely fashion if misuse is detected. 

Because of this, it’s crucial the system is highly reliable because onboarding should be fast and should work every single time for every customer. If something goes wrong, it may cause reputational damage, and regulatory penalties could also be applied.  

In addition to that, the exposure to exploitation and scams could drive customers off the platform. That’s why this needs to withstand just about any failure scenario we throw at it.  

 

We went ahead and implemented everything that was needed with very few issues. By the end of it, we felt relieved that everything looked good and safe.

The problem with Kafka Consumers

The way Spring Kafka deals with this sort of error by default is through a handler called SeekToCurrentErrorHandler which logs consumption errors that occur in the container layer and then skips to the next offset. However, this handler cannot cope with deserialisation exceptions and will enter an infinite retry loop. To fix this, consumers can use a different handler called ErrorHandlingDeserializer. This would be configured to deserialize keys and values by delegating the processing to the original deserializers being used. In our case, these were UUID for Keys and Avro for values.  

To override the default behaviour of just logging the error, a custom implementation of the Spring Kafka interface ConsumerRecordRecoverer can be created.

This should be plugged back into the SeekToCurrentErrorHandler when configuring the container factory. 

By over-relying on what Avro provides when it comes to payload structure, we had overlooked the fundamental issue of what happens if deserialisation itself doesn't succeed. It’s still possible to end up with incompatible producer/consumer pairing despite using Avro and the Confluent Schema Registry, e.g., a new producer that ignores schema checks. We chose to sidestep the non-deserializable records using a tried and tested approach of placing them onto an error topic before triggering an alert to prompt investigation.

 

 

The actual fix we made to resolve the issue was straightforward to implement, thanks to the awesome team at Spring (reference documentation here). However, our lesson in humility could have been very expensive, and is something any company could experience for themselves. In fact, we’ve come across many others discussing the same issues.

 

Final thoughts and learnings from the experience

Once everything was back to normal, we decided to ensure that we wouldn't face this again even if we changed the consumer deserialisation configuration. We couldn't test this by placing poison pills on our onboarding topic because that was being used by multiple teams with different backlog priorities and velocities. That could block consumers that have not yet implemented a defense against poison pills.