What Building My Notification System Taught Me About Distributed Systems
I started this project with a very innocent goal.
"Build a notification system."
Easy.
Receive a request.
Save it.
Send it.
Done.
...
About three weeks later I was reading about Kafka offsets, wondering why the same notification arrived twice, and questioning my life choices.
Turns out, I wasn't building a notification system.
I was accidentally learning distributed systems.
"Just send the notification"
The first version looked something like this.
Client
↓
Spring Boot API
↓
Send Email
↓
Return Response
Looks perfectly reasonable.
Until...
The email provider decides to take 5 seconds to respond.
Now your API is also taking 5 seconds.
Your users don't know the email provider is slow.
They only know your application is slow.
Congratulations.
You've inherited someone else's problems.
Kafka to the rescue
This is where Kafka entered the chat.
Instead of asking the API to do all the work immediately, it now just says
"Hey Kafka, here's a notification. I'll let someone else handle it."
Client
↓
Spring Boot API
↓
Publish Event
↓
Kafka
↓
Notification Consumer
│
├── PostgreSQL
└── WebSocket
Now the API responds almost immediately.
The consumer processes notifications whenever it gets them.
It's basically the difference between
"Do this right now."
and
"Put it on my desk. I'll handle it."
Much less stressful.
Kafka doesn't forget...
...which is both amazing and occasionally annoying.
Let's say the consumer receives a notification.
It saves it to the database.
Just before it tells Kafka
"Yep, I processed this."
the application crashes.
Now Kafka thinks
"Hmm... nobody processed this."
So after restart...
It sends the exact same notification again.
Kafka isn't broken.
It's actually doing exactly what it's supposed to do.
It would rather send the same message twice than lose it forever.
That's called at-least-once delivery.
Which sounds nice...
...until your user gets
"You have a new message."
twice.
Redis became the bouncer
My solution was surprisingly simple.
Every notification gets a unique eventId.
Before processing anything, the consumer asks Redis
"Have you seen this person before?"
If Redis says
"Yes."
The notification politely leaves.
If Redis says
"No."
The consumer processes it and Redis remembers that ID for a while.
Think of Redis as the security guard checking guest passes before anyone enters the party.
No pass?
Come in.
Already inside?
Nice try.
Failure isn't a bug
One thing distributed systems taught me is this:
Failures are expected.
Servers crash.
Networks disappear.
Kafka brokers restart.
Docker containers have bad days.
The question isn't
"What if something fails?"
The question is
"What happens when it fails?"
That tiny difference completely changes how you design software.
The Dead Letter Topic
Imagine a delivery person trying to deliver a package.
They knock.
Nobody answers.
They try again.
Still nothing.
Eventually they stop trying and leave a note saying
"Please collect your package."
That's basically what a Dead Letter Topic is.
Instead of retrying forever...
Kafka eventually says
"I'm putting this over here so someone can investigate."
Nothing gets silently lost.
Future You will appreciate that.
Correlation IDs are underrated
Without correlation IDs, debugging looks like this.
Received request
Saved notification
Kafka message processed
WebSocket sent
Cool.
But...
Which request?
Which notification?
Which user?
Nobody knows.
A Correlation ID is basically a tracking number.
Every log generated by that request carries the same ID.
Now debugging feels less like detective work and more like following breadcrumbs.
Load testing humbled me
The application worked perfectly.
With one user.
So did every project I've ever built.
The real questions begin when there are hundreds of requests arriving together.
Running load tests taught me something tutorials rarely mention:
Software behaves very differently when everyone wants to use it at the same time.
Computers are funny like that.
Things tutorials rarely tell you
Building this project taught me that...
- Kafka isn't difficult. Understanding why duplicates happen is.
- Redis isn't just a cache.
- Logs become your best friend somewhere around your third bug.
- "Works on my machine" has a surprisingly short lifespan.
- Distributed systems aren't hard because of code.
- They're hard because computers refuse to fail politely.
What's next?
This project still has plenty of room to grow.
Some things I'd love to add next are:
- Email and SMS notification channels
- Grafana dashboards
- Kubernetes deployment
- Horizontal consumer scaling
- Authentication
- Better observability
Because every finished side project eventually becomes another unfinished side project.
Final thoughts
I thought I was building a notification system.
Instead, I learned why companies invest so much in message queues, retries, circuit breakers and observability.
None of those features make your application look cooler.
They just make sure it keeps working when everything else decides not to.
And honestly...
That's much more interesting.