The Reality of Database Updates

When a celebrity posts a photo and receives 100,000 likes in minutes, a critical question arises:

Would Instagram really hit its database 100,000 times in one minute just to update a like counter?

The answer is simple: no — and it shouldn’t.

This is where batch processing becomes essential.

In this guide, we’ll explore:

• The pitfalls of naïve database updates at scale

• How large systems manage massive write traffic

• A practical batch-processing architecture

• Implementation using Redis and worker-based batching

This is not just theory; it’s how production systems operate.

The Naïve Approach and Its Limitations

A straightforward method to handle likes involves:

1. User clicks Like

2. Backend increments likes_count in the database

3. Immediate database write

Challenges at Scale

If 100,000 users like a post within a minute, this results in:

• 100,000 database write operations

• Heavy lock contention on the same row

• Increased latency for all users

• Risk of database throttling or outages

Relational databases are not designed for extremely high-frequency writes on the same record. If Instagram followed this approach, their database would struggle to cope.

How Large Systems Address the Problem

Big systems adhere to a key principle:

User experience must be fast; database writes can be delayed.

A like doesn’t need to be immediately stored in permanent storage. A short delay is acceptable and invisible to users.

The strategy involves:

• Quickly accepting likes

• Storing them in a fast in-memory system

• Persisting them to the database in batches

Understanding Batch Processing

Batch processing involves:

• Collecting multiple events over time

• Processing them together as a group

• Dramatically reducing system load

Instead of:

100,000 likes → 100,000 database writes

We achieve:

100,000 likes → 1 Redis counter → 1 batched database write

This results in a 100,000x improvement in write efficiency.

Architectural Overview

Here’s the high-level architecture:

1. User clicks Like

2. Backend updates Redis (fast, in-memory)

3. A background worker runs periodically

4. Worker reads accumulated likes from Redis

5. Worker updates the database in batches

Why Choose Redis?

Redis is ideal for this scenario because:

• It’s in-memory, making it extremely fast

• Supports atomic operations (INCR)

• Can handle millions of operations per second

• Temporary data storage is acceptable

A database ensures durability, while Redis provides speed.

Storing Likes in Redis

When a user likes a post:

redis.incr(`post:likes:${postId}`)

Benefits include:

• O(1) operation

• No database lock

• Immediate user response

At this stage:

• The UI can display the updated count

• The database remains untouched

The Role of the Batch Worker

A background worker runs every few seconds or minutes.

Worker Responsibilities

1. Fetch all like counters from Redis

2. Aggregate them

3. Write updates to the database

4. Reset Redis counters

Pseudo-flow:

for each postId in redisKeys:
  likes = redis.get(postId)
  UPDATE posts SET likes_count = likes_count + likes
  redis.del(postId)

This reduces thousands of updates to one update per post per interval.

Determining Batch Size and Frequency

This is a design decision:

• Every 5 seconds → more real-time, more database writes

• Every 1 minute → fewer database writes, slight delay

Production systems adjust based on:

• Traffic

• Database capacity

• Acceptable data freshness

Instagram doesn’t require millisecond-accurate likes, nor do most apps.

Handling Edge Cases

Redis Crash

If Redis crashes, likes in memory may be lost.

Mitigations:

• Enable Redis persistence (AOF/RDB)

• Accept minor data loss for non-critical metrics

Likes are eventually consistent, not financial transactions.

Worker Failure

If a worker crashes mid-batch:

• Redis data remains intact

• The next worker run continues processing

This ensures the system is fault-tolerant.

Duplicate Updates

Workers must be:

• Idempotent

• Or carefully delete Redis keys only after successful database writes

This prevents double-counting likes.

Why This Pattern is Industry Standard

This approach is used for:

• Like counters

• View counts

• Follower counts

• Analytics events

• Notifications

Any system with high write frequency employs batching. Similar patterns are found in:

• Instagram

• Twitter

• YouTube

• Netflix analytics

Key Takeaways

• Databases shouldn’t handle extremely high-frequency writes

• Redis absorbs traffic spikes

• Batch workers ensure system stability

• Eventual consistency is acceptable for metrics

For any application that might go viral, batch processing is essential.

Final Thought

Next time you see a post jump from 10K to 100K likes instantly, remember:

Behind the scenes, no database is being overwhelmed. A smart batching system is efficiently managing the load.

For students preparing for backend interviews or system design rounds, this pattern is invaluable. Understand it, implement it, and discuss it with confidence.

Implementation

Suppose you are sending a video on WhatsApp to your friend. The network you are using is the same network that many other people are using at the same time. There is no fixed path that guarantees your video will travel directly and reliably to the right person. If a disturbance happens in the middle while sending one big chunk of data, the entire transfer can fail and you would have to send the whole video again

It’s similar to talking on an old wired telephone with a private, end-to-end wire connection. If that wire gets disturbed, the whole communication breaks.
But in reality, networks are shared—many users, many routes, lots of traffic.

This is where packets come in.

Instead of sending the entire video as one big block, the network breaks the video into many small chunks, called packets.
Each packet contains:

1. A small part of the video

2. Metadata that describes how and where it should be delivered

Think of it like sending a parcel to your friend. You attach important details like the address, sender, receiver, and sometimes weight. Without metadata, the parcel wouldn’t know where to go. The same idea applies to packets.

Metadata inside a packet

Every packet includes three major components:

1. Header

This contains control information such as:

• Source address

• Destination address

• Sequence number

• Protocol

• TTL

• Checksum

The header tells the network where the packet came from, where it needs to go, and how it should be handled.

2. Payload

This is the actual data, the small part of your video being transmitted.

3. Trailer

This usually contains error-checking information, such as CRC, to verify whether the packet was delivered correctly.

By breaking the video into packets, the network can send each small chunk independently through different routes. If one packet gets lost, only that packet needs to be resent—not the entire video. This makes communication fast, reliable, and scalable, even when millions of people share the same network.

IPV4 Packet Headers

Interview Question

1. What is a packet?

Explain the definition and why networks use packets instead of sending data as one large block.

2. What are the components of a packet?

Expected parts: Header, Payload, Trailer.

3. What information is stored inside a packet header?

Source IP, Destination IP, TTL, Protocol, Checksum, Sequence Number, etc.

4. What is the payload in a packet?

Explain that it's the actual user data (video chunk, message, file part).

5. What is the role of the trailer in a packet?

Error detection, usually CRC or checksums.

6. Do packets always take the same path to reach the destination? Why or why not?

Explain dynamic routing + path independence.

7. What happens when a packet gets corrupted or lost during transmission?

TCP → retransmits
UDP → packet lost permanently

8. What is TTL (Time-To-Live) in a packet and why is it important?

Prevents infinite loops.

9. What is packet fragmentation? Why does it occur?

Triggered when packet size > MTU; packet is split into smaller fragments.

10. Can packets arrive out of order? How is this handled?

Yes. TCP reorders using sequence numbers.

• Vertical scaling - scale up

• Horizontal scaling - scale out

Vertical scaling - scale up

So one of the possibilities is that if the server is getting more requests and it is getting overloaded, you could monitor and see that your CPU usage is going to 90%. So what you could do is think from first principles. If my CPU and storage are insufficient for my server, then one of the things we can do is get a bigger server with a more powerful CPU and storage, right?

Exactly, this is known as vertical scaling, where we just modify the server and get a bigger server that can handle the compute and complexities of your server. There are advantages to this method, such as being able to migrate to a bigger server without having to change the coding part.

Migration from a small server to a bigger server is not a big deal; you just have to deploy your application on the bigger compute machine. This machine could be any cloud provider compute, whether it may be EC2 or a droplet, right? But if you think from a cost perspective, then a bigger server equals more cost, right? Also, keep in mind that if your application grows even bigger, you will have to migrate to an even bigger machine. And if the question arises in your mind that at some point your server might not handle the load and fails, then your whole application will crash. In this case, you might think, why am I just using one bigger server, and why can't I run the same application on 5 small servers rather than one big server? This is where horizontal scaling comes in.

Horizontal scalling - scale out

Well, instead of running the server on one bigger machine, we can run the same application on a bunch of small machines with the same configuration. In this setup, we need to figure out that every machine will have a different IP address, and your domain can only point to one machine at a time. This is where a new term comes in, called Reverse Proxies.

II will try to break down reverse proxies in simple terms. They are basically the gateway to your application. Think of it as your gate; it says that if you want to reach your application, I am the gate for it. Without me, you can't access it. (Here, we can access the application, but at a time, we can only point to a specific IP of the machine. We can't cater to every machine.) This tool allows you to fix this problem. It says that it will take all your requests coming from users and will transfer or proxy your request to one machine with a lower load, or it can follow some rule of OS scheduling called Round Robin. It basically tells you that the next request coming from your user will be forwarded to machine 1, then to machine 2, then to machine 3, one by one to each machine. This way, the load will be distributed.

Artitecture of Reverse Proxies

Again, this has some challenges and benefits. We had to optimize it as well as manage the complexity it will bring.

Below is the table I have provided; you can check it out.

Below is the Comparison Table in short

Please do like , comment and share

Thanks for reading :)

As a student considering a journey into DevOps, I can relate to your curiosity, as I have been on this path for some time and can offer guidance based on my experience. There are various reasons you might choose to learn DevOps, whether through personal exploration or recommendations from peers.

Common Mistake: A frequent error is choosing DevOps simply to avoid coding, as some of my friends have done, thinking it's the easier path.

Let me dispel this myth: DevOps requires coding expertise, with shell scripting and Python being heavily used for automation.

If you're genuinely curious about how applications are deployed to production, you're in the right place.

Taking First Step

Learning DevOps Tools in the Right Order

There are hundreds of tools in the DevOps ecosystem. The challenge is not learning everything, but learning in the right order so your foundation stays strong.
Here’s a practical learning path that actually aligns with real engineering workflows:

1) Git + GitHub

Start version-controlling every project you build.

Rule: Push all your code to GitHub. Keep every project well-maintained.

Why it matters:

• Collaborating with code becomes easier

• You learn branching, merging, and pull requests

• It builds your portfolio naturally

Key Skills to Learn

• git init, git clone, git status

• git add, git commit, git push

• Branching & Pull Requests

2) Linux Basics + Shell

Linux is the backbone of servers and cloud machines.

When you work inside the cloud, you interact through a terminal, so Linux knowledge is non-negotiable.

Recommendation: Learn a terminal editor like vim or nano.
When you SSH into a server, you won’t have VS Code waiting for you.

Key Concepts

• File system navigation (cd, ls, rm, mv, cp)

• Permissions & users

• System services & processes

• Networking basics (ping, curl, netstat)

3) Choose a Cloud Provider

After Linux, pick one cloud to start with.

For beginners, AWS is highly recommended because it’s widely used in the industry and has tons of documentation.

Use your Linux skills and start doing practical exercises, like:

• Creating EC2 instances

• SSH into instances using keys

• Hosting simple apps

The goal is to connect Linux knowledge with real cloud servers.

4) Docker (Containerization)

Now that you can deploy on the cloud, learn how to deploy efficiently with containers.

Why Docker?

• Reproducible environments

• Works the same on local + cloud

• Makes CI/CD sooo much easier

Learn

• Dockerfile

• Images, containers, volumes, networks

• Tagging & pushing images to Docker Hub

5) CI/CD (GitHub Actions)

Once you know Docker, automate builds and deployments.

CI/CD is not theory. You must build pipelines that test and deploy your apps automatically.

Start with GitHub Actions, then explore others later.

Important Concepts

• Build pipelines

• Deploy pipelines

• Secrets management

6) Terraform (Infrastructure as Code) (this for now you can skip and can jumps back later after learning )

Instead of manually creating servers on AWS, automate it.

Terraform teaches you:

• Declarative infra provisioning

• Reusability with modules

• Working with multi-cloud setups

You should be able to:

• Create VPC, EC2, IAM

• Use Terraform variables & outputs

• Store state securely (S3 + DynamoDB)

7) Kubernetes (Orchestration)

Once you understand Docker, learn how to scale containers.

Kubernetes = Running containers at scale.

Focus on:

• Pods, Deployments, ReplicaSets

• Services & Ingress

• ConfigMaps & Secrets

• Helm (bonus)

Make sure you run apps locally using kind kubectl or Kubernetes on cloud (EKS/GKE).

8) Observability (Prometheus + Grafana)

If you deploy apps, you must monitor them.

Observability includes:

• Metrics

• Logs

• Tracing

Prometheus helps collect metrics
Grafana visualizes them

Learn how to:

• Monitor applications

• Create dashboards

• Set up alerts

9) DevSecOps Basics (Trivy + Vault)

Security should never be an afterthought.

Start with simple tools:

• Trivy → Scan vulnerabilities

• Vault → Manage secrets

Use Trivy to scan:

• Docker images

• Kubernetes workloads

• Repositories

Use Vault to store secrets securely.

TLDR

The article provides a comprehensive guide for learning DevOps, starting with understanding the reasons for choosing this path. It emphasizes the importance of coding skills, particularly in shell scripting and Python. The learning journey includes mastering Git and GitHub for version control, Linux basics, choosing a cloud provider like AWS, learning Docker for containerization, and setting up CI/CD pipelines with GitHub Actions. It also covers advanced topics like Terraform for infrastructure as code, Kubernetes for container orchestration, observability with Prometheus and Grafana, and DevSecOps basics with tools like Trivy and Vault.

I am Suraj Vishwakarma from Bangalore, currently pursuing a B.Tech in Computer Science from Lovely Professional University. I have always been interested in playing with computers.

I got my first PC when I was in 6th grade. I had asked my father for one when I was in 3rd grade, but he eventually gave me a laptop. No one in my family had imagined that it would be the first computer for our entire family, and perhaps no one knew how to use it or had even seen a laptop before.

I learned to use Paint when I was in school. They took the whole class to the computer lab, and the very first task I remember was being asked to type our names. I was unable to find where the letters were located on the keyboard.

From Town to Big City

My family moved from Garhwa to Bangalore, and our whole life changed.