
Deadlocks - The Invisible Trap Slowing Down Your Web App
The other day, I was re-visiting my notes on operating systems when “deadlocks“ caught my eye. To illustrate, imagine a traffic jam where each car is stuck, waiting for the other to move; no one can go forward, and nothing progresses. That’s essentially what happens inside your system during a deadlock.
Given the reliance of web applications on underlying operating systems, I wondered if this often-overlooked issue issue quietly affects performance of my state-of-the-art Todo app.
Spoiler alert - it can and it does, the most commonly used OS for web servers(Linux) has made many developers and sysadmins tear their hair out over the “deadlock issue.”
What is Deadlock
In operating systems, deadlocks occur when two or more processes are stuck waiting for each other to release resources, creating a cycle of dependency where none of the processes can proceed.
Rather than diving into the technical details of deadlock conditions or solutions, let’s take a closer look at deadlocks from the perspective of software developers.
Why Should a Software Developer Care?
You might be thinking: I’m not building kernels, why should I care ?
-
Every web server relies on an OS - Understanding how your chosen OS handles resources like memory and file locks can prevent unwanted surprises. A web server stuck in a deadlock could slow down your entire application, affecting user experience and system performance. Example - A web server running on Linux might experience a deadlock when multiple users access the same file, but the application is responsible for handling it and gives a granular control. While Windows is more proactive and prevents deadlock so is better for enterprise apps.
-
Your code might create deadlocks - You don’t need to touch OS internals to encounter deadlocks. Writing bad or careless queries/code can lead to deadlocks in higher-level programming too.
Real-World Deadlock Examples in Development
Example 1 - Simultaneous updates in MySQL
Two transactions try to transfer money between the same accounts but in opposite directions. Each one locks a row the other needs.
-- Transaction A START TRANSACTION; UPDATE accounts SET balance = balance - 100 WHERE account_id = 1; -- Locks account 1 UPDATE accounts SET balance = balance + 100 WHERE account_id = 2; -- Locks account 2 COMMIT; -- Transaction B START TRANSACTION; UPDATE accounts SET balance = balance - 50 WHERE account_id = 2; -- Locks account 2 UPDATE accounts SET balance = balance + 50 WHERE account_id = 1; -- Locks account 1 COMMIT;
Result is that both transactions end up waiting on each other - creating a circular wait and triggering a deadlock.
Example 2 - Inconsistent Lock Order in a Node.js Web AP
A REST API allows money transfers. If two requests try to transfer in opposite directions, but lock accounts in different order, a deadlock can occur.
const mysql = require('mysql2/promise'); async function transfer(fromId, toId, amount) { const connection = await mysql.createConnection({ /* connection config */ }); try { await connection.beginTransaction(); // ⚠️ Risky: Lock order depends on input await connection.query( 'UPDATE accounts SET balance = balance - ? WHERE account_id = ?', [amount, fromId] ); await connection.query( 'UPDATE accounts SET balance = balance + ? WHERE account_id = ?', [amount, toId] ); await connection.commit(); } catch (err) { await connection.rollback(); console.error("Transaction failed:", err); } finally { await connection.end(); } }
Safer Version – Always Lock in Consistent Order
const ids = [fromId, toId].sort(); // Lock accounts in consistent order await connection.query( 'UPDATE accounts SET balance = balance - ? WHERE account_id = ?', [amount, ids[0]] ); await connection.query( 'UPDATE accounts SET balance = balance + ? WHERE account_id = ?', [amount, ids[1]] );
How Deadlocks Surface in Popular Web Stacks
Deadlocks can emerge not just from OS-level behavior but also due to how the web server (or runtime) handles concurrency, threads, and resources. Let’s take a look at where deadlocks might appear in different web server environments and runtimes:
Apache HTTP Server
- Platforms: Linux, Windows
- Issue: When using the prefork MPM model, each request spawns a separate process. Under high load, this can lead to resource exhaustion and potential deadlocks.
- Recommendation: Use worker or event MPM instead, which are more efficient and better at handling concurrency without locking conflicts.
Nginx
- Platform: Primarily Linux (limited Windows support)
- Strength: Nginx uses an event-driven, non-blocking architecture that scales exceptionally well with high traffic.
- Caution: On Windows, its performance degrades due to poorer support for asynchronous I/O, which can indirectly contribute to resource contention or latency under load.
Internet Information Services (IIS)
- Platform: Windows-only
- Behavior: Thread and connection pooling in IIS can lead to contention if not configured properly under heavy load.
- Recommendation: Tweak settings like application pool recycling, idle timeouts, and max concurrent requests to avoid contention that could cause application-level deadlocks.
NodeJs (Runtime, Often Used as Web Server)
- Platform: Cross-platform
- Strength: Its single-threaded event loop architecture inherently reduces the chances of deadlocks in traditional sense.
- Caveat: While Node.js avoids classic OS-level deadlocks, poor use of asynchronous code, callback hell, or unoptimized database interactions can still cause the application to “appear stuck” or non-responsive ( I like to call it kind of a logical deadlock )
Note - Node.js is’nt a web server by itself but in many modern stacks( like with Express) it functions as one by handling HTTP requests directly.
Apache Tomcat
- Platform: Java-based, cross-platform
- Issue: When deployed on Windows under high concurrency, its thread pool may become saturated. If threads are waiting on shared resources or locks, this can resemble a deadlock.
- Recommendation: Tune maxThreads in server.xml and ensure thread-safe resource access in your Java code.
Choosing the Right OS & Server for Your Application
Use Case | Recommended Stack High Traffic Web Apps | Linux with Apache (worker/event MPM) or Nginx - optimized for high concurrency and performance Real-time Apps | Node.js on Linux - non-blocking, event-driven architecture ideal for low-latency interactions Enterprise Apps | IIS on Windows - reliable for enterprise workloads with proper tuning of app pools and threads
Best Practices for Preventing deadlocks
1. Consistent Locking Order
Always acquire locks in a predefined order.
-- Always lock account 1 before account 2
START TRANSACTION;
UPDATE accounts SET balance = balance - 100 WHERE account_id = 1; -- Lock account 1
UPDATE accounts SET balance = balance + 100 WHERE account_id = 2; -- Lock account 2
COMMIT;
2. Set Transaction Timeouts
Set a timeout for your transaction to avoid waiting indefinitely.
SET innodb_lock_wait_timeout = 5; -- Timeout after 5 seconds
3. Enable Deadlock Detection
Enable deadlock detection in your database settings, allowing it to roll back a transaction when a deadlock is detected.
-- MySQL automatically detects deadlocks; you can check the status with:
SHOW ENGINE INNODB STATUS;
4. Minimize Locking Scope
Lock only the resources that are absolutely necessary, and avoid holding locks longer than needed.
-- Instead of locking the entire table, lock only the rows that need updating
START TRANSACTION;
UPDATE accounts SET balance = balance - 100 WHERE account_id = 1; -- Lock only the required row
COMMIT;
5. Use Read Committed Isolation Level
In databases that support transaction isolation levels, using the “Read Committed” isolation level can help prevent deadlocks by reducing unnecessary locks.
-- Set isolation level to Read Committed to reduce lock contention
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
Monitoring and Resolving Deadlocks
Tools like MySQL Workbench and pgAdmin provide visual representations of database activity and can help identify deadlock situations by showing locks and transactions. Some other Application Performance Monitoring tools are Datadog and NewRelic. If somebody wants I can go into deep-dive of these tools.
Notable Case Studies
1. Apache Deadlocks (2002)
In 2002, Apache faced deadlock issues primarily due to its multi-threaded handling of requests.
Cause - The threading model of Apache, combined with Linux’s resource management at the kernel level, created a situation where threads were waiting on each other indefinitely.
2. NFS (Network File System) deadlocks
NFS experienced deadlock situations when multiple machines accessed the same files across a network. Cause - If one machine crashed or had network issues, locks that were supposed to be released remained held, leading to other machines waiting indefinitely.
3. MySQL 5.0 Deadlocks
MySQL databases, especially version 5.0 released in 2005, faced significant deadlock scenarios during transaction processing.
Cause - Deadlock arose from the way transactions acquired locks on rows. Ex- Transaction A locked Row 1 and then tried to lock Row 2 while Transaction B had already locked Row 2 and was attempting to lock Row 1, they would block each other indefinitely.
4. Docker Deadlocks (2016)
In 2016, Docker containers exhibited deadlock issues due to resource contention. Cause - When one container became resource-heavy, it could monopolize CPU cycles or memory, leaving other containers starved for the same resources eventually leading to indefinite wait.
Conclusion
A proactive approach to deadlock prevention improves software reliability, enhances user experience, and protects organizations from costly downtime.
Additional Links
https://github.com/moby/moby/issues/28236
https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1565744
https://serverfault.com/questions/890162/how-to-solve-nfs-cross-mounting-ordering-dead-lock