Systems ThinkingNovember 17, 2026

Building a Deadlock Detector Mental Model for Async Locks in Node

E

Written by

Elena Holos

The bug that wouldn’t explain itself

I once stared at a “totally healthy” Node.js service that suddenly stopped responding—CPU wasn’t pegged, there were no crashes, and the logs looked boring. The only clue was a handful of requests that never completed. After a couple hours of instrumenting everything, I realized I didn’t actually understand the mental model for how my async locks were interacting.

This post is about a very specific failure mode I hit: a deadlock caused by mixing “await inside a critical section” with “lock acquisition order that changes under load.” The surprising part was that my code looked reasonable until I mapped it to the exact mental model of “who is holding the lock” and “what promise is waiting on what.”

I’ll show the failing code, then walk through how to reproduce it, detect it, and avoid it using a stricter lock mental model.


The mental model I needed: “locks create queues of waiting continuations”

In JavaScript/Node, await doesn’t block the thread. Instead, it pauses the current async function and resumes it later (when the awaited promise resolves). If you do that while holding a lock, you’ve effectively said:

“I’m going to hold exclusive ownership, but I’m allowing my continuation to get parked and only later re-enter the critical section.”

That means other code that needs the lock queues up behind that parked continuation. If the continuation itself is waiting on something that indirectly needs the same lock, you’ve created a deadlock.

To make this concrete, I used a minimal lock:

  • A Lock that queues waiters.
  • A critical section that holds the lock and performs an await.
  • A second function that acquires a lock and triggers a callback that tries to acquire the same lock (directly or indirectly).

Reproducing the deadlock with step-by-step code

1) A tiny async lock implementation

// lock.js class Lock { constructor() { this._locked = false; this._queue = []; } acquire() { if (!this._locked) { this._locked = true; return Promise.resolve({ release: () => this._release() }); } return new Promise((resolve) => { this._queue.push(resolve); }).then(() => ({ release: () => this._release() })); } _release() { if (this._queue.length > 0) { const nextResolve = this._queue.shift(); nextResolve(); } else { this._locked = false; } } } module.exports = { Lock };

What this does (and why it matters):

  • acquire() either resolves immediately (lock is free) or returns a promise that resolves later (lock is busy).
  • The “waiting” isn’t a thread—it's your function’s continuation parked behind the unresolved promise.
  • If that parked continuation depends on work that requires the same lock, no one can make progress.

2) The deadlocking interaction

Here’s the deadlock: taskA holds Lock A, then awaits something that triggers taskB, which tries to acquire Lock A. Since Lock A is still held by taskA, taskB never runs. Meanwhile taskA is waiting for taskB to do work—classic cycle.

// deadlock-demo.js const { Lock } = require("./lock"); const lockA = new Lock(); function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); } async function taskB() { const release = await lockA.acquire(); try { // This will never be reached if lockA is held by taskA. await sleep(10); return "B finished"; } finally { release.release(); } } async function taskA() { const release = await lockA.acquire(); try { // Hold lockA, then wait for an operation that calls taskB. // taskB will block on lockA -> deadlock cycle. const result = await taskB(); return `A got: ${result}`; } finally { release.release(); } } async function runOnce() { console.log("Starting deadlock demo..."); const p = taskA(); // Add a timeout so the demo ends even though it's deadlocked. const timeout = new Promise((_, reject) => setTimeout(() => reject(new Error("Timed out: likely deadlock")), 200) ); try { const res = await Promise.race([p, timeout]); console.log("Result:", res); } catch (e) { console.error("Error:", e.message); } } runOnce();

Run it:

node deadlock-demo.js

What you’ll see: it prints “Starting deadlock demo...” and then times out.


Detecting the deadlock mental model in practice

The first time I encountered this, I didn’t have “lock ownership” visibility. So I built a tiny piece of diagnostic logic: track which async callsite acquired a lock, and how long a lock has been held.

That gives you a direct mapping from the mental model (“who owns the lock?”) to evidence (“why is the owner stalled?”).

3) Instrumented lock (ownership + hold time)

// debug-lock.js class DebugLock { constructor(name) { this.name = name; this._locked = false; this._queue = []; this._owner = null; // token identifying owner this._acquiredAt = null; } acquire(ownerToken) { if (!this._locked) { this._locked = true; this._owner = ownerToken; this._acquiredAt = Date.now(); return Promise.resolve({ release: () => this._release() }); } return new Promise((resolve) => { this._queue.push(resolve); }).then(() => { this._locked = true; this._owner = ownerToken; this._acquiredAt = Date.now(); return { release: () => this._release() }; }); } getStatus() { return { name: this.name, locked: this._locked, owner: this._owner, heldMs: this._locked ? Date.now() - this._acquiredAt : 0, queuedWaiters: this._queue.length, }; } _release() { this._locked = false; this._owner = null; this._acquiredAt = null; if (this._queue.length > 0) { const nextResolve = this._queue.shift(); nextResolve(); } } } module.exports = { DebugLock };

4) Use the instrumented lock to observe the cycle

// deadlock-diagnose.js const { DebugLock } = require("./debug-lock"); const lockA = new DebugLock("lockA"); function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); } async function taskB(ownerToken) { const release = await lockA.acquire(ownerToken); try { await sleep(10); return "B finished"; } finally { release.release(); } } async function taskA() { const ownerToken = "taskA"; const release = await lockA.acquire(ownerToken); try { const interval = setInterval(() => { console.log("LOCK STATUS:", lockA.getStatus()); }, 50); try { // Wait for B while holding lockA. const result = await taskB("taskB"); return `A got: ${result}`; } finally { clearInterval(interval); } } finally { release.release(); } } async function runOnce() { console.log("Starting deadlock diagnosis..."); const p = taskA(); const timeout = new Promise((_, reject) => setTimeout(() => reject(new Error("Timed out: likely deadlock")), 250) ); try { const res = await Promise.race([p, timeout]); console.log("Result:", res); } catch (e) { console.error("Error:", e.message); console.log("Final LOCK STATUS:", lockA.getStatus()); } } runOnce();

Expected behavior:

  • You’ll repeatedly see locked: true and owner: "taskA" while heldMs increases.
  • queuedWaiters will be 1 (for taskB), because taskB is waiting to acquire lockA.

This instrumentation makes the mental model explicit: taskA is parked holding the lock, taskB is queued waiting for it, and taskA is waiting for taskB. Nothing can progress.


Fixing the mental model: “no lock across awaits that can re-enter the same lock”

There are several ways to fix deadlocks, but the mental model-driven one is simple:

Don’t hold a lock while awaiting something that might call code that tries to acquire that same lock (directly or indirectly).

In practice, I enforce this by splitting work into:

  1. A short critical section that updates shared state.
  2. The awaited operation outside the lock.

5) A corrected version

// deadlock-fixed.js const { Lock } = require("./lock"); const lockA = new Lock(); function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); } async function taskB() { const release = await lockA.acquire(); try { await sleep(10); return "B finished"; } finally { release.release(); } } async function taskA_fixed() { // Critical section: only synchronize shared state. const release = await lockA.acquire(); try { // Update some shared thing synchronously (no awaits here). // Imagine: mark "in progress" in memory. } finally { release.release(); } // Await outside the lock, so taskB can acquire lockA if it needs it. const result = await taskB(); return `A got: ${result}`; } async function runOnce() { console.log("Starting fixed demo..."); const p = taskA_fixed(); const timeout = new Promise((_, reject) => setTimeout(() => reject(new Error("Timed out")), 250) ); try { const res = await Promise.race([p, timeout]); console.log("Result:", res); } catch (e) { console.error("Error:", e.message); } } runOnce();

Run:

node deadlock-fixed.js

Now it completes.


The “mental model checklist” I now follow for async locks

After building and diagnosing this, I now treat each await inside a lock as a red flag to reason about re-entrancy. My checklist is:

  1. Ownership: Which code path acquires the lock, and what exact async work happens while it’s owned?
  2. Continuations: Where will the function resume after await? Is it safe that other tasks may run in between?
  3. Cycle possibility: Could the awaited operation call something that attempts to acquire the same lock (directly or indirectly)?
  4. Critical section boundaries: Can I shrink the critical section to non-awaiting code, then do the awaited work after releasing the lock?

That mental checklist is what turned a “mysterious hang” into a deterministic bug I could reproduce and fix.


Conclusion

I learned that the key mental model for async lock bugs in Node.js is not “threads,” but promises and continuations: if I hold a lock across an await, I may be parking the lock owner while other work queues behind it—creating a deadlock cycle if the awaited work needs the same lock. By instrumenting lock ownership and hold time, I mapped the symptoms to the cycle, and by shrinking critical sections to exclude re-entrant awaits, I made the system progress again.