Locks, Invariants & Deadlocks

Classic Java Monitor Deadlock

Classic Java Monitor Deadlock: practice a Java concurrency bug with symptoms like Requests hang, CPU not maxed, Threads blocked. Inspect runnable code,...

  • Deadlock Patterns
  • Deadlock
  • Java
  • Beginner

Production symptoms

  • Requests hang
  • CPU not maxed
  • Threads blocked

Failure scenario

Code

Java example
public class TransferService {
    private final Object accountLock = new Object();
    private final Object ledgerLock = new Object();

    public void transfer() {
        synchronized (accountLock) {
            sleepQuietly(50);
            synchronized (ledgerLock) {
                System.out.println("transfer completed");
            }
        }
    }

    public void reconcile() {
        synchronized (ledgerLock) {
            sleepQuietly(50);
            synchronized (accountLock) {
                System.out.println("reconcile completed");
            }
        }
    }

    private void sleepQuietly(long millis) {
        try {
            Thread.sleep(millis);
        } catch (InterruptedException ignored) {
            Thread.currentThread().interrupt();
        }
    }
}

Prod Symptoms

A transfer path and a reconciliation path both protect account and ledger state, but they acquire the same monitors in different orders.

Key signal: This often feels like 'the app is stuck' even though the process is still alive.

  • A small set of requests never completes
  • Timeout rate rises while the JVM process remains alive
  • CPU is not necessarily high
  • Health checks may still pass if they do not touch the blocked path
  • Thread dumps show request threads BLOCKED on Java monitors
  • Restart clears the current hang, but the issue returns under the same interleaving

Run Locally

  • transfer-thread acquires accountLock
  • reconcile-thread acquires ledgerLock
  • Each thread then waits for the monitor held by the other
  • The main thread waits in join()
  • The Java process remains alive
  • The final println is usually never reached

What to look for

  • Threads in BLOCKED state
  • One thread waiting for ledgerLock while holding accountLock
  • Another thread waiting for accountLock while holding ledgerLock
Optional inspect commands
jps
jstack <pid>
ClassicMonitorDeadlockDemo.java
public class ClassicMonitorDeadlockDemo {
    private static final Object accountLock = new Object();
    private static final Object ledgerLock = new Object();

    public static void main(String[] args) throws InterruptedException {
        Thread t1 = new Thread(() -> {
            synchronized (accountLock) {
                System.out.println(Thread.currentThread().getName() + " acquired accountLock");
                sleepQuietly(200);
                synchronized (ledgerLock) {
                    System.out.println(Thread.currentThread().getName() + " acquired ledgerLock");
                }
            }
        }, "transfer-thread");

        Thread t2 = new Thread(() -> {
            synchronized (ledgerLock) {
                System.out.println(Thread.currentThread().getName() + " acquired ledgerLock");
                sleepQuietly(200);
                synchronized (accountLock) {
                    System.out.println(Thread.currentThread().getName() + " acquired accountLock");
                }
            }
        }, "reconcile-thread");

        t1.start();
        t2.start();

        t1.join();
        t2.join();

        System.out.println("This line will usually never be reached.");
    }

    private static void sleepQuietly(long millis) {
        try {
            Thread.sleep(millis);
        } catch (InterruptedException ignored) {
            Thread.currentThread().interrupt();
        }
    }
}

Note: This demo forces the bad interleaving by letting each thread acquire one monitor before requesting the other.

Diagnosis and fix

Explanation

This is a Java monitor deadlock caused by inconsistent lock ordering.

Key signal: The root cause is not synchronized itself. The root cause is that the code has no single lock order for the shared invariants.

  • synchronized acquires an intrinsic monitor for the locked object
  • transfer() acquires accountLock first, then ledgerLock
  • reconcile() acquires ledgerLock first, then accountLock
  • If two threads enter these methods at the same time, each thread can hold one lock and wait forever for the other
  • The JVM marks those worker threads as BLOCKED because they are waiting to enter synchronized blocks
  • One blocked thread is not enough to prove deadlock; the key signal is a wait cycle

How to Diagnose

When a JVM process appears stuck but is still alive, capture a thread dump and inspect lock ownership.

  • Use jps to find the JVM pid
  • Use jstack <pid> or jcmd <pid> Thread.print to capture thread state
  • Look for threads in BLOCKED state
  • Read both sides of the cycle: which monitor each thread owns and which monitor it is waiting to acquire
  • Map stack frames back to the code paths that acquire locks in opposite order
  • Correlate with recent changes that added nested synchronization or new cross-resource workflows
Commands
jps
jstack <pid>
Alternative
jcmd <pid> Thread.print
Expected diagnostic evidence
Found one Java-level deadlock:
=============================
"transfer-thread":
  waiting to lock monitor <monitor-address> (object <object-id>, a java.lang.Object),
  which is held by "reconcile-thread"

"reconcile-thread":
  waiting to lock monitor <monitor-address> (object <object-id>, a java.lang.Object),
  which is held by "transfer-thread"

Java stack information for the threads listed above:
===================================================
"transfer-thread":
        at ClassicMonitorDeadlockDemo.lambda$main$0(ClassicMonitorDeadlockDemo.java:...)
        - waiting to lock <object-id> (a java.lang.Object)
        - locked <object-id> (a java.lang.Object)
"reconcile-thread":
        at ClassicMonitorDeadlockDemo.lambda$main$1(ClassicMonitorDeadlockDemo.java:...)
        - waiting to lock <object-id> (a java.lang.Object)
        - locked <object-id> (a java.lang.Object)

Found 1 deadlock.

Note: A Java-level deadlock is usually stable: repeated thread dumps show the same cycle until the process is restarted.

How to Fix

  • Define one global lock order for shared resources and follow it everywhere
  • Document the lock hierarchy near the locks or the methods that acquire them
  • Keep nested synchronized blocks small and deliberate
  • Prefer one clear owner for a shared invariant when two locks protect one conceptual state transition
  • Do not treat bigger timeouts, restarts, or more request threads as fixes
  • Replacing synchronized with ReentrantLock does not fix inconsistent lock ordering by itself
ClassicMonitorDeadlockFixed.java
public class ClassicMonitorDeadlockFixed {
    private static final Object accountLock = new Object();
    private static final Object ledgerLock = new Object();

    public static void main(String[] args) throws InterruptedException {
        Thread t1 = new Thread(ClassicMonitorDeadlockFixed::transfer, "transfer-thread");
        Thread t2 = new Thread(ClassicMonitorDeadlockFixed::reconcile, "reconcile-thread");

        t1.start();
        t2.start();

        t1.join();
        t2.join();

        System.out.println("Completed without deadlock.");
    }

    private static void transfer() {
        synchronized (accountLock) {
            sleepQuietly(200);
            synchronized (ledgerLock) {
                System.out.println(Thread.currentThread().getName() + " completed transfer");
            }
        }
    }

    private static void reconcile() {
        synchronized (accountLock) {
            sleepQuietly(200);
            synchronized (ledgerLock) {
                System.out.println(Thread.currentThread().getName() + " completed reconcile");
            }
        }
    }

    private static void sleepQuietly(long millis) {
        try {
            Thread.sleep(millis);
        } catch (InterruptedException ignored) {
            Thread.currentThread().interrupt();
        }
    }
}

Note: The fix is the consistent order, not the specific choice of accountLock before ledgerLock.