Check-Then-Act Race in an Idempotency Guard

Check-Then-Act Race in an Idempotency Guard: practice a Java concurrency bug with symptoms like Duplicate side effect, Thread-safe collection still...

Check-then-act races
Atomicity
Concurrent Collections
Java
Intermediate

Production symptoms

Duplicate side effect
Thread-safe collection still wrong
Intermittent race

Failure scenario

Code

Java example

class IdempotencyGuard {
    private final Set<String> processed = ConcurrentHashMap.newKeySet();

    void handle(String messageId) {
        if (!processed.contains(messageId)) {
            chargeCustomer(messageId);
            processed.add(messageId);
        }
    }
}

Prod Symptoms

A service uses a thread-safe set or map to suppress duplicate work, but the check and the side effect are separate operations.

Two workers process the same message, request, payment, or webhook at nearly the same time
Logs show both workers saw the id as not processed
The dedup set contains only one id afterward, so the final in-memory state looks clean
The duplicate is visible downstream: two sends, two writes, two charges, callbacks, or audit rows
The bug is intermittent and gets easier to trigger under retry storms, duplicate deliveries, or webhook retries

Run Locally

Both workers check the same id
Both observe that it is absent
Both perform the side effect
The final set size is one, which can hide the duplicate work

Inspect hints

Look at side-effect evidence, not only final in-memory state
Search for contains/get followed by add/put around idempotency, deduplication, or cache-fill logic
A thread dump usually cannot reconstruct the check-then-act window after it has passed

Run

javac CheckThenActIdempotencyDemo.java
java CheckThenActIdempotencyDemo

CheckThenActIdempotencyDemo.java

import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;

public class CheckThenActIdempotencyDemo {
    private static final Set<String> processed = ConcurrentHashMap.newKeySet();
    private static final AtomicInteger sideEffects = new AtomicInteger();
    private static final CountDownLatch bothChecked = new CountDownLatch(2);

    public static void main(String[] args) throws Exception {
        Thread first = new Thread(() -> handleBroken("msg-42"), "worker-1");
        Thread second = new Thread(() -> handleBroken("msg-42"), "worker-2");

        first.start();
        second.start();

        first.join();
        second.join();

        System.out.println("processed ids = " + processed.size());
        System.out.println("side effects  = " + sideEffects.get());
    }

    private static void handleBroken(String messageId) {
        try {
            if (!processed.contains(messageId)) {
                bothChecked.countDown();
                bothChecked.await();

                sideEffects.incrementAndGet();
                processed.add(messageId);
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

Note: Both workers pass the contains check before either worker records the id as processed.

Diagnosis and fix

Explanation

contains() and add() are each thread-safe here, but the workflow between them is not atomic.

Key signal: The atomic boundary must cover the claim to do the work, not just the collection operation.

Thread A checks the set and sees the id is absent
Thread B checks the same id before Thread A records it
Both threads perform the side effect
Both eventually record the same id, so the final set hides the duplicate side effect
This is a check-then-act race: the condition can change between the check and the action

How to Diagnose

Use downstream evidence and code review around idempotency boundaries.

Compare final in-memory dedup state with side-effect evidence: emails, charges, writes, callbacks, or audit rows
Look for contains/get followed by side effects and then add/put
Review retry, webhook, duplicate-delivery, and duplicate-message paths
Look for logs where two workers both decide they own the same id
Remember that a clean final set does not prove the side effect ran only once

Typical output

processed ids = 1
side effects  = 2

How to Fix

For in-JVM duplicate suppression, use the return value of add(), putIfAbsent(), compute(), merge(), or another atomic claim operation
Perform the in-JVM side effect only in the branch that won the atomic claim
Do not use contains() followed by side effect and then add()
For payments, emails, webhooks, or cross-pod idempotency, use a durable idempotency key, unique constraint, idempotency table, or state machine
Be careful claiming before the side effect: if the side effect fails, retry semantics must be explicit

CheckThenActIdempotencyFixed.java

import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;

public class CheckThenActIdempotencyFixed {
    private static final Set<String> processed = ConcurrentHashMap.newKeySet();
    private static final AtomicInteger sideEffects = new AtomicInteger();

    public static void main(String[] args) throws Exception {
        CountDownLatch start = new CountDownLatch(1);
        Thread first = new Thread(() -> handleAfterStart(start, "msg-42"), "worker-1");
        Thread second = new Thread(() -> handleAfterStart(start, "msg-42"), "worker-2");

        first.start();
        second.start();
        start.countDown();

        first.join();
        second.join();

        System.out.println("processed ids = " + processed.size());
        System.out.println("side effects  = " + sideEffects.get());
    }

    private static void handleAfterStart(CountDownLatch start, String messageId) {
        try {
            start.await();
            if (processed.add(messageId)) {
                sideEffects.incrementAndGet();
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

Note: The set add is only an in-JVM atomic claim. For durable business idempotency, make the claim in durable storage and model failure/retry states explicitly.