HeapDumpOnOutOfMemoryError: Production Setup That Works

-XX:+HeapDumpOnOutOfMemoryError is not a heap dump strategy. It's one line of one. I know because I've now watched it fail to produce a dump in eight different ways on a laptop, and a few of them look exactly like success until the morning you need the file.

Here's the one that surprised me most. Same flags, same directory, same service, two crashes in a row:

$ java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps ...
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /dumps/java_pid1.hprof ...
Heap dump file created [52769160 bytes in 0.051 secs]

(restart, same volume)

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /dumps/java_pid1.hprof ...
Unable to create /dumps/java_pid1.hprof: File exists

First crash, you get a dump. Second crash, the JVM refuses to overwrite the first one and you get nothing. That output is from OpenJDK 21.0.12.1 (the Ubuntu 24.04 build, x86_64, 2 cores, 8 GB), and the reason is a detail of how containers number their processes. We'll get there.

What the log printed

If a service died with an OutOfMemoryError and there's no .hprof, read the lines right after the error. These are the messages I got on 21.0.12.1:

Dumping heap to /tmp/lab/java_pid351.hprof ...
Heap dump file created [52771874 bytes in 0.035 secs]

That's the good one: 35 milliseconds, because the heap was 64 MB.

Unable to create /tmp/lab/d7/java_pid663.hprof: Permission denied
Unable to create /tmp/lab/nonexistent/x: No such file or directory
Unable to create /tmp/lab/d2/fixed.hprof: File exists
Dump file is incomplete: No space left on device

Four ways to fail. They're printed by the VM straight to stdout, not by your logging framework, so a pipeline that keeps only leveled, timestamped lines drops them.

And then there are the failures that print nothing at all (cases 7 and 8 below).

What an OOM dump can and can't tell you

A heap dump is a snapshot of every object on the heap, with its fields and its references, written in the HPROF format. It's what lets you ask "what is holding on to these 4 million byte[]" with an answer instead of a guess. The tools that read it (Eclipse MAT and friends) turn that into retained sizes and paths to GC roots, and the leak-hunting workflow builds on it. This article is about getting the file, not reading it.

It can't see anything native: Metaspace, direct buffer pools, thread stacks and malloc arenas live outside the heap, and for those you want NMT. And it's a picture of the moment the error was thrown, not the hours of growth before it. For a before-and-after you need a deliberate second dump, covered below.

Eight ways I lost the dump

The lab: a report exporter that keeps every row it has produced in a list until the end of the run. Each "row" is a 256 KB byte[], so a 64 MB heap fills in under a second.

import java.nio.ByteBuffer;
import java.util.*;

public class Exporter {
    static final List<Object> rows = new ArrayList<>();
    public static void main(String[] a) throws Exception {
        switch (a[0]) {
            case "heap" -> { while (true) rows.add(new byte[256 * 1024]); }   // the exporter never lets go of a row: the leak
            case "direct" -> { while (true) rows.add(ByteBuffer.allocateDirect(1024 * 1024)); }
            case "twice" -> {
                for (int i = 0; i < 2; i++) {
                    try { while (true) rows.add(new byte[256 * 1024]); }
                    catch (OutOfMemoryError e) { rows.clear(); System.out.println("caught #" + i); }
                }
                System.out.println("done");
            }
        }
    }
}

Pin: OpenJDK 21.0.12.1, java Exporter.java-style single class compiled with javac. Everything below uses -Xmx64m -XX:+HeapDumpOnOutOfMemoryError plus whatever the case adds. I ran all of it directly, no container (this sandbox has no Docker daemon, so nothing here is Kubernetes output; I flag that where it matters). The limits are small so each dump is about 50 MB. Delete them when you're done. A leftover .hprof on a small disk is how you create a second, more confusing failure.

1. No HeapDumpPath at all. The dump goes to the process's working directory as java_pid<pid>.hprof. In a container image whose working directory is read-only, or /, or a jar-only layer, that's the end of it. Nothing in the error told you where it went except the Dumping heap to java_pid590.hprof ... line, with a relative path.

2. The directory doesn't exist. -XX:HeapDumpPath=/tmp/lab/nonexistent/x printed Unable to create ...: No such file or directory. The JVM doesn't create directories.

3. The directory isn't writable. I ran as nobody against a chmod 555 directory: Permission denied. This is the one you hit in Kubernetes with a non-root user and a volume mounted root-owned.

4. The file already exists. Both with a fixed file name and, the sneaky version, with the default name. The JVM won't overwrite. And the default name is java_pid<pid>. Inside a container your JVM is usually PID 1, so the default name is java_pid1.hprof every time. I reproduced that with unshare --pid --fork --mount-proc so java really was PID 1, ran it twice against the same directory, and got the output at the top of the page. If the volume survives a container restart, which is exactly what you set it up to do, your second crash is the one you lose. That's the best argument for a unique name per crash, and case 6 is why a pattern won't give you one.

5. The disk fills. I mounted a 20 MB tmpfs and pointed HeapDumpPath at it. The result was a file of exactly 20,971,520 bytes and the message Dump file is incomplete: No space left on device. The truncated file is still there. It looks like a dump. It isn't one, and a tool will either refuse it or show you a heap that stops halfway.

6. A %p in the path. I tried -XX:HeapDumpPath=/tmp/lab/d2/exp-%p.hprof expecting a PID in the name. On 21.0.12.1 the file was literally called exp-%p.hprof. The source agrees: I read the dump code at jdk-21+35 and there's no pattern expansion in it, just "existing directory means append the default name; otherwise treat the whole thing as the file name." (I haven't checked later releases. If you're on one, test it with a 64 MB heap before you trust a pattern.)

7. Not every OutOfMemoryError goes through the dump path. I ran the direct mode with -XX:MaxDirectMemorySize=16m. The error printed, as Cannot reserve 1048576 bytes of direct buffer memory (allocated: 16777216, limit: 16777216), and the dump directory stayed empty. That error is thrown by Java code in the JDK (java.nio.Bits), not reported by the VM, so the flag never fires. That's a failure with no message at all. If you run Netty or anything with big direct buffers, your dump flag silently doesn't cover that failure.

8. Only the first OOM gets a dump. The twice mode catches the first OutOfMemoryError, clears the list, and then does it again. Output: one Dumping heap line, one java_pid566.hprof, then caught #0, caught #1, done. The second OOM produced nothing, no message either. Which is a reasonable thing to know the day somebody wraps a request handler in catch (OutOfMemoryError).

That's eight, and the ones I'd have bet on, disk full and permissions, are only two of them.

Why the dump went missing, in one paragraph

The file couldn't be written (1 to 5), the JVM never tried (7 and 8), or you named it in a way the JVM doesn't interpret (6). None of it is a bug, and none of it raises an alarm.

What the JVM does between the error and the file

I went to the source for this because I wanted the order of things, and I wanted to know why case 8 happens. In HotSpot, a Java heap OutOfMemoryError ends up in report_java_out_of_memory in debug.cpp (I read it at jdk-21+35). It has a once-only guard: a static flag that's compare-and-swapped from 0 to 1 by the first thread that arrives, and never reset. That's case 8. Only that first caller does the work below, in this order:

OutOfMemoryError about to be thrown (VM-reported kind)
   |
   v
first reporter in the process?  --no-->  throw, nothing else happens
   | yes
   v
1. HeapDumpOnOutOfMemoryError  -> write the .hprof
2. OnOutOfMemoryError commands (-XX:OnOutOfMemoryError=...)
3. CrashOnOutOfMemoryError     -> abort with an hs_err file
4. ExitOnOutOfMemoryError      -> exit code 3, no shutdown hooks

Two practical things fall out of that order. The dump is taken before the process is told to exit, so -XX:+ExitOnOutOfMemoryError doesn't cost you the file (I checked: dump written, Terminating due to java.lang.OutOfMemoryError: Java heap space, exit code 3). And the dump runs without a GC first. The OOM path builds the dumper with the "garbage collect first" option off, so the dump includes unreachable objects that were still sitting on the heap. Good for you if the garbage is evidence. Noisier when you open it.

Then the HeapDumpPath rule from the same file: unset or empty means java_pid<pid>.hprof in the working directory; an existing directory gets the default name appended; anything else is the file name. HeapDumpGzipLevel above zero switches the extension to .hprof.gz. And the file is created with owner-only permissions (-rw------- in every run), which matters in the next section.

What took me the longest was not any of this. It was believing, for most of a day, that case 4 was something I'd done wrong in my lab. A dump that exists blocks the next one, a container's PID is 1 every time, and the volume outlives the container. All three facts are true independently, and I only saw the interaction when I forced the PID.

The setup I'd put in front of production

Start with the flags, then the storage, then the way out of the container.

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/dumps/oom
-XX:+ExitOnOutOfMemoryError

HeapDumpOnOutOfMemoryError defaults to false and is a manageable flag. So is HeapDumpPath. Manageable means you can change it on a running JVM, and I checked: jcmd <pid> VM.set_flag HeapDumpOnOutOfMemoryError true then jcmd <pid> VM.set_flag HeapDumpPath /dumps/oom both reported success and VM.flags showed them. If you inherited a service that never had the flag, you can turn it on without a restart. (And you should still put it in the launch config.)

HeapDumpPath as a directory, never a file. A directory means the default name, which is java_pid<pid>, and in a container that's the PID 1 problem again. So what's the fix? Two that work, and one that doesn't.

• Give each crash its own directory. Mount a volume and start the JVM from a tiny entrypoint that makes a directory named with the pod name and a timestamp, then passes that as HeapDumpPath. A shell exec java ... keeps the JVM as PID 1, which is fine, since the directory is what's unique.
• Rotate on startup. The entrypoint moves any existing java_pid1.hprof (or whatever is in /dumps/oom) to a name with a timestamp before the JVM starts. Cheap, and it means the dump that already exists is never in the way.
• Use a %p pattern. Doesn't work on 21.0.12.1. See case 6.

Size the volume for the worst case, not the average. The OOM dump is roughly the used heap at the moment of failure. In my 64 MB run that was about 52.7 MB. Budget for the whole -Xmx, plus headroom, for at least two dumps, because of case 4. And in Kubernetes the dump counts against whatever storage the volume is backed by. If it's an emptyDir it lives as long as the pod (it survives a container restart, not a reschedule) and, as I understand the docs, writes to it count toward the pod's ephemeral-storage limit. I couldn't run a cluster here, so check that on yours before you rely on it. Check it with a deliberately small heap and a deliberately small limit, in staging.

Getting a dump on demand

You'll want one that isn't triggered by a crash: a second snapshot of a growing process for a diff, or a dump from a process that's sick but alive.

jcmd <pid> GC.heap_dump /dumps/manual/before.hprof        # live objects only (does a full GC first)
jcmd <pid> GC.heap_dump -all /dumps/manual/before.hprof   # everything, no GC first
jcmd <pid> GC.heap_dump -gz=1 /dumps/manual/before.hprof.gz

On a 1 GB heap holding about 300 MB of live data and generating garbage, I got:

Take the compressed row with a large grain of salt: my "live data" is zero-filled byte arrays and compresses absurdly well. Real heaps are full of strings and object headers and won't. Measure yours.

What matters more is the pause. A ticker thread in the same JVM, sleeping 5 ms at a time, logged stalls of 344 ms and 560 ms and 941 ms, matching the three dumps. The process is frozen while it's written. At 300 MB on a warm page cache that's under a second. I haven't measured a 14 GB heap and neither should you extrapolate from mine; it's the sort of number that decides whether a readiness probe fails and the orchestrator kills the pod mid-dump. Test it on a copy.

Two more jcmd details. The file is written by the target JVM, so the path is resolved in the target's file system and working directory. My relative rel.hprof appeared in the lab directory where the target was running, not where I typed the command. In a container that means inside the container, and you need kubectl cp or equivalent to get it out. And the same existing-file rule applies: asking jcmd to write live.hprof twice gave Unable to create ...: File exists.

What each knob costs

That last row is also the non-fix. Raising the heap so the dump has more room doesn't give you a better dump. It gives you a larger one.

Dumps as a pipeline, not a flag

The design that avoids all eight cases treats the dump as data with a lifecycle: where it's written, how it gets out, who's allowed to read it, when it's deleted.

Write to a volume that outlives the container and has a known size. Get the file off the node, with a sidecar or a job or a kubectl cp step in your runbook, because a dump sitting on a node you're about to drain isn't a dump. And keep a retention rule so yesterday's dump doesn't block today's.

And treat the file as a secret. I put a fake API key in a String in the lab and then ran grep -a on the .hprof. It printed the key. Twice. Heap dumps contain whatever was on the heap: tokens, session ids, customer payloads, and card numbers if you're unlucky. The files are created 0600, which helps on a shared node and does nothing for the moment you copy one to a ticket. Restrict who can read the volume. Encrypt wherever the dump rests. Don't attach one to a public bug report. If you must share, analyse it on a controlled box and share the findings.

Test the dump path before you need it

The single most useful thing I'd add to a deploy checklist: a staging run that forces an OOM with the production flags, production user, production volume and a small heap, and then checks that a valid file is on the other side. Not "the flag is present". The file, copied out, opened.

What to alert on: the volume's free space; the count of files in the dump directory (so you notice the day an old dump would block a new one); and any restart with exit code 3 or 137 and no new file. Exit code 137 is a different failure entirely: the kernel killed the process, the JVM never ran any of the code above, and there is never a dump. That's the OOMKilled article, and the comparison tells them apart.

For a leak that is slow, a periodic jcmd GC.heap_dump at a scheduled interval beats waiting for the crash, because you get the before and after. The alert set for memory signals says when to take it.

Version notes: what I'd re-check on an upgrade

Everything above is OpenJDK 21.0.12.1, with the HotSpot source read at jdk-21+35. Re-check these:

• Whether HeapDumpPath expands %p or similar. It doesn't here.
• The HeapDumpGzipLevel flag exists and is manageable on 21 (I read it from PrintFlagsFinal); I don't know the first release that had it. If you run an older JDK, check.
• jcmd GC.heap_dump options (-all, -gz=) worked as shown on 21. Older JDKs use different spellings (-live). Run jcmd <pid> help GC.heap_dump on yours.
• The once-per-process behaviour. I read it in the source and saw it in the lab. If a newer release changes that, case 8 changes.
• The set of OOMs that reach the dump path. Direct buffer didn't. I didn't test Metaspace, native thread or array-size errors, so I'm not telling you whether they dump. Try yours.

I didn't run JDK 25 for any of this, and I didn't run Kubernetes. The honest version of "how long did this take" is that sorting out which of the eight were real, rather than my lab being wrong, ate more than a day. The table above is so you don't spend that day.

If you remember nothing else

• The flag gets you a dump only when the OOM is VM-reported, the first one in the process, and the path is writable, free, and not already occupied.
• In a container the default dump name is java_pid1.hprof every time. A persistent volume plus no cleanup means the second crash gets no dump.
• HeapDumpPath is a directory or a file name. No %p on 21.0.12.1, I tested it.
• A dump freezes the process: 344 ms for 300 MB live in my lab. You decide whether your probes survive a 14 GB one, by testing.
• A heap dump is a copy of your secrets. Treat the volume that way.

Quick answers about heap dumps on OOM

Where does HeapDumpOnOutOfMemoryError write the file by default?
To the JVM's current working directory, named java_pid<pid>.hprof (.hprof.gz if HeapDumpGzipLevel is above 0). Set -XX:HeapDumpPath to a directory you've checked is writable.

Why didn't my Java app create a heap dump on OutOfMemoryError?
Usually the path wasn't writable, the disk was full, a dump file of that name already existed, or the error wasn't one the VM reports (direct buffer errors thrown from Java code skip it on 21). It's also once per process, so a second OOM in the same JVM won't dump. See the eight cases above.

Can I use %p in HeapDumpPath?
Not on OpenJDK 21.0.12.1. I got a file literally named exp-%p.hprof. Use a directory, and make it unique per crash with your entrypoint script.

How big is a heap dump and how long does it take?
About the size of the used heap at that moment: 52.7 MB for a full 64 MB heap, 319 MB for 300 MB of live data in my lab. It took 35 ms and 340 ms respectively. Large heaps take longer and the process is frozen the whole time, so test your own.