Metaspace in JVM Builds

This is a post in my JVM Args for Builds series where I break down how these settings effect JVM-based build systems. A number of these settings are complicated and all have compounding effects on each other as the JVM is a complex machine, but used the right way can be the difference between having a high-value feedback build system versus a high-cost one.

Metaspace can contribute significantly to the non-heap memory footprint. If you want to learn all the details of how it works I highly recommend Thomas Stüfe[[1]]'s blog post series on the topic[[2]] - I will only be covering the relevant details as they relate to JVM-based build systems, not JVM-based applications.

TLDR;

If you're using Gradle:

• with JDK 17+, override Gradle's JVM arg defaults and do not set -XX:MaxMetaspaceSize.
• with JDK 16 or older, make sure to have XX:MaxMetaspaceSize set to prevent ClassLoader leaks from running wild. Measure your usage with jcmd <PID> VM.metaspace and read towards the end of this post to see an example.
• -XX:MetaspaceSize should not be set for most projects - this governs when to trigger a garbage collection the first time it is exceeded. The defaults work well on this.

If you're in any other JVM build system, read the documentation but unless it has a default setting for -XX:MaxMetaspaceSize you likely don't need to do anything.

The Story behind Metaspace in Gradle

In my experience with Android projects and Gradle a number of memory settings have been shared in the community. I noticed a shift from MaxPermgenSize to MaxMetaspaceSize as the Android community moved from JDK 6 to JDK 8+, but I didn't understand at the time that there was an effort to replace PermGen with a new system[[3]] and that the new system behaved entirely differently. Since that time Gradle introduced usage of the MaxMetaspaceSize in its default JVM args in 2018[[4]] to promote failing fast on memory leaks. The Android community was trying to adapt to ever larger codebases and since there aren't a lot of people reading JVM release notes, the popular consensus became that we needed to use the setting Gradle provided[[5]].

Unfortunately what got missed was that the Metaspace system was specifically designed to not have a limit imposed - that was actually the fundamental problem with PermGen[[6]] where its size could not grow on-demand even if system resources were available. There was also no discussion about how MaxMetaspaceSize limits both non-class metadata and CompressedClassSpaceSize. Simultaneously the Metaspace system had become notorious for not returning unused memory back to the system and this issue wasn't addressed until Java 16[[7]], so it made a lot of sense at the time to limit Metaspace until various ClassLoader leaks were addressed, but so much time has passed that few in the Gradle community are considering removing the limit.

Another issue related to this is the awareness of how to apply JVM args correctly in Gradle builds. I've used other systems in the past (Maven, Ant) that didn't support as much functionality but made such configuration settings simpler to apply. In recent years this got a lot easier and the knowledge is becoming more commonplace, but most engineers who work with Gradle as a build system remain unaware.

Why is having a Metaspace limit bad?

CompressedClassSpaceSize defaults to 1GB[[8]] if CompressedClassPointers is enabled (which is it by default for JVM heaps under 32GB). If MaxMetaspaceSize is less than 1GB the JVM will use heuristics to determine how much to allocate. Compressed class pointers also reduce memory usage by allowing the JVM to represent class metadata with less memory.

So setting MaxMetaspaceSize has a few consequences:

• If we're at the limit and the JVM has more class pointers to assign into memory it will OOM. Also the OOM will not reference Metadata but instead CompressedClassSpace.
• If we're at the limit and the JVM has more non-class metadata to write it will OOM on Metadata.

If you've never run into this because your project isn't too large, you don't need to worry about this and continue happily with the defaults. But if you've got a few thousand classes and have started to see Caused by: java.lang.OutOfMemoryError: Metaspace you should keep reading.

Metaspace Growth & Implications

First, let's put into perspective why and when this will matter for a given codebase. Since we know we have two competing spaces in memory, let's break down what each class roughly puts into them. If you're seriously interested in more detail than this breakdown you can read JVM source code[[9]].

• Class Space
  • The Klass structure represents the fundamental metadata for the class in the JVM. It contains information about the class name, superclass, interfaces, fields, methods, and more.
  • vtable (virtual method table) holds pointers to the implementations of the class’s virtual methods and any inherited from its super class.
  • itable (interface method table) stores pointers to methods that implement interfaces.
  • The nonstatic ordinary object pointer map holds metadata about the instance fields of the class that are object references (non-primitive types).
  • Misc - additional internal JVM data structures, and padding for alignment.
• Non-Class Metadata
  • The constant pool, which is variable sized.
  • Method metadata: bytecode, variable and exception tables, and more
  • Annotations

I came across Thomas Stüfe's blog posts and he has an excellent real world profiling example with jcmd VM.metaspace [[10]] to get these results:

For standard classes (assuming classes loaded by bootstrap and app loaders are considered standard), we come to an average of ~5-7k of non-class space and 600-900 bytes class space per class.

Anonymous (lambda) classes are much smaller, no surprise, but interestingly ratio between class and non-class space usage is also warped: we need much more class space in relation to non-class space. That is not surprising since Lambda classes are minuscule, but the overhead for the Klass structure cannot shrink below sizeof(Klass) structure itself. So, we come to about 1k non-class space, .5k class space.

Let's take Thomas's numbers[[11]] (~5-7k non-class space, 600-900 bytes class space) and had the JVM defaults we would only be limited by CompressedClassSpaceSize and be able to run 1-1.5 million classes, which is definitely more code than I ever hope to write. However with a Gradle's current defaults setting MaxMetaspaceSize limit to 768MB and Thomas's estimate that 80% of that is given to CompressedClassSpaceSize, we are left with ~150MB for non-class metadata. That would mean a limit of 20k-30k classes - not an easy limit to hit but I'm sure the larger and more complex Android apps are.

Does this issue impact my codebase?

We've learned that we can profile metaspace usage via jcmd VM.metaspace, but what does that look like practically?

For starters you can grab the PID from a running Gradle Daemon:

GRADLE_PID=$(pgrep -f '.*GradleDaemon.*' | xargs -I{} ps -o pid= -o lstart= -p {} | sort -k2,3 | tail -n 1 | awk '{print $1}')
jcmd $GRADLE_PID VM.metaspace

I've written a bash script that shows how to grab the PID from a running Gradle process. You should be able to apply this to other JVM-based build tools.

Running this against my android-ci repository during a clean build doesn't indicate much metaspace usage, but since it's literally an Android Studio starter wizard project that is expected.

However running against the NowInAndroid codebase shows some interesting things right off the bat. At the time of writing it still has MaxMetaspace set to 1GB.

  
    35593:

Total Usage - 4558 loaders, 64652 classes (1444 shared):
  Non-Class: 19007 chunks,   301.93 MB capacity,  301.49 MB (>99%) committed,   299.15 MB (>99%) used,     2.33 MB ( <1%) free,    15.47 KB ( <1%) waste , deallocated: 1477 blocks with 461.70 KB
      Class: 6956 chunks,     43.12 MB capacity,   43.00 MB (>99%) committed,    40.52 MB ( 94%) used,     2.48 MB (  6%) free,   600 bytes ( <1%) waste , deallocated: 2144 blocks with 563.65 KB
       Both: 25963 chunks,   345.06 MB capacity,  344.49 MB (>99%) committed,   339.66 MB ( 98%) used,     4.82 MB (  1%) free,    16.05 KB ( <1%) waste , deallocated: 3621 blocks with 1.00 MB


Virtual space:
  Non-class space:      320.00 MB reserved,     301.50 MB ( 94%) committed,  5 nodes.
      Class space:      832.00 MB reserved,      43.06 MB (  5%) committed,  1 nodes.
             Both:        1.12 GB reserved,     344.56 MB ( 30%) committed.


Chunk freelists:
   Non-Class:

 16m: (none)
  8m: (none)
  4m: (none)
  2m:    2, capacity=4.00 MB, committed=0 bytes (  0%)
  1m: (none)
512k: (none)
256k: (none)
128k: (none)
 64k:    2, capacity=128.00 KB, committed=0 bytes (  0%)
 32k: (none)
 16k: (none)
  8k: (none)
  4k:    2, capacity=8.00 KB, committed=0 bytes (  0%)
  2k:    2, capacity=4.00 KB, committed=0 bytes (  0%)
  1k: (none)
Total word size: 4.14 MB, committed: 0 bytes (  0%)

       Class:

 16m: (none)
  8m: (none)
  4m:    1, capacity=4.00 MB, committed=0 bytes (  0%)
  2m: (none)
  1m: (none)
512k:    1, capacity=512.00 KB, committed=0 bytes (  0%)
256k:    1, capacity=256.00 KB, committed=0 bytes (  0%)
128k: (none)
 64k:    1, capacity=64.00 KB, committed=0 bytes (  0%)
 32k:    1, capacity=32.00 KB, committed=0 bytes (  0%)
 16k:    1, capacity=16.00 KB, committed=0 bytes (  0%)
  8k:    1, capacity=8.00 KB, committed=0 bytes (  0%)
  4k:    1, capacity=4.00 KB, committed=0 bytes (  0%)
  2k:    1, capacity=2.00 KB, committed=0 bytes (  0%)
  1k:    1, capacity=1.00 KB, committed=0 bytes (  0%)
Total word size: 4.87 MB, committed: 0 bytes (  0%)

        Both:

 16m: (none)
  8m: (none)
  4m:    1, capacity=4.00 MB, committed=0 bytes (  0%)
  2m:    2, capacity=4.00 MB, committed=0 bytes (  0%)
  1m: (none)
512k:    1, capacity=512.00 KB, committed=0 bytes (  0%)
256k:    1, capacity=256.00 KB, committed=0 bytes (  0%)
128k: (none)
 64k:    3, capacity=192.00 KB, committed=0 bytes (  0%)
 32k:    1, capacity=32.00 KB, committed=0 bytes (  0%)
 16k:    1, capacity=16.00 KB, committed=0 bytes (  0%)
  8k:    1, capacity=8.00 KB, committed=0 bytes (  0%)
  4k:    3, capacity=12.00 KB, committed=0 bytes (  0%)
  2k:    3, capacity=6.00 KB, committed=0 bytes (  0%)
  1k:    1, capacity=1.00 KB, committed=0 bytes (  0%)
Total word size: 9.01 MB, committed: 0 bytes (  0%)



Waste (unused committed space):(percentages refer to total committed size 344.56 MB):
        Waste in chunks in use:     16.05 KB ( <1%)
        Free in chunks in use:      4.82 MB (  1%)
                In free chunks:      0 bytes (  0%)
Deallocated from chunks in use:      1.00 MB ( <1%) (3621 blocks)
                       -total-:      5.83 MB (  2%)

chunk header pool: 25978 items, 1.79 MB.

Internal statistics:

num_allocs_failed_limit: 12.
num_arena_births: 9494.
num_arena_deaths: 378.
num_vsnodes_births: 6.
num_vsnodes_deaths: 0.
num_space_committed: 5613.
num_space_uncommitted: 48.
num_chunks_returned_to_freelist: 1088.
num_chunks_taken_from_freelist: 27040.
num_chunk_merges: 404.
num_chunk_splits: 17341.
num_chunks_enlarged: 10366.
num_inconsistent_stats: 0.


Settings:
MaxMetaspaceSize: 1.00 GB
CompressedClassSpaceSize: 832.00 MB
Initial GC threshold: 21.00 MB
Current GC threshold: 496.25 MB
CDS: on
 - commit_granule_bytes: 65536.
 - commit_granule_words: 8192.
 - virtual_space_node_default_size: 8388608.
 - enlarge_chunks_in_place: 1.
 - use_allocation_guard: 0.
  
  

There's a couple things to unpack from those logs:

• CompressedClassSpace would normally default to 1 GB, but MaxMetaspaceSize limits the entire space and therefore CompressedClassSpace becomes 832 MB.
• At the end of this build if we take a look at VirtualSpace and whats been committed we see 301.50 MB for non-class space (metaspace) and 43.06 MB for class space. CompressedClassSpace cannot change or grow, its at 832 MB. Everything has plenty of room, no OOM crash.

What happens if I run the same build with drastically less MaxMetaspaceSize? Expectedly we saw an OOM crash, but we're going to analyze when this would happen if the codebase did grow larger.

  
> Configure project :benchmarks
This version of the Baseline Profile Gradle Plugin was tested at most with the Android
Gradle Plugin version Android Gradle Plugin version 8.3.0 and it may not work as intended.
Current version is Android Gradle Plugin version 8.6.0.

FAILURE: Build failed with an exception.

* What went wrong:
Metaspace

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 10s
5 actionable tasks: 1 executed, 4 up-to-date
Configuration cache entry stored.
  
  

  
Total Usage - 2558 loaders, 27915 classes (1423 shared):
  Non-Class: 7612 chunks,    104.90 MB capacity,  104.90 MB (100%) committed,   103.46 MB ( 99%) used,     1.44 MB (  1%) free,     6.64 KB ( <1%) waste , deallocated: 482 blocks with 174.23 KB
      Class: 3643 chunks,     18.33 MB capacity,   18.14 MB ( 99%) committed,    16.71 MB ( 91%) used,     1.43 MB (  8%) free,   288 bytes ( <1%) waste , deallocated: 954 blocks with 267.43 KB
       Both: 11255 chunks,   123.23 MB capacity,  123.04 MB (>99%) committed,   120.17 MB ( 98%) used,     2.87 MB (  2%) free,     6.92 KB ( <1%) waste , deallocated: 1436 blocks with 441.66 KB


Virtual space:
  Non-class space:      128.00 MB reserved,     104.94 MB ( 82%) committed,  2 nodes.
      Class space:      112.00 MB reserved,      18.19 MB ( 16%) committed,  1 nodes.
             Both:      240.00 MB reserved,     123.12 MB ( 51%) committed.


Chunk freelists:
   Non-Class:

 16m: (none)
  8m: (none)
  4m:    2, capacity=8.00 MB, committed=0 bytes (  0%)
  2m:    2, capacity=4.00 MB, committed=0 bytes (  0%)
  1m:    2, capacity=2.00 MB, committed=0 bytes (  0%)
512k: (none)
256k: (none)
128k: (none)
 64k:    2, capacity=128.00 KB, committed=0 bytes (  0%)
 32k:    2, capacity=64.00 KB, committed=0 bytes (  0%)
 16k: (none)
  8k: (none)
  4k: (none)
  2k:    2, capacity=4.00 KB, committed=0 bytes (  0%)
  1k:    2, capacity=2.00 KB, committed=0 bytes (  0%)
Total word size: 14.19 MB, committed: 0 bytes (  0%)

       Class:

 16m: (none)
  8m:    1, capacity=8.00 MB, committed=0 bytes (  0%)
  4m:    1, capacity=4.00 MB, committed=0 bytes (  0%)
  2m: (none)
  1m:    1, capacity=1.00 MB, committed=0 bytes (  0%)
512k:    1, capacity=512.00 KB, committed=0 bytes (  0%)
256k: (none)
128k:    1, capacity=128.00 KB, committed=0 bytes (  0%)
 64k: (none)
 32k:    1, capacity=32.00 KB, committed=0 bytes (  0%)
 16k: (none)
  8k:    1, capacity=8.00 KB, committed=0 bytes (  0%)
  4k:    1, capacity=4.00 KB, committed=0 bytes (  0%)
  2k:    1, capacity=2.00 KB, committed=0 bytes (  0%)
  1k: (none)
Total word size: 13.67 MB, committed: 0 bytes (  0%)

        Both:

 16m: (none)
  8m:    1, capacity=8.00 MB, committed=0 bytes (  0%)
  4m:    3, capacity=12.00 MB, committed=0 bytes (  0%)
  2m:    2, capacity=4.00 MB, committed=0 bytes (  0%)
  1m:    3, capacity=3.00 MB, committed=0 bytes (  0%)
512k:    1, capacity=512.00 KB, committed=0 bytes (  0%)
256k: (none)
128k:    1, capacity=128.00 KB, committed=0 bytes (  0%)
 64k:    2, capacity=128.00 KB, committed=0 bytes (  0%)
 32k:    3, capacity=96.00 KB, committed=0 bytes (  0%)
 16k: (none)
  8k:    1, capacity=8.00 KB, committed=0 bytes (  0%)
  4k:    1, capacity=4.00 KB, committed=0 bytes (  0%)
  2k:    3, capacity=6.00 KB, committed=0 bytes (  0%)
  1k:    2, capacity=2.00 KB, committed=0 bytes (  0%)
Total word size: 27.86 MB, committed: 0 bytes (  0%)



Waste (unused committed space):(percentages refer to total committed size 123.12 MB):
        Waste in chunks in use:      6.92 KB ( <1%)
        Free in chunks in use:      2.87 MB (  2%)
                In free chunks:      0 bytes (  0%)
Deallocated from chunks in use:    441.66 KB ( <1%) (1436 blocks)
                       -total-:      3.31 MB (  3%)

chunk header pool: 11272 items, 802.39 KB.

Internal statistics:

num_allocs_failed_limit: 12.
num_arena_births: 5116.
num_arena_deaths: 0.
num_vsnodes_births: 3.
num_vsnodes_deaths: 0.
num_space_committed: 1970.
num_space_uncommitted: 0.
num_chunks_returned_to_freelist: 12.
num_chunks_taken_from_freelist: 11256.
num_chunk_merges: 12.
num_chunk_splits: 7226.
num_chunks_enlarged: 4333.
num_inconsistent_stats: 0.


Settings:
MaxMetaspaceSize: 128.00 MB
CompressedClassSpaceSize: 112.00 MB
Initial GC threshold: 21.00 MB
Current GC threshold: 128.00 MB
CDS: on
 - commit_granule_bytes: 65536.
 - commit_granule_words: 8192.
 - virtual_space_node_default_size: 8388608.
 - enlarge_chunks_in_place: 1.
  
  

And through a few iterations I found the minimum passing MaxMetaspaceSize of 160MB

  
Total Usage - 3587 loaders, 33850 classes (1438 shared):
  Non-Class: 10872 chunks,   134.75 MB capacity,  134.69 MB (>99%) committed,   132.89 MB ( 99%) used,     1.79 MB (  1%) free,     8.23 KB ( <1%) waste , deallocated: 1149 blocks with 402.37 KB
      Class: 4935 chunks,     22.47 MB capacity,   22.41 MB (>99%) committed,    20.50 MB ( 91%) used,     1.90 MB (  8%) free,   416 bytes ( <1%) waste , deallocated: 1194 blocks with 324.70 KB
       Both: 15807 chunks,   157.22 MB capacity,  157.09 MB (>99%) committed,   153.39 MB ( 98%) used,     3.69 MB (  2%) free,     8.63 KB ( <1%) waste , deallocated: 2343 blocks with 727.06 KB


Virtual space:
  Non-class space:      192.00 MB reserved,     134.69 MB ( 70%) committed,  3 nodes.
      Class space:      128.00 MB reserved,      22.44 MB ( 18%) committed,  1 nodes.
             Both:      320.00 MB reserved,     157.12 MB ( 49%) committed.


Chunk freelists:
   Non-Class:

 16m: (none)
  8m:    2, capacity=16.00 MB, committed=0 bytes (  0%)
  4m: (none)
  2m: (none)
  1m:    2, capacity=2.00 MB, committed=0 bytes (  0%)
512k: (none)
256k:    2, capacity=512.00 KB, committed=0 bytes (  0%)
128k: (none)
 64k: (none)
 32k: (none)
 16k: (none)
  8k: (none)
  4k: (none)
  2k: (none)
  1k: (none)
Total word size: 18.50 MB, committed: 0 bytes (  0%)

       Class:

 16m: (none)
  8m:    1, capacity=8.00 MB, committed=0 bytes (  0%)
  4m: (none)
  2m: (none)
  1m:    1, capacity=1.00 MB, committed=0 bytes (  0%)
512k:    1, capacity=512.00 KB, committed=0 bytes (  0%)
256k: (none)
128k: (none)
 64k: (none)
 32k:    1, capacity=32.00 KB, committed=0 bytes (  0%)
 16k: (none)
  8k: (none)
  4k: (none)
  2k: (none)
  1k: (none)
Total word size: 9.53 MB, committed: 0 bytes (  0%)

        Both:

 16m: (none)
  8m:    3, capacity=24.00 MB, committed=0 bytes (  0%)
  4m: (none)
  2m: (none)
  1m:    3, capacity=3.00 MB, committed=0 bytes (  0%)
512k:    1, capacity=512.00 KB, committed=0 bytes (  0%)
256k:    2, capacity=512.00 KB, committed=0 bytes (  0%)
128k: (none)
 64k: (none)
 32k:    1, capacity=32.00 KB, committed=0 bytes (  0%)
 16k: (none)
  8k: (none)
  4k: (none)
  2k: (none)
  1k: (none)
Total word size: 28.03 MB, committed: 0 bytes (  0%)



Waste (unused committed space):(percentages refer to total committed size 157.12 MB):
        Waste in chunks in use:      8.63 KB ( <1%)
        Free in chunks in use:      3.69 MB (  2%)
                In free chunks:      0 bytes (  0%)
Deallocated from chunks in use:    727.06 KB ( <1%) (2343 blocks)
                       -total-:      4.41 MB (  3%)

chunk header pool: 15815 items, 1.09 MB.

Internal statistics:

num_allocs_failed_limit: 12.
num_arena_births: 7366.
num_arena_deaths: 192.
num_vsnodes_births: 4.
num_vsnodes_deaths: 0.
num_space_committed: 2512.
num_space_uncommitted: 0.
num_chunks_returned_to_freelist: 288.
num_chunks_taken_from_freelist: 16084.
num_chunk_merges: 132.
num_chunk_splits: 9922.
num_chunks_enlarged: 5467.
num_inconsistent_stats: 0.


Settings:
MaxMetaspaceSize: 160.00 MB
CompressedClassSpaceSize: 128.00 MB
Initial GC threshold: 21.00 MB
Current GC threshold: 160.00 MB
CDS: on
 - commit_granule_bytes: 65536.
 - commit_granule_words: 8192.
 - virtual_space_node_default_size: 8388608.
 - enlarge_chunks_in_place: 1.
 - use_allocation_guard: 0.
  
  

What are the important differences?

num_arena_births: This represents the number of times the JVM has created new “arenas” in the metaspace. Arenas are contiguous memory regions used by the JVM to manage Metaspace allocations. When the JVM needs more memory for class or non-class data, it creates a new arena (essentially reserving more memory from the OS).

num_arena_deaths: This tracks how many arenas have been destroyed (or “died”) because they were no longer needed. This happens when the JVM releases memory back to the OS or frees up arenas that are no longer in use. Arena deaths typically occur after garbage collection if class loaders are unloaded and their associated metadata is no longer needed.

num_chunks_returned_to_freelist: This number tells you how many chunks of memory have been returned to the freelist after being used and deallocated. Essentially, when the JVM no longer needs certain memory chunks, they are returned to the freelist, so they can be reused later.

num_chunks_taken_from_freelist: This represents the number of chunks that have been reallocated from the freelist when the JVM needed to allocate new memory. Instead of asking the OS for new memory, the JVM reuses memory chunks from the freelist, reducing fragmentation and overhead.

• 1GB MaxMetaspace:
  • 5480 loaders, 86611 classes (1455 shared)
  • num_arena_births: 9494
  • num_arena_deaths: 378
  • num_chunks_returned_to_freelist: 1088.
  • num_chunks_taken_from_freelist: 27040.
• 160 MaxMetaspace:
  • 3587 loaders, 33850 classes (1438 shared)
  • num_arena_births: 7366.
  • num_arena_deaths: 192.
  • num_chunks_returned_to_freelist: 288.
  • num_chunks_taken_from_freelist: 16084.
• 128 MaxMetaspace:
  • 2558 loaders, 27915 classes (1423 shared)
  • num_arena_births: 5116
  • num_arena_deaths: 0
  • num_chunks_returned_to_freelist: 12.
  • num_chunks_taken_from_freelist: 11256.

After reducing the MaxMetaspaceSize the JVM loaded fewer classes, indicating that memory pressure led to class unloading or prevented further loading. Arena births reached 7366 in passing builds but could not reach that count before the OOM crash in builds using less than 160MB. We can see the failing one doesn't return many chunks back to the freelist and that makes sense with high memory pressure on essential chunks.

We are able to confirm Thomas's claim that 80% of MaxMetaspaceSize is reserved for CompressedClassSpace, but then the non-class metaspace is allowed to grow within the total maximum size to compete with it. When space runs out we observe the OOM, and since this is a small project without a lot going on its more easily reproducible on clean builds with no cache.

Let's also check against Thomas's rough estimates for class and non-class space on a per class basis. If we look at the number of classes referenced by the minimum passing build (33,850). Does the math match up?

600-900 byte class space size at 33,850 classes is 19MB - 29MB

5kb-7kb non-class space at 33,850 classes is 165MB - 231MB

Yup. So now we have an idea of the range of memory we need to provide to ensure this project passes. And now you know how to calculate this (roughly) for your project too.

Build System Recommendations

Metaspace varies a lot from project to project and setting an arbitrary limit can create GC pauses and unexpected OOMs that would confuse developers who think JVM heap, Metaspace, and Codecache are all shared.

If you're an Android developer using AGP 8.5.0 or higher you are required to be using JDK 17, so my recommendation is try not setting -XX:MaxMetaspaceSize in build system JVM args. Specifically for Gradle that means you should override its defaults and not set anything for -XX:MaxMetaspaceSize. The only performance improvement by adding a setting is found through -XX:MetaspaceSize which will allow your build to skip growing Metaspace. You should figure out what to set that to based on profiling your build with jcmd VM.metaspace [[10]] and observing the peak usage.

Update: JVM docs definition for `-XX:MetaspaceSize` [[12]]

Sets the size of the allocated class metadata space that will trigger a garbage collection the first time it is exceeded. This threshold for a garbage collection is increased or decreased depending on the amount of metadata used. The default size depends on the platform.

Therefore we really shouldn't be setting this and allow the JVM to perform GC at regular healthy intervals in metaspace to remove unused references.

If you're not using JDK 17+ or a recent version of Gradle you may still be suffering from ClassLoader leak bugs and it still makes sense to have -XX:MaxMetaspaceSize set, but add a TODO to upgrade your JDK and remove this setting eventually when you've been able to confirm the issue is resolved. Your build system and fellow teammates will thank you for a better and more reliable developer experience.

If you're not an Android developer you probably still use some framework. Spring is super common in JVM backend systems and it also usually is built with a sizable list of dependencies. The above recommendations likely apply to you, especially if you're using Gradle to run your builds.

You could also go further and monitor jcmd VM.metaspace output for the committed class and non-class space in a build. I'll cover how to set that up properly in another blog post.

Thank You

Zac Sweers, Nicklas Ansman, and Inaki Villar for the reviews and kind encouragement!

References:

1. Red Hat JVM Engineer, OpenJDK contributor [[1]]
2. "What is Metaspace?" by Thomas Stüfe [[2]]
3. JEPS-122: Removing PermGen [[3]]
4. Gradle Metaspace Initial Commit [[4]]
5. Gradle issue tracker conversation - default metaspace not applied [[5]]
6. "Garbage Collection in Java" by Java Latte [[6]]
7. JEPS-387: Fixing Returning Memory for Metaspace [[7]]
8. "What is Compressed Class Space?" by Thomas Stüfe [[8]]
9. JDK11 Source Code: Memory Allocation Structs in Metaspace [[9]]
10. "Analyzing Metaspace with jcmd" by Thomas Stüfe [[10]]
11. "Sizing Metaspace: Metaspace default size" by Thomas Stüfe [[11]]
12. Java Platform, Standard Edition Tools Reference [[12]]

[[1]]: Red Hat JVM Engineer, OpenJDK contributor https://stuefe.de/about/

[[2]]: "What is Metaspace?" by Thomas Stüfe https://stuefe.de/posts/metaspace/what-is-metaspace/

[[3]]: "JEP-122: Remove the Permanent Generation" https://openjdk.org/jeps/122

[[4]]: "Lower default memory limits for Gradle processes" https://github.com/gradle/gradle/pull/7343

[[5]]: StackOverflow - Gradle Engineer recommends Gradle defaults https://stackoverflow.com/questions/54045132/after-upgrading-to-gradle-5-and-android-plugin-3-3-my-build-fails-with-metaspa

[[6]]: "Garbage Collection in Java" by Java Latte https://java-latte.blogspot.com/2013/08/garbage-collection-in-java.html

[[7]]: "JEP 387: Elastic Metaspace" - Proposal for Returning Memory https://openjdk.org/jeps/387

[[8]]: "What is Compressed Class Space?" by Thomas Stüfe https://stuefe.de/posts/metaspace/what-is-compressed-class-space/

[[9]]: JDK11 Source Code: Memory Allocation Structs in Metaspace https://hg.openjdk.org/jdk/jdk11/file/1ddf9a99e4ad/src/hotspot/share/memory/allocation.hpp#l239

[[10]]: "Analyzing Metaspace with jcmd" by Thomas Stüfe https://stuefe.de/posts/metaspace/analyze-metaspace-with-jcmd

[[11]]: "Sizing Metaspace: Metaspace default size" by Thomas Stüfe https://stuefe.de/posts/metaspace/sizing-metaspace/#metaspace-default-size

[[12]]: Java Platform, Standard Edition Tools Reference https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html#BABFAFAE