STF-383: Fix off-heap memory growth in FileMode.MEMORY

[oschwald]

Summary

• Avoid off-heap buffer cache growth in MEMORY mode — FileChannel.read() into a heap buffer populates sun.nio.ch.Util.BufferCache, a per-thread cache that retains the largest direct buffer ever requested for the JVM lifetime. Under chunked MEMORY mode that meant chunkSize × loading-threads of off-heap pinned permanently. Switching to FileInputStream bypasses the cache. FileMode.MEMORY_MAPPED is unaffected (it uses FileChannel.map, not read).
• Test MEMORY mode under chunkSizes matrix — adds Reader(File, FileMode, int) package-private ctor and a testMemoryMode parametrized over the existing chunkSizes() provider, so the multi-chunk + remainder branch in BufferHolder now has integration coverage.
• Add mise configuration — mise.toml / mise.lock for local Java + Maven version management, matching minfraud-api-java. CI uses actions/setup-java directly and is unaffected.
• Preparing for 4.1.0 — version bump + changelog entry.

Note: the linked Linear ticket suggested Files.readAllBytes() as the fix, but that wouldn't work — Files.readAllBytes is implemented as Channels.newInputStream(FileChannel) and still routes through FileChannel.read → BufferCache. Only java.io.FileInputStream (whose native read(byte[]) copies via SetByteArrayRegion without going through NIO) actually bypasses the cache. This was previously commented on STF-383.

Test plan

• mvn test (199/199 pass, including 3 new chunked MEMORY-mode test runs)
• mvn checkstyle:check clean
• CI matrix passes (Java 17/21/24 × Linux/Windows/macOS)

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request addresses an issue with unbounded off-heap memory growth in FileMode.MEMORY by replacing FileChannel.read() with FileInputStream. This change prevents the JDK from caching large direct ByteBuffers in per-thread storage (sun.nio.ch.Util.BufferCache). The update also includes a version bump to 4.1.0, the addition of mise configuration for development environment management, and new tests for memory mode. Feedback was provided regarding potential integer overflow when calculating chunk counts for extremely large databases, which could lead to an incorrectly sized buffers array.

[gemini-code-assist]

The calculation of fullChunks and totalChunks uses an explicit cast from long to int without checking for overflow. If the database size is extremely large relative to the chunkSize (e.g., a multi-terabyte database with a small custom chunkSize), fullChunks could overflow, leading to an incorrectly sized buffers array and potential data corruption or IndexOutOfBoundsException during the reading loop. While MaxMind databases are typically within int range for chunk counts, it is safer to validate this or use Math.toIntExact.

                    var fullChunksLong = size / chunkSize;
                    var remainder = (int) (size % chunkSize);
                    var totalChunksLong = fullChunksLong + (remainder > 0 ? 1 : 0);
                    if (totalChunksLong > Integer.MAX_VALUE) {
                        throw new IOException("Database size exceeds supported chunk count for the given chunkSize");
                    }
                    var totalChunks = (int) totalChunksLong;
                    var fullChunks = (int) fullChunksLong;
                    var buffers = new ByteBuffer[totalChunks];

[oschwald]

(Claude replying on Greg's behalf.)

Thanks for flagging — leaving the cast unchanged in this PR for a few reasons:

1. Pre-existing, not introduced by this PR. The same (int) (size / chunkSize) pattern (and the parallel (int) (size % chunkSize)) was in the prior multi-chunk MEMORY path before the refactor, so this isn't a regression we're shipping.
2. Not practically reachable. A real MaxMind DB at multi-TB sizes doesn't exist; well before this cast would overflow we'd hit byte[] size limits (Integer.MAX_VALUE), ByteBuffer.allocate(int)'s int cap, and JVM heap limits.
3. Project guidance (CLAUDE.md) says to not add validation for scenarios that can't happen — at this layer, chunkSize is set by the library, not external input.

If/when very-large-DB support beyond ~2GB chunks becomes a goal, the right move is probably a broader audit of the int casts and array indexing throughout BufferHolder / MultiBuffer, not just Math.toIntExact in this one spot. Out of scope for the current PR.

[horgh]

Interesting:

  1. Pre-existing short-read bug, now fixed (worth calling out). The old multi-chunk MEMORY path had a latent bug:

  for (var buffer : buffers) {
      var read = channel.read(buffer);   // may be a *short* read
      if (read == -1) { break; }
      totalRead += read;
      buffer.flip();                     // flips at the short position
  }

  FileChannel.read(ByteBuffer) is not contractually obligated to fully fill the buffer — a short read would leave the chunk partially populated and flip() would expose only the
  bytes that were actually read, silently truncating the in-memory database. The new readFully loop handles this correctly. Good catch (even if accidental).

Not sure if we'd want to mention in the changelog.

[oschwald]

(Claude replying on Greg's behalf.)

Done in 232dd73. Traced the introducing commit via git log -S: the buggy for (var buffer : buffers) { channel.read(buffer); buffer.flip(); } loop was added in 2a1fa7f "Make buffers read-only for all paths" and first shipped in v4.0.0, so 4.0.0/4.0.1/4.0.2 are all affected.

Added a second bullet under the 4.1.0 section noting:

• The silent-truncation risk specifically for the multi-chunk path.
• That it only affects databases larger than ~2GB (the default chunk size).
• That the new chunked read loop retries until each chunk is fully populated.

The single-chunk MEMORY path has the same FileChannel.read(ByteBuffer) contract concern but at least throws when channel.read(buffer) != size, so only the multi-chunk path silently truncates — that's the one worth calling out.

[horgh]

  2. Minor robustness regression: size source no longer matches the open fd. BufferHolder.java:33-34:

  try (FileInputStream stream = new FileInputStream(database)) {
      long size = database.length();

  database.length() is a path-based stat(2), not a query against the open fd. The old code used channel.size() on the already-open channel, which is atomic with the open.
  Theoretical race: if the file is replaced between new FileInputStream(...) and database.length(), you could get a size that doesn't match the bytes the stream will deliver —
  and then readFully either throws "Unexpected end of stream" (truncation) or silently stops short of the file's real end (extension/replacement with a larger file). Same general
   class of race as before, but slightly wider window.

  Suggestion: use stream.getChannel().size() instead. Lazily fetching the channel doesn't populate the NIO buffer cache — only channel.read(heapBuffer) does that, and we're not
  calling it. This keeps the size lookup consistent with the open file descriptor without reintroducing the leak.

[oschwald]

(Claude replying on Greg's behalf.)

Fixed in 2624387 — database.length() → stream.getChannel().size(). Verified the suggestion doesn't reintroduce the cache: FileChannelImpl.getChannel() is lazy and .size() is an fstat-only syscall, no IOUtil.read / Util.getTemporaryDirectBuffer. Added a one-line comment noting both why (fd-atomic) and that getChannel().size() is cache-safe.