feat: add JMH benchmark module (Fesod core suite + Fesod vs POI comparison)

[GOODBOY008]

What this PR does

Adds a fesod-benchmark module with JMH benchmarks for Fesod, reworked per the review feedback on this PR: pure JMH classes only, no CI integration, no report/analysis code. Closes #572.

Suites (package org.apache.fesod.sheet.benchmark)

Suite	What it measures
FesodBenchmark	The core hot paths — read and write, XLSX and CSV, 1K/10K/100K rows × 20 columns, average time per op
FesodVsPoiBenchmark	Fesod vs Apache POI on identical data: plain write, plain read (POI's DOM-based WorkbookFactory) and batched streaming read; XLSX and XLS (XLS truncated to 65,534 data rows)

Structure

fesod-benchmark/
  benchmark.md                        # how to run, execution contracts, interpreting results
  pom.xml                             # shade -> target/benchmarks.jar; -Pbenchmark / -Pbenchmark-test
  src/main/java/.../sheet/benchmark/
    FesodBenchmark.java               # core suite
    FesodVsPoiBenchmark.java          # comparison suite
    BenchmarkConfiguration.java       # dataset sizes / file formats
    data/BenchmarkData.java           # 20-column data model
    util/DataGenerator.java           # fixed-seed, fixed date anchor (deterministic)
    util/BenchmarkFileUtil.java       # scratch files under target/benchmark-testdata
  src/test/java/.../sheet/benchmark/util/
    DataGeneratorTest.java            # determinism, date windows, row completeness
    BenchmarkFileUtilTest.java

Running

./mvnw -B -ntp -pl fesod-benchmark -am package -DskipTests

java -jar fesod-benchmark/target/benchmarks.jar FesodBenchmark -prof gc
java -jar fesod-benchmark/target/benchmarks.jar FesodVsPoiBenchmark -p fileFormat=XLSX

See fesod-benchmark/benchmark.md for the execution contracts and notes on interpreting results (JMH error bars, allocation-per-op via -prof gc, why the LARGE size matters for streaming behavior).

Notes for reviewers

• Everything CI / report / analysis related was removed: no workflow file, no committed baseline, no comparator or runner harness — benchmarks run only manually through benchmarks.jar.
• The standalone read/write/fill analysis suites that overlapped with the core suite were dropped (the fill templates also benchmarked nothing — they lacked the {...} placeholders, so the fills were no-ops).
• Review findings fixed in the surviving classes: try-with-resources for every reader/writer/workbook, fully deterministic data (a fixed date anchor replaces the previous LocalDate.now(), which broke the fixed-seed reproducibility), removal of the POI "streaming read" placeholder that was byte-identical to the plain POI read, plus naming and dead-code cleanup.
• Smoke-verified locally: all benchmark methods execute and produce sane numbers (e.g. SMALL/CSV write ≈ 5.4 ms/op, read ≈ 2.9 ms/op; on SMALL/XLSX Fesod reads ~3× and writes ~3× faster than POI in this setup).
• The earlier performance-regression-gate design (release-tag-triggered CI with a committed baseline) is parked on a side branch and will be proposed separately if/when the project wants that discussion.

[GOODBOY008]

@delei @alaahong

CI Benchmark Run Completed Successfully: https://github.com/GOODBOY008/fastexcel/actions/runs/17709908635

To view the benchmark reports:

1. Download the benchmark-results artifact from: https://github.com/GOODBOY008/fastexcel/actions/runs/17709908635/artifacts/4006114037
2. Unzip the downloaded file
3. Open benchmark-reports/benchmark-comparison.html in your browser

There are a few issues to address:

1. In the Performance Comparisons section of the HTML report, the content is incomplete. A dataset and a format column need to be added.
2. For the 1M dataset scenario, the POI run failed, so no benchmark results were generated.

[psxjoy]

I'm really excited about this PR. However, it's quite large, so the code review will take some time.

Also, no offense intended, but I'd like to ask: Did you use AI-generated code in this PR?

[GOODBOY008]

I'm really excited about this PR. However, it's quite large, so the code review will take some time.

Also, no offense intended, but I'd like to ask: Did you use AI-generated code in this PR?

@psxjoy Yes, some parts (like the comparison report, memory profiler logic, and quickstart scripts) were AI-assisted.AI is quite effective in these scenarios, I’ve verified them to make sure they work correctly.

I noticed the artifact wasn’t accessible, so I’ve uploaded the results for your review.
benchmark-results.zip

[delei]

Hi, @GOODBOY008
Thank you for submitting the PR.

Regarding this PR, I still have some questions:

• It seems that the file ./fastexcel-benchmark/scripts/benchmark-runner.sh does not exist?
• Introducing JMH benchmark testing is highly necessary, but currently we don't need to run it through CI.
• If possible, I suggest deleting the code for generating reports and analyzing results, and only keeping the JMH classes.

Please refer to the above suggestions and make appropriate modifications to the PR content. After that, we will vote on this PR together with other reviewers ASAP.

[GOODBOY008]

Hi @delei
Thanks for your feedback.

For the first point, I understand the concern about the PR size — my intention was to split the work into stages, so this submission might look a bit large.

Regarding the second and third points:
• Running benchmarks in CI helps produce relatively stable and reproducible results. Running them locally is often influenced by background tasks and can take a long time.
• As for report generation and analysis, they make it easier to compare multiple runs, especially when evaluating different scenarios. Doing this entirely by hand would be quite time-consuming.

I’m fine with keeping only the JMH core classes for now, but I’d like to highlight the above considerations.

[GOODBOY008]

@delei PTAL

[netlify]

✅ Deploy Preview for fesod ready!

Name	Link
🔨 Latest commit	3a2c467
🔍 Latest deploy log	https://app.netlify.com/projects/fesod/deploys/69240bb276a415000840ef16
😎 Deploy Preview	https://deploy-preview-575--fesod.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

This PR introduces a comprehensive JMH-based benchmark module for the FastExcel library, enabling performance testing and comparisons with Apache POI across various operations (read, write, fill) and dataset sizes. The module includes memory profiling capabilities, test data generation utilities, and comparison benchmarks to validate FastExcel's performance claims.

Changes:

• New fesod-benchmark module with complete JMH integration and Maven configuration
• Benchmark suites for read, write, and fill operations across multiple dataset sizes and file formats
• Memory profiling utilities with GC tracking and detailed statistics
• Comparison benchmarks between FastExcel and Apache POI
• Comprehensive test data generation with configurable characteristics

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
pom.xml	Added fesod-benchmark module to parent POM
fesod-benchmark/pom.xml	New Maven configuration with JMH dependencies and shade plugin
fesod-benchmark/benchmark.md	Documentation for running and interpreting benchmarks
MemoryProfiler.java	Utility for real-time memory profiling with GC tracking
DataGenerator.java	Test data generation with multiple data types and characteristics
BenchmarkFileUtil.java	File management utilities for benchmark operations
BenchmarkData.java	Data model with 20 fields covering various Excel data types
BenchmarkConfiguration.java	Configuration enums for dataset sizes and file formats
AbstractBenchmark.java	Base class providing common benchmark functionality
WriteBenchmark.java	Write operation benchmarks for different sizes and scenarios
ReadBenchmark.java	Read operation benchmarks with multiple listener patterns
FillBenchmark.java	Template fill operation benchmarks
FastExcelVsPoiBenchmark.java	Comparison benchmarks between FastExcel and Apache POI
ComparisonBenchmarkRunner.java	Runner for executing comparison benchmarks

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Variable name xlsEXTRA_LARGEFile uses inconsistent capitalization mixing underscores with camelCase. Should be xlsxLargeFile to match the naming pattern of other variables like xlsxSmallFile and xlsxMediumFile.

Suggested change

	private String xlsEXTRA_LARGEFile;
	private String xlsxLargeFile;

[Copilot]

Method name readXlsEXTRA_LARGE uses inconsistent capitalization with underscores and uppercase. Should be readXlsxLarge to follow Java naming conventions and match the pattern of other methods like readXlsxSmall and readXlsxMedium.

Suggested change

	public void readXlsEXTRA_LARGE(Blackhole blackhole) throws Exception {
	public void readXlsxLarge(Blackhole blackhole) throws Exception {

[Copilot]

Method name readXlsEXTRA_LARGEWithStreaming uses inconsistent capitalization. Should be readXlsxLargeWithStreaming to follow Java naming conventions.

Suggested change

	public void readXlsEXTRA_LARGEWithStreaming(Blackhole blackhole) throws Exception {
	public void readXlsxLargeWithStreaming(Blackhole blackhole) throws Exception {

[Copilot]

Method name readXlsEXTRA_LARGECountingOnly uses inconsistent capitalization. Should be readXlsxLargeCountingOnly to follow Java naming conventions.

Suggested change

	public void readXlsEXTRA_LARGECountingOnly(Blackhole blackhole) throws Exception {
	public void readXlsxLargeCountingOnly(Blackhole blackhole) throws Exception {

[Copilot]

Method name readXlsEXTRA_LARGECollecting uses inconsistent capitalization. Should be readXlsxLargeCollecting to follow Java naming conventions.

Suggested change

	public void readXlsEXTRA_LARGECollecting(Blackhole blackhole) throws Exception {
	public void readXlsxLargeCollecting(Blackhole blackhole) throws Exception {

[Copilot]

Potential overflow in int multiplication before it is converted to long by use in a numeric context.

Suggested change

	duration > 0 ? (rowCount * 1000 / duration) : "N/A");
	duration > 0 ? (rowCount * 1000L / duration) : "N/A");

[Copilot]

This method overrides AbstractBenchmark.tearDownBenchmark; it is advisable to add an Override annotation.

Suggested change

	@Override

[Copilot]

This method overrides AbstractBenchmark.setupBenchmark; it is advisable to add an Override annotation.

Suggested change

	protected void setupBenchmark() throws Exception {
	// Custom setup logic if needed
	}
	@Override
	protected void setupBenchmark() throws Exception {
	// Custom setup logic if needed
	}
	@Override

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (15)

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/comparison/FastExcelVsPoiBenchmark.java:290

• ExcelReader implements Closeable (close() calls finish()). If readAll() throws, finish() is skipped and the underlying stream/cache may not be released. Use try-with-resources so the reader is always closed.

        try {
            ExcelReader excelReader = EasyExcel.read(testFile, BenchmarkData.class, new ReadListener<BenchmarkData>() {
                        @Override
                        public void invoke(BenchmarkData data, AnalysisContext context) {
                            processedRows.incrementAndGet();
                            blackhole.consume(data);
                        }

                        @Override
                        public void doAfterAllAnalysed(AnalysisContext context) {
                            // Processing complete
                        }
                    })
                    .build();

            excelReader.readAll();
            excelReader.finish();

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/comparison/FastExcelVsPoiBenchmark.java:368

• Same resource-safety issue as benchmarkFesodRead: if readAll() throws, finish() is skipped and the reader isn't closed. Use try-with-resources so the reader is always closed.

        try {
            ExcelReader excelReader = EasyExcel.read(testFile, BenchmarkData.class, new ReadListener<BenchmarkData>() {
                        @Override
                        public void invoke(BenchmarkData data, AnalysisContext context) {
                            batch.add(data);
                            processedRows.incrementAndGet();

                            if (batch.size() >= batchSize) {
                                blackhole.consume(new ArrayList<>(batch));
                                batch.clear();
                            }
                        }

                        @Override
                        public void doAfterAllAnalysed(AnalysisContext context) {
                            if (!batch.isEmpty()) {
                                blackhole.consume(batch);
                                batch.clear();
                            }
                        }
                    })
                    .build();

            excelReader.readAll();
            excelReader.finish();

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/FillBenchmark.java:306

• The vertical fill template references {.priority}, but BenchmarkData has no priority property. This will either leave the cell blank or fail template resolution, so the benchmark won't reflect a real fill scenario.

        Map<String, Object> row1 = new HashMap<>();
        row1.put("A", "Dynamic Fill Test");
        row1.put("B", "Status");
        row1.put("C", "Priority");

        Map<String, Object> row2 = new HashMap<>();
        row2.put("A", "{.id}");
        row2.put("B", "{.status}");
        row2.put("C", "{.priority}");

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/FillBenchmark.java:180

• createSimpleTemplate() says it creates placeholder rows, but it writes only literal strings (no {...} placeholders). As a result, fillSimpleMap / fillSingleObject won't actually exercise template variable replacement, making the benchmark results misleading.

        // Create a simple template with placeholder rows
        Map<String, Object> row1 = new HashMap<>();
        row1.put("name", "Simple Fill Test");
        row1.put("date", "2023-01-01");
        row1.put("version", "1.0");

        Map<String, Object> row2 = new HashMap<>();
        row2.put("description", "This is a simple fill test");
        row2.put("author", "Test Author");
        row2.put("status", "Active");

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/FillBenchmark.java:335

• fillMultipleLists() fills FillWrapper("data1", ...) and FillWrapper("data2", ...), but the generated multiListTemplateFile only contains {date} / {version} placeholders and no list placeholders. The list fills will be no-ops, so this benchmark won't measure multi-list filling.

    private String createMultiListTemplate() {
        String templatePath = BenchmarkFileUtil.getTempFilePath(
                BenchmarkConfiguration.FileFormat.XLSX, BenchmarkConfiguration.DatasetSize.LARGE, "MultiListTemplate");

        // Create multi-list template
        Map<String, Object> row1 = new HashMap<>();
        row1.put("Report", "Performance Report");
        row1.put("Date", "{date}");
        row1.put("Version", "{version}");

        Map<String, Object> row2 = new HashMap<>();
        row2.put("Metric", "Value");
        row2.put("Status", "Threshold");
        row2.put("Notes", "Comments");

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/utils/BenchmarkFileUtil.java:91

• Files.walk(...) returns a Stream that must be closed; the current implementation never closes it, which can leak file handles (especially on Windows) and cause intermittent cleanup failures during repeated benchmark runs.

                Files.walk(testDataPath)
                        .filter(path -> path.getFileName().toString().startsWith("temp_"))
                        .forEach(path -> {
                            try {
                                Files.deleteIfExists(path);

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/core/AbstractBenchmark.java:187

• Files.walk(...) returns a Stream that needs to be closed. Not closing it can leak file handles during repeated benchmark runs and make temp-file cleanup flaky.

                Files.walk(outputPath)
                        .filter(path -> path.getFileName().toString().startsWith("temp_"))
                        .forEach(path -> {
                            try {
                                Files.deleteIfExists(path);

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/comparison/FastExcelVsPoiBenchmark.java:170

• ExcelWriter implements Closeable (close() calls finish()). In this benchmark, if write() or finish() throws, the writer won't be closed, which can leak temp files/streams and skew benchmark stability. Prefer try-with-resources and rely on close() for cleanup.

This issue also appears in the following locations of the same file:

• line 273
• line 343

        try {
            ExcelWriter excelWriter =
                    EasyExcel.write(outputFile, BenchmarkData.class).build();
            WriteSheet writeSheet = EasyExcel.writerSheet("TestData").build();

            excelWriter.write(testDataList, writeSheet);
            excelWriter.finish();

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/utils/DataGenerator.java:258

• DataGenerator uses a fixed Random seed for reproducibility, but the generated dates depend on LocalDate.now() / LocalDateTime.now(), so the dataset will change over time. That makes benchmark results harder to compare across runs/days even with the same seed.

    private LocalDate generateRandomDate() {
        LocalDate now = LocalDate.now();
        LocalDate fiveYearsAgo = now.minusYears(5);
        long daysBetween = java.time.temporal.ChronoUnit.DAYS.between(fiveYearsAgo, now);
        long randomDays = Math.floorMod(random.nextLong(), daysBetween);
        return fiveYearsAgo.plusDays(randomDays);
    }

    /**
     * Generate random datetime within the last year
     */
    private LocalDateTime generateRandomDateTime() {
        LocalDateTime now = LocalDateTime.now();
        LocalDateTime oneYearAgo = now.minusYears(1);
        long secondsBetween = java.time.temporal.ChronoUnit.SECONDS.between(oneYearAgo, now);
        long randomSeconds = Math.floorMod(random.nextLong(), secondsBetween);
        return oneYearAgo.plusSeconds(randomSeconds);

fesod-benchmark/pom.xml:220

• The benchmark-test profile claims to be a quick smoke test, but passing .* will run all benchmarks in the module (including the non-parameterized ones that generate large/extra-large datasets). This makes the CI/smoke profile much heavier than intended.

                                    <arguments>
                                        <argument>-f</argument>
                                        <argument>1</argument>
                                        <argument>-i</argument>
                                        <argument>1</argument>
                                        <argument>-wi</argument>
                                        <argument>0</argument>
                                        <argument>-p</argument>
                                        <argument>datasetSize=SMALL</argument>
                                        <argument>-p</argument>
                                        <argument>fileFormat=XLSX</argument>
                                        <argument>.*</argument>
                                    </arguments>

fesod-benchmark/benchmark.md:88

• benchmark.md says EXTRA_LARGE is for “comparison benchmarks only”, but the operations benchmarks (WriteBenchmark, ReadBenchmark) also use DatasetSize.EXTRA_LARGE. This is confusing for users trying to choose an appropriate benchmark size.

| `SMALL` | 1,000 | Quick development feedback |
| `MEDIUM` | 10,000 | Standard CI benchmarks |
| `LARGE` | 100,000 | Performance analysis |
| `EXTRA_LARGE` | 1,000,000 | Stress testing (comparison benchmarks only) |

pom.xml:82

• PR title/description talk about “FastExcel” and link to the fast-excel/fastexcel repo, but the actual change adds a fesod-benchmark module under org.apache.fesod.* in this repository. Please align the PR metadata (title/description) with the Fesod codebase to avoid confusion for reviewers and release notes.

        <module>fesod-shaded</module>
        <module>fesod-examples</module>
        <module>fesod-sheet</module>
        <module>fesod-benchmark</module>
    </modules>

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/WriteBenchmark.java:48

• The module and packages are all org.apache.fesod.*, but this class-level Javadoc still refers to “FastExcel”. This is confusing in the Fesod codebase and in generated benchmark reports.

/**
 * Comprehensive benchmarks for FastExcel write operations
 */

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/ReadBenchmark.java:48

• The module and packages are all org.apache.fesod.*, but this class-level Javadoc still refers to “FastExcel”. This is confusing in the Fesod codebase and in generated benchmark reports.

/**
 * Comprehensive benchmarks for FastExcel read operations
 */

fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/FillBenchmark.java:53

• The module and packages are all org.apache.fesod.*, but this class-level Javadoc still refers to “FastExcel”. This is confusing in the Fesod codebase and in generated benchmark reports.

This issue also appears in the following locations of the same file:

• line 171
• line 298
• line 322

/**
 * Comprehensive benchmarks for FastExcel fill operations
 */

[GOODBOY008]

@delei Following up on your review (#575 (comment)) — I've reworked the PR per your feedback:

• No CI runs — the workflow file is gone; the module contains no CI integration at all.
• Only JMH classes kept — the runner/comparator harness, committed baseline data, memory profiler and report generation were all removed. What remains: FesodBenchmark (read/write × XLSX/CSV × 1K/10K/100K rows), FesodVsPoiBenchmark (Fesod vs Apache POI on identical data) and their small support code (deterministic data generator + file util), runnable manually via the shaded benchmarks.jar (see fesod-benchmark/benchmark.md). The net diff vs main is now ~1.6K lines.
• I also dropped the standalone read/write/fill analysis suites that overlapped with the core suite, and fixed the review findings in the surviving classes (try-with-resources everywhere, deterministic data via a fixed date anchor, removal of the duplicate POI "streaming read" placeholder).

The regression-gate/CI idea is parked on a side branch and can be proposed separately if/when the project wants that discussion. PTAL when you have time.

[bengbengbalabalabeng]

Consider using Files.deleteIfExists(Path). This method throws an IOException with a clear cause when the deletion fails, which makes it easier to diagnose issues—especially in CI environments.

[bengbengbalabalabeng]

The same advice also applies to FesodVsPoiBenchmark.java

[bengbengbalabalabeng]

Consider using the FesodSheet API as a replacement for EasyExcel.

[bengbengbalabalabeng]

The same advice also applies to FesodVsPoiBenchmark.java

[bengbengbalabalabeng]

The time spent deleting temporary files should not be included in the current write benchmark.

It’s better to move the cleanup logic to the @TearDown phase and perform the deletion in a single batch to avoid interfering with the benchmark results.

[bengbengbalabalabeng]

The same advice also applies to FesodVsPoiBenchmark.java