Skip to content

feat: add JMH benchmark module (Fesod core suite + Fesod vs POI comparison) #575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bengbengbalabalabeng merged 4 commits into from
Oct 14, 2026
Merged

feat: add JMH benchmark module (Fesod core suite + Fesod vs POI comparison) #575

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3c62005
feat(benchmark): add JMH benchmark module
GOODBOY008 Oct 6, 2026
1084578
refactor(benchmark): use Lombok for the benchmark data model
GOODBOY008 Oct 12, 2026
ef0bd99
refactor(benchmark): adopt Lombok across the remaining support classes
GOODBOY008 Oct 12, 2026
b1b3f5d
refactor(benchmark): use 3 forks for the comparison suites too
GOODBOY008 Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions fesod-benchmark/benchmark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Fesod Benchmark Module

JMH benchmarks for Fesod spreadsheet operations. This module is a **manual tool**:
it is not wired into any CI workflow and is not part of the Fesod public API.

It covers three complementary suites (package `org.apache.fesod.sheet.benchmark`):

| Suite | What it measures |
|---|---|
| `FesodBenchmark` | The core hot paths — **read** and **write**, XLSX and CSV, 1K/10K/100K rows × 20 columns, average time per op |
| `FesodVsPoiBenchmark` | Plain-API comparison with **Apache POI** on the same data: write (`ExcelWriter` vs XSSF/HSSF `Workbook`) and read (Fesod streaming reader vs POI's DOM-based `WorkbookFactory`); XLSX and XLS (XLS truncated to 65,534 data rows) |
| `StreamingBenchmark` | Streaming-API comparison for the **large-file** scenario, XLSX only: **read** — Fesod streaming reader vs POI **SAX event model** (`XSSFReader` + `XSSFSheetXMLHandler`); **write** — Fesod batched `ExcelWriter` vs POI **SXSSF** sliding-window workbook; 1K/10K/100K rows |

Shared support code: `BenchmarkConfiguration` (dataset sizes, file formats),
`data.BenchmarkData` (the 20-column data model), `util.DataGenerator`
(fixed-seed, date-anchored — same seed always produces the same rows) and
`util.BenchmarkFileUtil` (scratch files live under `target/benchmark-testdata`,
which `mvn clean` removes).

## Building and running

Build the shaded benchmark jar, then use the standard JMH launcher:

```bash
./mvnw -B -ntp -pl fesod-benchmark -am package -DskipTests

# Whole core suite (all sizes × formats)
java -jar fesod-benchmark/target/benchmarks.jar FesodBenchmark

# One slice, e.g. LARGE XLSX reads only
java -jar fesod-benchmark/target/benchmarks.jar 'FesodBenchmark\.read$' \
-p datasetSize=LARGE -p fileFormat=XLSX

# Fesod vs POI, XLSX only
java -jar fesod-benchmark/target/benchmarks.jar FesodVsPoiBenchmark -p fileFormat=XLSX

# Streaming read for the large-file scenario: Fesod vs POI SAX event model
java -jar fesod-benchmark/target/benchmarks.jar StreamingBenchmark

# Add allocation-per-op (bytes/op) — the most direct memory-efficiency signal
java -jar fesod-benchmark/target/benchmarks.jar FesodBenchmark -prof gc
```

Maven shortcuts:

```bash
# Full run of a suite via the exec plugin
./mvnw -B -ntp -pl fesod-benchmark verify -Pbenchmark '-Dbenchmark.pattern=FesodBenchmark'

# Quick smoke run (SMALL/CSV, 1 fork, minimal iterations) — sanity check after changes
./mvnw -B -ntp -pl fesod-benchmark verify -Pbenchmark-test
```

## Execution contracts

The annotation contracts are part of each suite and should be kept stable so that
results collected at different times stay comparable:

| Suite | Mode | Forks | Warmup | Measurement | JVM args |
|---|---|---|---|---|---|
| `FesodBenchmark` | avgt, ms/op | 3 | 3 × 1s | 5 × 2s | `-Xms1g -Xmx1g -XX:+UseG1GC` |
| `FesodVsPoiBenchmark` | avgt, ms/op | 3 | 3 × 5s | 5 × 5s | `-Xms2g -Xmx2g` |
| `StreamingBenchmark` | avgt, ms/op | 3 | 3 × 5s | 5 × 5s | `-Xms2g -Xmx2g` |

All suites use 3 forks so cross-fork JVM/JIT variability is visible in the
error bars. The core suite deliberately keeps a smaller fixed heap (`1g`,
G1GC) — a roomy constant heap removes GC adaptivity from the timing signal —
while the comparison suites use `2g` so POI's in-memory workloads are not the
constraint being measured.

## Notes on interpreting results

- **Environment matters.** Time measurements fluctuate with the host (CPU,
contention, disk). Compare runs from the same machine and similar load, and
trust JMH's error bars (`score ± error`): overlapping intervals between two
runs mean "no observable difference".
- **`-prof gc` allocation per op is the most stable signal** (typically ±0.1%
run-to-run on the same machine). For Fesod's *streaming without OOM* promise,
allocation per op should stay linear in the number of rows and independent of
file size — an accidental full-file buffering shows up here as an
orders-of-magnitude jump long before timing changes.
- **LARGE (100K rows) is where streaming behavior is actually exercised.** At
1K/10K rows even a non-streaming implementation looks fine; memory
characteristics only separate at larger sizes.
- **Read-path workload symmetry (vs-POI suites).** The write comparisons are
symmetric — both libraries convert the same objects into the same 20 columns.
The read comparisons are deliberately asymmetric, mirroring each library's
idiomatic end-to-end usage: Fesod runs its full pipeline (parse + type
conversion such as `BigDecimal`/dates + reflective row-object mapping into
`BenchmarkData`), while the POI side consumes raw cell text — POI has no
built-in row-object mapping, and hand-rolling one in the benchmark would
measure benchmark code rather than either library. The asymmetry is
conservative for Fesod: the POI side does strictly less work per row, so the
comparison cannot overstate Fesod's advantage.
- **Data is deterministic**: fixed seed and a fixed date anchor
(`2024-01-01`) mean any difference between two runs comes from the code or
the environment, not from the data.

## Extending

- New benchmark methods are added like any JMH benchmark (`@Benchmark` method +
`@Param`s). Keep the class-level contract annotations unchanged when adding
to an existing suite; create a new suite class if a different contract is
needed.
- If a benchmark writes files, write them through `BenchmarkFileUtil` so they
land in `target/benchmark-testdata`. Write benchmarks target a fixed file
(overwritten each invocation) and delete it from the trial `@TearDown` via
`BenchmarkFileUtil.delete` — never inside the measured method, so cleanup
I/O stays out of the reported time.
231 changes: 231 additions & 0 deletions fesod-benchmark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,231 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.apache.fesod</groupId>
<artifactId>fesod-parent</artifactId>
<version>${revision}</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>fesod-benchmark</artifactId>
<name>fesod-benchmark</name>
<description>JMH benchmarks measuring Fesod read/write performance and comparing it with Apache POI</description>

<properties>
<jmh.version>1.37</jmh.version>
<exec-maven-plugin.version>3.1.0</exec-maven-plugin.version>
<uberjar.name>benchmarks</uberjar.name>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.fesod</groupId>
<artifactId>fesod-sheet</artifactId>
<version>${project.version}</version>
</dependency>

<!-- JMH Dependencies -->
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>provided</scope>
</dependency>

<!-- Apache POI for comparison benchmarks -->
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
</dependency>

<!-- Logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
</dependency>

<!-- Testing -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<annotationProcessorPaths>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
</path>
<path>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
</path>
</annotationProcessorPaths>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${uberjar.name}</finalName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
<filters>
<filter>
<!--
Shading signed JARs will fail without this.
http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar
-->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

<profiles>
<!-- Full benchmark run: mvn verify -Pbenchmark -Dbenchmark.pattern=.*Fesod.* -->
<profile>
<id>benchmark</id>
<properties>
<benchmark.pattern>.*</benchmark.pattern>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${exec-maven-plugin.version}</version>
<executions>
<execution>
<id>run-benchmarks</id>
<phase>integration-test</phase>
<goals>
<goal>java</goal>
</goals>
<configuration>
<mainClass>org.openjdk.jmh.Main</mainClass>
<arguments>
<argument>${benchmark.pattern}</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<!-- Quick smoke run of the core suite: minimal forks/iterations, SMALL/CSV only -->
<!-- Usage: mvn verify -Pbenchmark-test -->
<profile>
<id>benchmark-test</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${exec-maven-plugin.version}</version>
<executions>
<execution>
<id>run-benchmark-smoke-test</id>
<phase>integration-test</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>java</executable>
<arguments>
<argument>-cp</argument>
<argument>target/benchmarks.jar</argument>
<argument>org.openjdk.jmh.Main</argument>
<argument>FesodBenchmark\.</argument>
<argument>-f</argument>
<argument>1</argument>
<argument>-wi</argument>
<argument>1</argument>
<argument>-w</argument>
<argument>1s</argument>
<argument>-i</argument>
<argument>2</argument>
<argument>-r</argument>
<argument>1s</argument>
<argument>-p</argument>
<argument>datasetSize=SMALL</argument>
<argument>-p</argument>
<argument>fileFormat=CSV</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
Loading
Loading