[Bug] After long running Hugegraph, hg-server container status turned unhealty

[neoblackcap]

Bug Type (问题类型)

server status (启动/运行异常)

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

• Server Version: 1.7.0 (Apache Release Version)
• Backend: HStore x 1, SSD
• OS: 4 vCPUs, 8 G RAM, Alinux 4
• Data Size: 100k edges

Expected & Actual behavior (期望与实际表现)

Server should run in healthy status. If anything wrong happened, it should make some log. But I use docker logs command got nothing back.

Is there anything logging relative configuration exists? I wish I could use docker logs or some way to visit verbose server log

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

[dosubot]

@neoblackcap The Docker health check hits curl http://localhost:8080/versions [1], so "unhealthy" means that endpoint stopped responding. A few things to check:

Accessing logs — docker logs only shows console (stdout) output. HugeGraph writes its main logs to files inside the container at /hugegraph-server/logs/. To read them:

# check the server log directly
docker exec <container> cat /hugegraph-server/logs/hugegraph-server.log

# or tail it
docker exec <container> tail -500 /hugegraph-server/logs/hugegraph-server.log

You can also mount the logs directory to the host for easier access: -v /host/path:/hugegraph-server/logs [2].

Enabling verbose logging — The logging config is at /hugegraph-server/conf/log4j2.xml [3]. You can mount a custom version and change the root logger level from INFO to DEBUG to get more detail [4].

Common causes of unhealthy after long running — With small data (100k edges) and 8GB RAM, memory exhaustion is less likely, but there are known scenarios where accumulated task results or stuck tasks prevent GC from reclaiming memory, eventually making the server unresponsive [5] [6]. Also, there is a reported HStore isolation bug in 1.7.0 that could cause unexpected behavior with HStore backend [7].

I'd suggest starting by checking the server log file inside the container — that should reveal what happened around the time the health check started failing.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[bitflicker64]

Hi @neoblackcap — there are two separate things going on here.

1. Why docker logs shows nothing on 1.7.0. In the 1.7.0 image the server writes its logs to a file inside the container (/hugegraph-server/logs/) instead of stdout, so docker logs is empty by design. This was fixed in #2980 (commit c0a2b93), which landed after the 1.7.0 release. I checked Docker Hub to be sure: the 1.7.0 image was last rebuilt on 2026-03-22, still before that fix (2026-04-05), so no 1.7.0 image has working docker logs — pulling 1.7.0 again won't help. The fix is only in the latest image (pushed 2026-07-21), and will ship in 1.8.0.

Two options:

• Pull hugegraph/hugegraph:latest and docker logs <container> will show the server logs.
• Stay on 1.7.0 and read the log file directly:

  docker exec <container> tail -500 /hugegraph-server/logs/hugegraph-server.log
  

  (or mount it out with -v /host/logs:/hugegraph-server/logs).

2. Why the container turns unhealthy. Note that the logging fix only makes logs visible — it doesn't change why the container goes unhealthy. The healthcheck is curl -fsS http://localhost:8080/versions, so "unhealthy" means the REST server stopped answering. To find the cause, grab the server log around when it started failing, plus:

docker stats <container> --no-stream        # memory/GC pressure
docker exec <container> jstack 1 > td.txt    # is the JVM stuck?

Common culprits after long runs are JVM/GC pressure (heap ≈ MaxRAMPercentage=50) and accumulated task results; there are also a few open HStore issues in 1.7.0 (e.g. #3095) — not necessarily related, but worth ruling out. Sharing that log would help us pinpoint it.

[contrueCT]

Thanks for reporting this issue.

There appear to be two separate problems involved here:

1. docker logs shows no HugeGraph Server output

   In HugeGraph 1.7.0, the startup script redirects JVM output to files inside the container, and the Log4j2 console appender is not connected to the main HugeGraph logger. This has already been fixed by:

   fix(docker): enable docker logs for pd/store/server containers #2980

   After this change, Docker deployments can access application logs directly through docker logs.

2. The container remains running but becomes unhealthy

   HugeGraph 1.7.0 uses tail -f /dev/null in the Docker entrypoint to keep the container alive. If the HugeGraph Java process exits, the tail process continues running, so the container remains in the Up (unhealthy) state and Docker's restart policy is not triggered.

   This container supervision problem has already been fixed by:

   fix(docker): supervise Java process in entrypoints instead of tail -f /dev/null #3051

   The updated entrypoint monitors the actual Java process and exits when that process exits, allowing Docker to restart the container according to its restart policy.

   Dockerfile-level health checks were also added in:

   chore(docker): add HEALTHCHECK & clean Dockerfiles #3052

These PRs fix the missing Docker logs and the behavior where a failed Java process leaves the container permanently unhealthy. They have been merged into the ASF master branch and will be included in a future official release.

However, the information currently available is not sufficient to determine why the Java process exited or why the REST endpoint stopped responding in this particular deployment. It could be an OOM, prolonged GC, a deadlock, thread-pool exhaustion, or another HStore-related failure.

For HugeGraph 1.7.0, please check the log file inside the container:

docker exec <container> tail -500 /hugegraph-server/logs/hugegraph-server.log

It would also be helpful to provide the following information from around the time the container becomes unhealthy:

docker stats <container> --no-stream
docker exec <container> ps -ef
docker exec <container> jstack <java-pid>
docker inspect <container> --format='{{json .State}}'

With the server log and thread dump, we can determine whether an additional server or HStore fix is required.