Skip to content

Double check nats is shut down gracefully in docker #2282

New issue
New issue
Closed
Bug
Closed
Double check nats is shut down gracefully in docker#2282
Bug
Labels
Priority:p1-urgentUrgent priorityType:BugType:MaintenanceE.g. technical debt, packaging, etc.

Description

@butonic
butonic
opened on Feb 5, 2026

I saw a corrupted KV_ids-storage-users stream. opencloud would consume all available CPU and memory. AFAICT this only happens when nats is killed while writing to disk. This is unrelated to the Jepson report and more of our own doing I fear. My guess is that we are still not gracefully shutting down nats when the opencloud process receives the SIGTERM from docker. In kubernetes nats is deployed as a dedicated service so this is not as much of a problem there.

We did look into the graceful shutdown but there are still corner cases.

Maybe there is still an exit(0) somewhere that kills nats while it is writing the stream.

👀

well it is not an exit(0) but the suture runtime does not seem to wait for all services. I have patched the main.go to panic on exit and wait 15 seconds when the command has finished. I also wanted to know when nats is being shut down gracefully:

diff --git a/opencloud/cmd/opencloud/main.go b/opencloud/cmd/opencloud/main.go
index 7968efc1d87..9c2022982fc 100644
--- a/opencloud/cmd/opencloud/main.go
+++ b/opencloud/cmd/opencloud/main.go
@@ -3,13 +3,22 @@ package main
 import (
        "fmt"
        "os"
+       "time"
 
        "github.com/opencloud-eu/opencloud/opencloud/pkg/command"
 )
 
+var osExit = os.Exit
+
 func main() {
+       osExit = func(code int) {
+               panic(fmt.Sprintf("os.Exit(%d) called", code))
+       }
        if err := command.Execute(); err != nil {
                fmt.Fprintln(os.Stderr, err.Error())
                os.Exit(1)
        }
+       fmt.Println("waiting 15 sec")
+       time.Sleep(15 * time.Second)
+       fmt.Println("program exit")
 }
diff --git a/services/nats/pkg/command/server.go b/services/nats/pkg/command/server.go
index 342036167e8..9deec13b3af 100644
--- a/services/nats/pkg/command/server.go
+++ b/services/nats/pkg/command/server.go
@@ -5,6 +5,7 @@ import (
        "crypto/tls"
        "fmt"
        "os/signal"
+       "time"
 
        "github.com/opencloud-eu/opencloud/pkg/config/configlog"
        pkgcrypto "github.com/opencloud-eu/opencloud/pkg/crypto"
@@ -91,7 +92,10 @@ func Server(cfg *config.Config) *cobra.Command {
                        gr.Add(runner.New(cfg.Service.Name+".svc", func() error {
                                return natsServer.ListenAndServe()
                        }, func() {
+                               logger.Info().Msg("gracefully stopping nats server")
                                natsServer.Shutdown()
+                               time.Sleep(5 * time.Second)
+                               logger.Info().Msg("nats server gracefully stopped")
                        }))
 
                        grResults := gr.Run(ctx)

and this is the output when sending SIGTERM:

...
2026-02-05T17:12:56+01:00 INF http server listening at http://127.0.0.1:9140 '' '' host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:124 pkg=rhttp service=frontend
2026-02-05T17:12:56+01:00 WRN core access token not set host.name=wrecker line=/home/jfd/Repositories/reva/internal/http/interceptors/auth/auth.go:248 pkg=rhttp service=frontend traceid=290edc069b0d5925f452194e8f67c4ab
2026-02-05T17:12:56+01:00 WRN http end="05/Feb/2026:17:12:56 +0100" host=127.0.0.1 host.name=wrecker line=/home/jfd/Repositories/reva/internal/http/interceptors/log/log.go:112 method=GET pkg=rhttp proto=HTTP/1.1 service=frontend size=0 start="05/Feb/2026:17:12:56 +0100" status=401 time_ns=111219 traceid=290edc069b0d5925f452194e8f67c4ab uri=/ url=/
2026-02-05T17:13:02+01:00 INF context cancelled line=/home/jfd/Repositories/opencloud/services/frontend/pkg/command/events.go:102 service=frontend
2026-02-05T17:13:02+01:00 INF stopping event handler line=/home/jfd/Repositories/opencloud/services/frontend/pkg/command/server.go:94 service=frontend
2026-02-05T17:13:02+01:00 WRN received signal line=/home/jfd/Repositories/opencloud/pkg/log/logrus_wrapper.go:50 service=idm signal=15
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=app-provider
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=http service=frontend
2026-02-05T17:13:02+01:00 INF clean server shutdown start line=/home/jfd/Repositories/opencloud/pkg/log/logrus_wrapper.go:50 service=idm
2026-02-05T17:13:02+01:00 INF gracefully stopping nats server line=/home/jfd/Repositories/opencloud/services/nats/pkg/command/server.go:95 service=nats
2026-02-05T17:13:02+01:00 INF service "appprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=app-provider
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=storage-publiclink
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=app-provider
2026-02-05T17:13:02+01:00 INF context cancelled line=/home/jfd/Repositories/opencloud/services/graph/pkg/service/v0/service.go:581 service=graph
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=http service=ocm
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=http service=storage-system
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=sharing
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=auth-app
2026-02-05T17:13:02+01:00 INF Context canceled. Shutting down event handler line=/home/jfd/Repositories/opencloud/services/storage-users/pkg/event/service.go:51 service=storage-users
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-publiclink
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=auth-basic
2026-02-05T17:13:02+01:00 INF service "data" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=storage-system
2026-02-05T17:13:02+01:00 INF service "ocm" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=ocm
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-app
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=app-registry
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=auth-machine
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=storage-system
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9216 network.transport=tcp protocol=http service=storage-system
2026-02-05T17:13:02+01:00 INF service "data" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=ocm
2026-02-05T17:13:02+01:00 INF service "storageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=groups
2026-02-05T17:13:02+01:00 INF service "gateway" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-machine
2026-02-05T17:13:02+01:00 INF service "userprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "applicationauth" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-app
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=gateway
2026-02-05T17:13:02+01:00 INF service "app" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF service "publicstorageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-publiclink
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=auth-machine
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=users
2026-02-05T17:13:02+01:00 INF service "appregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=app-registry
2026-02-05T17:13:02+01:00 INF waiting for services to exit line=/home/jfd/Repositories/opencloud/pkg/log/logrus_wrapper.go:50 service=idm
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=http service=storage-users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9246 network.transport=tcp protocol=grpc service=auth-app
2026-02-05T17:13:02+01:00 INF service "authregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "userprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9242 network.transport=tcp protocol=grpc service=app-registry
2026-02-05T17:13:02+01:00 INF service "" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=ocm
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=users
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=auth-service
2026-02-05T17:13:02+01:00 INF service "sciencemesh" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=ocm
2026-02-05T17:13:02+01:00 INF service "groupprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=groups
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=ocm
2026-02-05T17:13:02+01:00 INF service "permissions" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "publicshareprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=sharing
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-service
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9178 network.transport=tcp protocol=grpc service=storage-publiclink
2026-02-05T17:13:02+01:00 INF service "storageregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "data" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=storage-users
2026-02-05T17:13:02+01:00 INF Shutting down server line=/home/jfd/Repositories/opencloud/services/activitylog/pkg/command/server.go:142 server=activitylog service=activitylog transport=http
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=auth-service
2026-02-05T17:13:02+01:00 INF service "archiver" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF starting graceful shutdown line=/home/jfd/Repositories/opencloud/opencloud/pkg/runtime/service/service.go:526 service=opencloud
2026-02-05T17:13:02+01:00 INF service "data" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9280 network.transport=tcp protocol=http service=ocm
2026-02-05T17:13:02+01:00 INF service "storageregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=gateway
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9158 network.transport=tcp protocol=http service=storage-users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9160 network.transport=tcp protocol=grpc service=groups
2026-02-05T17:13:02+01:00 INF service "usershareprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=sharing
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=storage-users
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF service "gateway" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=gateway
2026-02-05T17:13:02+01:00 INF service "ocminvitemanager" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF service "ocmproviderauthorizer" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-basic
2026-02-05T17:13:02+01:00 INF graceful shutdown done duration=0.065834 line=/home/jfd/Repositories/opencloud/opencloud/pkg/runtime/service/service.go:587 service=opencloud
2026-02-05T17:13:02+01:00 INF service "authregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=gateway
2026-02-05T17:13:02+01:00 INF service "ocs" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF service "ocmshareprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF service "ocmcore" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
waiting 15 sec
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9142 network.transport=tcp protocol=grpc service=gateway
2026-02-05T17:13:02+01:00 INF service "" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF service "storageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9215 network.transport=tcp protocol=grpc service=storage-system
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9146 network.transport=tcp protocol=grpc service=auth-basic
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=storage-shares
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9157 network.transport=tcp protocol=grpc service=storage-users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9150 network.transport=tcp protocol=grpc service=sharing
2026-02-05T17:13:02+01:00 INF service "storageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9140 network.transport=tcp protocol=http service=frontend
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9282 network.transport=tcp protocol=grpc service=ocm
2026-02-05T17:13:02+01:00 INF service "sharesstorageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-shares
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=storage-shares
2026-02-05T17:13:02+01:00 INF Initiating Shutdown... line=/home/jfd/Repositories/opencloud/services/nats/pkg/logging/nats.go:21 service=nats
2026-02-05T17:13:02+01:00 INF Initiating JetStream Shutdown... line=/home/jfd/Repositories/opencloud/services/nats/pkg/logging/nats.go:21 service=nats
2026-02-05T17:13:02+01:00 INF JetStream Shutdown line=/home/jfd/Repositories/opencloud/services/nats/pkg/logging/nats.go:21 service=nats
2026-02-05T17:13:02+01:00 INF Server Exiting.. line=/home/jfd/Repositories/opencloud/services/nats/pkg/logging/nats.go:21 service=nats
2026-02-05T17:13:03+01:00 INF clean server shutdown complete, exiting line=/home/jfd/Repositories/opencloud/pkg/log/logrus_wrapper.go:50 service=idm
2026-02-05T17:13:07+01:00 INF nats server gracefully stopped line=/home/jfd/Repositories/opencloud/services/nats/pkg/command/server.go:98 service=nats
2026-02-05T17:13:10+01:00 ERR failed to get consumer error="context deadline exceeded" line=/home/jfd/Repositories/opencloud/services/search/pkg/service/event/service.go:216 service=search
2026-02-05T17:13:10+01:00 ERR failed to get consumer error="context deadline exceeded" line=/home/jfd/Repositories/opencloud/services/postprocessing/pkg/service/service.go:439 service=postprocessing
2026-02-05T17:13:15+01:00 ERR failed to get consumer error="context deadline exceeded" line=/home/jfd/Repositories/opencloud/services/search/pkg/service/event/service.go:216 service=search
2026-02-05T17:13:15+01:00 ERR failed to get consumer error="context deadline exceeded" line=/home/jfd/Repositories/opencloud/services/postprocessing/pkg/service/service.go:439 service=postprocessing
{"level":"error","pid":330462,"error":"eu.opencloud.api.gateway: service not found","time":"2026-02-05T17:13:17+01:00","caller":"/usr/lib/go/src/runtime/asm_amd64.s:1693","message":"error registering app provider: could not get gateway client"}
program exit

we definitely are not waiting for all services to shut down 😞

Activity

  1. butonic
    added
    Type:Bug
    on Feb 5, 2026
  2. github-project-automation
    added this to OpenCloud Team Boardon Feb 5, 2026
  3. github-project-automation
    moved this to Qualification in OpenCloud Team Boardon Feb 5, 2026
  4. rhafer
    moved this from Qualification to Prio <= 3 in OpenCloud Team Boardon Feb 9, 2026
  5. rhafer
    moved this from Prio <= 3 to Prio 2 in OpenCloud Team Boardon Feb 9, 2026
  6. rhafer
    added
    Priority:p2-highHigh priority
    on Feb 9, 2026
  7. db-ot
    added the
    Bug
    issue type on Feb 9, 2026
  8. dragonchaser
    self-assigned this
    on Feb 10, 2026
  9. dragonchaser
    moved this from Prio 2 to In Progress in OpenCloud Team Boardon Feb 10, 2026
  10. dragonchaser
    added
    Type:MaintenanceE.g. technical debt, packaging, etc.
    on Feb 11, 2026

  11. dragonchaser commented on Mar 3, 2026

    @dragonchaser
    dragonchaser
    on Mar 3, 2026
    Member

    I have tried to rewrite it with rungroups somewhat compatible to suture, without success (fails to properly connect to grpc). I would recommend a complete rewrite (see #2323). Dropping this for now.

    We need to have a chat about this @butonic

  12. dragonchaser
    removed their assignment
    on Mar 3, 2026
  13. dragonchaser
    pinned and unpinned a comment on Mar 3, 2026
  14. dragonchaser
    moved this from In Progress to Prio 2 in OpenCloud Team Boardon Mar 3, 2026

  15. 6 remaining items

  16. dragonchaser commented on Apr 20, 2026

    @dragonchaser
    dragonchaser
    on Apr 20, 2026
    Member

    There is a Hotfix-PR that implements above mentioned patch, see #2631

  17. dragonchaser
    added 2 commits that reference this issue on Apr 21, 2026

    Hotfix for opencloud-eu#2282

    3961f4c

    Hotfix for #2282

    c329439

  18. dragonchaser commented on Apr 23, 2026

    @dragonchaser
    dragonchaser
    on Apr 23, 2026
    Member

    Hotfix is merged, we should prioritize replacement of suture in the next sprint.

  19. dragonchaser commented on Apr 23, 2026

    @dragonchaser
    dragonchaser
    on Apr 23, 2026
    Member

    @micbar how to progress here? I am not really a fan of backporting the hotfix. I suggest reimplementing the initsystem before the next prod-release. I would not suggest backporting this either, everyone running into this issue should upgrade to the corresponding version with the new init system then.

  20. micbar commented on Apr 23, 2026

    @micbar
    micbar
    on Apr 23, 2026
    Member

    How would we help people on the lts release?

  21. dragonchaser
    added a commit that references this issue on Apr 24, 2026

    Hotfix for #2282

    5e88961
  22. rhafer
    added a commit that references this issue on Apr 28, 2026

    Revert "Hotfix for #2282"

    c7eeb89
  23. rhafer
    closed this as completedin 7a14978on Apr 28, 2026
  24. rhafer
    added 2 commits that reference this issue on Apr 28, 2026

    Revert "Hotfix for #2282"

    15fc596

    fix: avoid suture.Supervisor being shutdown early

    8ec8233
  25. sdwilsh
    added a commit that references this issue on May 16, 2026

    chore(deps): update opencloudeu/opencloud-rolling docker tag to v6.2.0

    a7c43b9
  26. flash7777
    added 3 commits that reference this issue on Oct 13, 2026

    Hotfix for opencloud-eu#2282

    1c5b7ee

    Revert "Hotfix for opencloud-eu#2282"

    6d4de0d

    fix: avoid suture.Supervisor being shutdown early

    f468944
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Priority:p1-urgentUrgent priorityType:BugType:MaintenanceE.g. technical debt, packaging, etc.

    Type

    Bug

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions