Repository navigation
Backup fails because of a stale Restic lock #8229
Description
When the backup destination (for example a B2 or S3 bucket) is slow to respond, a module backup can leave a "stale lock" file in its Restic repository. Nothing removes such a lock automatically, so later backups of the same application can keep failing until an administrator runs restic unlock by hand.
Steps to reproduce
The problem is timing-dependent and was observed on a production node:
- Configure a scheduled backup to a cloud destination (B2/S3)
- During the backup, the remote storage becomes slow: requests take more than 30 seconds to complete
- Restic fails to create its lock with
unexpected HTTP response (504): 504 Gateway Time-out, retries for about 15 minutes, then gives up
Save(<lock/c558447a75>) returned error, retrying after 694.836292ms: unexpected HTTP response (504): 504 Gateway Time-out
[...]
Save(<lock/c558447a75>) failed: unexpected HTTP response (504): 504 Gateway Time-out
{"message_type":"exit_error","code":1,"message":"unable to create lock in backend: unexpected HTTP response (504): 504 Gateway Time-out"}
[ERROR] restic stats failed for <destination>/nethvoice-proxy/<uuid>: Command '[...'stats', '--json', 'latest']' returned non-zero exit status 1.
Save(<lock/a64a42d591>) returned error, retrying after 1.405563479s: unexpected HTTP response (504): 504 Gateway Time-out
[...]
unable to create lock in backend: unexpected HTTP response (504): 504 Gateway Time-out
[ERROR] retention policy failed for <destination>/nethvoice-proxy/<uuid>: Command '[...'forget', '--prune', '--keep-last=5']' returned non-zero exit status 1.
nethvoice-proxy6: backup 1 failed (exit code 1)
Expected behavior
A temporary problem of the remote storage makes that backup run fail, and the next scheduled run works again once the remote is healthy.
Actual behavior
Sometimes the next runs keep failing because the repository is locked (repository is already locked exclusively). The administrator must remove the lock manually.
The 504 error is generated by the HAProxy front end of the rclone-gateway service, after its 30-second timeout server. When HAProxy gives up, the lock file upload may still be in progress. After each failed attempt Restic sends a request to delete the lock file, but if the remote storage completes the upload after that delete request (or the delete request fails too), the lock file stays in the repository and Restic does not know about it:
- Restic considers a lock "stale" when it is older than 30 minutes, but it does not ignore stale locks when it acquires a new one: only
restic unlockremoves them, and NS8 never runs it. - An orphaned exclusive lock (from
forget --prune) blocks every later backup of that application. An orphaned shared lock (frombackuporstats) blocks the retention policy, so the backup is reported as failed.
The same kind of orphaned lock can also be left when Restic is interrupted (node reboot, out-of-memory, rclone-gateway restart) while it is refreshing or removing its own lock.
Proposed fix: before each module backup, run restic unlock (without --remove-all). It removes only locks older than 30 minutes; a running Restic process refreshes its lock every 5 minutes, so active locks are never removed.
As a separate improvement, the HAProxy timeout server of rclone-gateway could be raised above Restic's own 5-minute --stuck-request-timeout, so that a slow remote does not cut requests at 30 seconds.
Components
NethServer 8 core 3.23.0-dev
See also
- Restic lock implementation: https://github.com/restic/restic/blob/master/internal/repository/lock_file.go
Activity
DavidePrincipi commentedon Oct 13, 2026
In Testing 3.23.1-dev.5
Test case
See how to test section in NethServer/ns8-core#1351
DavidePrincipi commentedon Oct 13, 2026
VERIFIED -- Test report for NethServer/ns8-core#1351
Two test nodes, both with the same Nextcloud instance backing up to the same Restic repository on an S3 destination:
3.23.1-dev.4, without the fix3.23.1-dev.5, with the fixSetup: from node A, a read-only
restic checkwas started and killed after it took its exclusive lock. The lock file stayed in the repository.repository is already locked exclusively, backup failedsuccessfully removed 1 locks, backup and retention policy succeeded, backup status recorded without errorsThe test also confirms that a lock created by another node is removed once it is older than 30 minutes.
DavidePrincipi commentedon Oct 15, 2026
Test results
Tested on a single-node test cluster (core 3.23.1-dev) with NethServer/ns8-core#1351 and NethServer/ns8-core#1352 installed, backing up a Mattermost instance to Backblaze B2 and to an S3-compatible bucket (OVH).
How the slow remote storage was simulated: a small test proxy was placed inside the rclone-gateway container, between HAProxy and the rclone REST backend. It forwards every request unchanged, but delays by 45 seconds the requests that create a Restic lock. Two variants were used: hold the response after the remote storage has stored the lock, or forward the request to the remote storage late, even if Restic has already given up.
504 Gateway Time-out, thenunable to create lock in backendafter about 15 minutes: the same pattern as the production log. No lock left: Restic deleted it after each failed attempt.repository is already locked by PID 1 on <node> ... lock was created ... 23m45s ago, so the backup was reported as failed. This is the failure seen in production.successfully removed 1 locks(#1351), then the backup and the retention policy completed successfully, with no lock left.Notes:
503 Service Unavailablefor about one minute after the backend was available again, until the health check marked it up.DavidePrincipi commentedon Oct 15, 2026
Released https://github.com/NethServer/ns8-core/releases/tag/3.23.1