Repository navigation
Measure CF-D upgrades appropriately with uptimer #1176
Description
CF-D pipelines currently use uptimer to capture certain measurements relating to downtime during CF-D upgrades, and to fail them when those measurements exceed certain thresholds. However, due the retry logic we've added to deploys, the thresholds are not being applied appropriately – uptimer may fail a successful deploy, then on a retry the deploy will essentially be a no-op, resulting in no downtime for uptimer to fail on because no change occurred.
### Tasks
- [ ] Reinvestigate and update the thresholds we set on upgrades.
- [x] Remove retry logic from (upgrade?) deploys to ensure that uptimer works well.
- [x] Set reasonable value for `APP_STATS_THRESHOLD` in uptimer config
- [ ] Update uptimer errors post-tear down to improve clarity
Activity
jochenehret commentedon May 13, 2024
Good point. I think we should remove the "retry" for the "bosh-deploy-cf-develop" tasks in the "upgrade-deploy" and "experimental-deploy" jobs.
The first attempts of "bosh-deploy-cf-develop" have indeed been failing often recently, mostly because of the newly introduced "Stats availability" test: cloudfoundry/uptimer#115.
The current default value for
APP_STATS_THRESHOLDis set to 0: https://github.com/cloudfoundry/cf-deployment-concourse-tasks/blob/bbca3545a2a9781f6aa8635b9440a664f184a848/bosh-deploy/task.yml#L113We should increase this value slightly. Not sure however what a reasonable threshold might be.
jochenehret commentedon May 13, 2024
Opened PR: #1177
ctlong commentedon May 14, 2024
@jochenehret called out that uptimer emits failure messages pretty consistently during the tear down, e.g.
[UPTIMER] 2024/05/08 20:09:08 FAILURE (App pushability, 2/10): exit status 1, which are confusing coming right after the measurement summaries.We should improve those messages, either
I've added a task to the issue description for this.