You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This one is a bit tricky to judge because it is quite noisy. Note that the runs that failed (x in the plot) are not used to compute the mean that we use for the analysis. This is running with runtime 0.0.4 (no packages changed, except maybe coiled), which leads me to think that this regression report is a result of noise.
The only thing that makes me doubt is that in upstream about the same date we see an increment in durations, but not so bad yet to call it regression.
Upstream:
On upstream, even though not reported as regression since the signal it's very noisy, wall time seems to be better since this was reported, Although the average memory seems to be increasing.
@gjoseph92 any thoughts on this? trying to asses whether I should close this, or keep it open.
However, the movement is the opposite of what I'd expect. The first PR should have made runtime go down from baseline, then the follow-up should have made it go back up to baseline. Unless the "fix" really didn't work properly.
It's kind of unbelievable how much noisiness and variance there is on 0.0.4. I would say that this workload is simply unrunnable on those older versions.
It's kind of unbelievable how much noisiness and variance there is on 0.0.4. I would say that this workload is simply unrunnable on those older versions.
Activity
ncclementi commentedon Sep 2, 2022
This one is a bit tricky to judge because it is quite noisy. Note that the runs that failed (x in the plot) are not used to compute the mean that we use for the analysis. This is running with runtime 0.0.4 (no packages changed, except maybe coiled), which leads me to think that this regression report is a result of noise.
The only thing that makes me doubt is that in upstream about the same date we see an increment in durations, but not so bad yet to call it regression.
Upstream:
@ian-r-rose do you have any thoughts?
ncclementi commentedon Sep 6, 2022
On upstream, even though not reported as regression since the signal it's very noisy, wall time seems to be better since this was reported, Although the average memory seems to be increasing.
@gjoseph92 any thoughts on this? trying to asses whether I should close this, or keep it open.
gjoseph92 commentedon Sep 6, 2022
I wonder if this is due to
However, the movement is the opposite of what I'd expect. The first PR should have made runtime go down from baseline, then the follow-up should have made it go back up to baseline. Unless the "fix" really didn't work properly.
I also would expect it to be affected by dask/distributed#6975.
It's kind of unbelievable how much noisiness and variance there is on 0.0.4. I would say that this workload is simply unrunnable on those older versions.
ncclementi commentedon Sep 6, 2022
Sure, but the pictures on my latest comment are from upstream (see here https://coiled.github.io/coiled-runtime/coiled-upstream-py3.9.html) and I see that wall time went down 09/02 but avg memory went up is this expected
ncclementi commentedon Nov 2, 2022
Closing, as current upstream looks like
