Repository navigation
CUDA not available when calling backwards before using CUDA #1198
Description
When I use TorchSharp and I perform a gradient calculation before loading in the CUDA binaries, then TorchSharp no longer recognizes that CUDA is available.
For example:
var lin = torch.nn.Linear(10, 1, false);
lin.forward(torch.rand(10)).backward();
Console.WriteLine(torch.cuda.is_available()); // FalseOn the other hand, if I call a CUDA function before the backward() call, it will work:
Console.WriteLine(torch.cuda.is_available()); // True
var lin = torch.nn.Linear(10, 1, false);
lin.forward(torch.rand(10)).backward();
Console.WriteLine(torch.cuda.is_available()); // TrueI've traced this issue back to LibTorch, over here. The exact quote is:
// NB: The once_flag here implies that if you try to call any CUDA
// functionality before libATen_cuda.so is loaded, CUDA is permanently
// disabled for that copy of ATen.@NiklasGustafsson How do you think this should be handled?
In PyTorch I'm pretty sure that they load in the CUDA binaries with the first import torch call. This would be easy to add to TorchSharp, but is that the desired behavior?
Activity
[-]CUDA not available when calling ` backwards` fore using CUDA[/-][+]CUDA not available when calling ` backwards` before using CUDA[/+]NiklasGustafsson commentedon Jan 16, 2024
Hmm... I think we just need to place a call to 'cuda.is_available()' on some path that is guaranteed to be run before any real work is done. A static initializer block (is that what they're called? I use them so rarely) somewhere in the
torchstatic class, perhaps? What do you think, @shaltielshmid?shaltielshmid commentedon Jan 16, 2024
I was actually thinking we can make torch.CUDA be a field (like torch.CPU, torch.META, etc.). This would require adding an if and replacing it with null, if it doesn't exist.
Alternatively, a static initializer is also a great solution. It would guarantee that cuda would be initialized before any use of torch operations.
NiklasGustafsson commentedon Jan 16, 2024
We need something that isn't going to throw an exception if the CUDA backend isn't there. Calling cuda.is_available() is safe that way.
shaltielshmid commentedon Jan 16, 2024
I was thinking of this:
But I'm not set on it. Should I create a PR with the static initializer?
NiklasGustafsson commentedon Jan 16, 2024
As long as the solution does not throw in the absence of a CUDA backend, I'm okay with it. The field should be read-only, right?
shaltielshmid commentedon Jan 16, 2024
Yes! We should probably also set torch.CPU and torch.META to be readonly as well.
I did like your solution with
cuda.is_available() ? new... : null- any reason not to use that?NiklasGustafsson commentedon Jan 16, 2024
I'm not sure if it initializes the CUDA backend. Don't know that off the top of my head.
shaltielshmid commentedon Jan 16, 2024
It does - that's where I got the TryInitialize idea :)
NiklasGustafsson commentedon Jan 16, 2024
Great. Then calling is_available is more readable, IMO
shaltielshmid commentedon Jan 16, 2024
Great! I'll have something ready soon