dotnet
torch.jit.ScriptModule with Cuda has error
Hello,
I trying to get LLMs and Visionmodels running with Torchsharp. I got quite success with this. I use huggingface and timm. But it only works with 'cpu' not with 'gpu'. I have no idea why.
greeting Michael
c# side of the coin (bad code)::: private static torch.jit.ScriptModule model = torch.jit.load("D:/Save/fbdeit_scripted.pt"); public static void Test() { using var grad = torch.no_grad(); var path = "D:\Save\MrBeam.bmp";//24 bit Bitmap var image = new DataExtraction().ExtractData(path); image.ToString( ).Show(); var resutl = (Tensor) model.forward(image.unsqueeze(0).to(float32)); // throws me an error if cuda is set ????????????????????? //var resutl = (Tensor) model.To("Cuda").forward(image.unsqueeze(0).to(float32).To("Cuda")); var clsidx = torch.argmax(resutl); ((int)clsidx).ToString().Show(); }
python side of the coin (yes really bad code) :::
import urllib from PIL import Image import torch import timm import requests import torch import torchvision.transforms as transforms from timm.data.constants import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD
model = torch.hub.load('facebookresearch/deit:main', 'deit_base_patch16_224', pretrained=True) model.eval()
url = "file:///D:/Save/Löwe.bmp" image = Image.open(urllib.request.urlopen(url))
transform = transforms.Compose([ transforms.Resize(256, interpolation=3), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD), ])
img = transform(image)
out = model.to('cuda')(img.unsqueeze(0).to('cuda')) clsidx = torch.argmax(out) out.size()
scripted_model = torch.jit.script(model.to('cpu')) scripted_model.save("D:/Save/fbdeit_scripted.pt") frozen = torch.jit.load("D:/Save/fbdeit_scripted.pt") frozen.forward(img.unsqueeze(0)) clsidx = torch.argmax(out) clsidx
Getting the picture right is pain too:::::
public Tensor ExtractData(string Path)
{
using var grad = torch.no_grad();
var InputImg = System.Drawing.Image.FromFile(Path);//24 byte bmp
var myArray = (byte[])new ImageConverter().ConvertTo(InputImg, typeof(byte[]));
var width =BitConverter.ToInt32(myArray[18..(18 + 4)]);
var height = BitConverter.ToInt32(myArray[22..(22 + 4)]);
var channels = BitConverter.ToInt32(myArray[50..(50 + 4)]);
var offset = BitConverter.ToInt32(myArray[10..(10 + 4)]);
var data = myArray[offset..];
//(H x W x C)
var tensor = torch.tensor(data, dtype: ScalarType.Int32).view(height,width,3).to(float32);
var _resize = 256;
var _inter = InterpolationMode.Bilinear;
var IMAGENET_DEFAULT_MEAN = torch.tensor( new double[] { 0.229, 0.224, 0.225 });
var IMAGENET_DEFAULT_STD = torch.tensor(new double[] { 0.485, 0.456, 0.406 });
Func<Tensor, Tensor> resize = (x) =>
{
x = x.permute(2,1,0);//(C x H x W)
var h = x.size(1);
var w = x.size(2);
var ratio = (double)_resize / Math.Min(h, w);
x = torch.nn.functional.interpolate(x.unsqueeze(0), scale_factor: new double[] { ratio, ratio }, mode: _inter, align_corners: false, recompute_scale_factor: false);
return x.squeeze(0).permute(2, 1, 0);//(H x W x C)
};
Func<Tensor, Tensor> range = (x) =>
{
x = x / 255.0f;
x[x.isnan()] = 0.0f;
return x;
};
Func<Tensor, Tensor> flip0 = (x) =>
{ return x.permute(2,0,1); };
Func<Tensor, Tensor> flip1 = (x) =>
{ return x.permute(1,2,0); };
Func<Tensor, Tensor> normalize = (x) =>
{return (x - IMAGENET_DEFAULT_MEAN) / IMAGENET_DEFAULT_STD;};
var trans = torchvision.transforms.Compose
(
transforms.Lambda(resize),
transforms.Lambda(flip0),
transforms.CenterCrop(224),
transforms.Lambda(range),
transforms.Lambda(flip1),
transforms.Lambda(normalize)
);
tensor = trans.call(tensor);
return tensor.permute(2,0,1);
}
Hey @Sprinzl, can you tell me the torchsharp version and the operating system you are using? Can you check if you have the correct cuda-enabled nuggets isntalled?
Hi, @alinpahontu2912 i am using TorchSharp-cuda-windows with LibTorch 2.5.1. Cuda usally works fine for me if I use it for my own modules. Maybe it has something to do with the Libtorch from Python because I run it from local folders.... I am using Windows. I found a difference. for python: '2.6.0+cu118' for c#: Torch 2.4.0.0
Hey @Sprinzl, thank you for the fast answer. I don't understand what you mean by running in local folders. Torchsharp is currently being built with libtorch 2.4.0, so that is ok. Can you show me the installed nuggets in your project ? I suspect it might be some misconfiguration there.
You only need to install TorchSharp-cuda-windows package and that will automatically get you all required dependencies ( meaning torchsharp and libtorch-cuda). Did you also install the separate libtorch nugget yourself ?
It might be something with the python pip version of torch and cuda. I do not know. I think I will kill Python and reinstall.
I think I see some issues with your code. Would you mind trying: var result = (Tensor) model.cuda().forward(image.unsqueeze(0).to(torch.float32).cuda()); I believe you need to specify data types as torch._datatype for them to be properly defined and also the way to change the device processing the model/doing the training is by using .cpu() or .cuda(). Would you mind giving this a try?
I tried. It did not work. What I found out, or maybe it is a guess, is what you make a forward pass before you save the model with jit. It has to be also set to eval. But GPU - no idea. I aready know what I have to set it to the right tensor dtype. Next guess is to reinstall torch to the right alining version in python, because it uses different cuda-backends. Also I do not know if backprop has some influence on this issue.
Hey @Sprinzl, have you tried something else, do you have some update for this issue?
Hey @Sprinzl, have you tried something else, do you have some update for this issue?
Not really. I have to dig deeper on this issue. I have some other tasks right now. The AI project is not confirmed by the client ... jet. I try to find out the walls of "resistence".
I think PHI-4-multimodal-instruct could be best guess right now, as the code base did not change and where is already a pure torchsharp implementation in existence and where was not really a change in the architecture between phi-2,3,4 from my understanding. With instruct ability this could be really interesting. But I will find out. But on the long run it wil be harder to get the LLMs into torchsharp because of architecture tweeks.
Maybe it has something to do with gpu vram limit....I have to try smaller models first.
Greetings Michael