dotnet
Classes of torchvision\ops
Is there any plan to add classes of torchvision/ops?
roi_align and boxes are required, they are not implemented in TorchSharp
return torch.ops.torchvision.roi_align(
input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
)
TORCHVISION.OPS
suggestion: Need to approach this ALSO from the perspective of Onnx contributed Ops e.g. com.microsoft.vision
flowchart TD
image-->|com.microsoft.vision\nops| TorchSharp\nmodel-->|com.microsoft.vision\nops| output\nimage
@kaiidams
Why there are no Torchaudio.ops?
Are they in here? https://github.com/pytorch/audio/tree/main/torchaudio/csrc
e.g. com.microsoft.audio
flowchart TD
audio-->|com.microsoft.audio\nops| TorchSharp\nmodel-->|com.microsoft.audio\nops| output\naudio
Most of the Torchtext.ops are here https://github.com/pytorch/text/tree/main/torchtext/csrc
com.microsoft.nlp
flowchart TD
text-->|com.microsoft.nlp\nops| TorchSharp\nmodel-->|com.microsoft.nlp\nops| output\ntext
@GeorgeS2019
Probably ONNX runtime approach is not related to this.
Why there are no Torchaudio.ops?
torchaudio has C code that uses Kaldi and Sox, FFMpeg, which is not implemented in TorchSharp. IMHO, Modern models don't depend on these except I/O, but still useful in some cases. torchaudio doesn't call them ops.
@NiklasGustafsson I don't know about what happened to torchvision.ops, if they are missing features, do you have plan to add them?
There's a small number of ops in torchvision.ops. Feel free to contribute more, if you have time. It would be good to have an issue to track them. I'll open something up.
BTW, it seems like FFMPEG is disabled by default in PyTorch, now:
https://pytorch.org/vision/stable/#torchvision.set_video_backend
You have to build from source to enable it, apparently.
seems there is no prebuild package for torchvision, what is the best way to add it after build from source?
seems there is no prebuild package for torchvision, what is the best way to add it after build from source?
TorchVision is now available on NuGet: https://www.nuget.org/packages/TorchVision, starting with version 0.98.1, when it was split out from TorchSharp.
I try to call the native method from torchvision, torch.ops.torchvision.nms https://github.com/pytorch/vision/blob/main/torchvision/csrc/ops/nms.cpp https://github.com/pytorch/vision/blob/main/torchvision/ops/_register_onnx_ops.py
how to do it in TorchSharp?
I believe someone already implemented nms in C#.
https://github.com/dotnet/TorchSharp/blob/2f89e580090624ab7329391841e0884c506a0813/src/TorchVision/Ops.cs#L48
To answer your question, though -- I haven't understood where the native code binaries and header files for TV are to be found. If you figure that out, we should bundle it in with the other native code we already have.
They are under https://github.com/pytorch/vision/tree/main/torchvision/csrc/ops, contains cpu and cuda implementations. We don't need the IO operations, so FFMPEG and JPEG are not needed. Currently I'll try build from souce and use it on Windows.
Have you found where pre-built binaries are available for download?
Note that the current libtorch binaries are 1.11, with CUDA 11.3 -- that will matter for building compatible TV binaries, I believe.
I try to call the native method from torchvision, torch.ops.torchvision.nms https://github.com/pytorch/vision/blob/main/torchvision/csrc/ops/nms.cpp https://github.com/pytorch/vision/blob/main/torchvision/ops/_register_onnx_ops.py
For Onnxruntime, here is some preliminary discussion how to register Onnx Ops WIP [Documentation]: C# Workflow for consuming "Augmented" Onnx model with Custom Operators
Fortunately I have done the build, there are 3 files, torchvision.dll, torchvision.exp, torchvision.lib, and quite small, only 2 MB
@xhuan8 -- this is really cool! There are a couple of things to think about next:
-
A different output names (more on that later), since 'torchvision.dll' will conflict with the DLL name for the .NET project.
-
How we distribute the native library. Currently, we have a package (libtorch-*) containing the CPU backends for all three platforms, and one for CUDA on Windows, and one for CUDA on Linux. We needed this because the CUDA backends are gigantic.
So, we have to decide where these native libraries go, whether in the same package as the managed TorchVision binaries, or in a separate one. It will depend on the overall size (separate Windows + MacOS + Linux binaries), as well as whether a CUDA backend can be loaded on a machine without a CUDA-capable GPU. This will impact the library names we pick, too.
Also, how to automate the build -- do we do this manually and add the binaries to the build, or do we somehow integrate the pytorch/vision repo into the TorchSharp build process? We should consider pytorch/text and pytorch/audio at the same time.
Thanks for doing this work, it's going to make a huge difference!
TORCHVISION.OPS
suggestion: Need to approach this ALSO from the perspective of Onnx contributed Ops e.g. com.microsoft.vision
@GeorgeS2019 -- I love your enthusiasm for ONNX runtime ops. However, I want to say that TorchSharp is about providing a .NET layer on top of the native library (libraries) underlying PyTorch. Nothing less, nothing more. It's simplistic, perhaps, but the main point is that it simple. Other projects can and should go beyond TorchSharp and provide all kinds of additional features, including interactions with ONNX.
Thus, ONNX runtime ops (contributed or not) lie outside the scope of what TorchSharp will be designed for.
try to load torchvision native dll with LoadLibrary, it fails with error 1114, @NiklasGustafsson @kaiidams do you have any idea? the dll upload to: https://drive.google.com/file/d/1LHd1jwuFlFT87-vT09hzsOWpUUeD-TKF/view?usp=sharing
Instead of loading a native dll, the latest way is to embedded a static library *.lib through *.csproj and compile that into a normal c# dll. More and more of Microsoft products which are previously c++ are now in c#
Godot4 .NET6 NativeAOT is an excellent recent example.
The Godot c# library attaches a c++ Shared Library _internal ( or _internal.lib )
@xhuan8 An interesting MIT License Visual Programming approach with pre/post vision processing node(Ops) around ONNX
FYI: could be interesting and the example involves TorchScript, however not related to TorchSharp, more investigation needed.
@xhuan8 If this is the build from torchvision, it is a C++ torchvision library. You'll need to make a C wrapper so that C# can use it with P/Invoke.
(py310) C:\local>dumpbin /dependents "libtorchvision.dll"
Microsoft (R) COFF/PE Dumper Version 14.33.31630.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file libtorchvision.dll
File Type: DLL
Image has the following dependencies:
c10.dll
c10_cuda.dll
torch_cuda_cu.dll
torch_cuda_cpp.dll
torch_cpu.dll
KERNEL32.dll
MSVCP140.dll
VCRUNTIME140.dll
VCRUNTIME140_1.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-filesystem-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
Summary
D000 .data
1000 .nvFatBi
149000 .nv_fatb
7000 .pdata
4E000 .rdata
2000 .reloc
1000 .rsrc
F0000 .text
If Torch C++ provides torch::fft::hfft2() then we have THSTensor_hfft2 for C# to call. https://github.com/dotnet/TorchSharp/blob/07047718d3f9a7c9f946223ea8646b2988cdc65d/src/Native/LibTorchSharp/THSFFT.cpp#L64
@NiklasGustafsson To build torchvision.dll you'll need Python (and zlib, libpng, CUDA, etc) I think it should be built outside TorchSharp.
@NiklasGustafsson To build torchvision.dll you'll need Python (and zlib, libpng, CUDA, etc) I think it should be built outside TorchSharp.
Yeah, you're right... Ideally, we would re-implement the C++ bodies of TorchVision in C# -- that would solve a lot of problems.
@xhuan8 -- would that be feasible, do you think? To just re-implement the ops in C# using Torch operators, like how nms() was implemented?
@NiklasGustafsson It takes time to implement with C# and verify the correctness, and also not sure how is the performace without cuda.
Currently I'll use the c++ wrapper, the method name from torchvision is little bit strange, like ?nms@ops@vision@@YA?AVTensor@at@@AEBV34@0N@Z
If the ops are implemented in terms of TorchSharp ops, CUDA should come for free.
That name is a C++ mangled name, which means that it's missing the 'extern "C"' declaration.