Is OpenVINO EP work? #67

kimi0230 · 2023-12-11T10:10:52Z

kimi0230
Dec 11, 2023

Hi there,
Thank you for your dedication and contributions.
It looks like OnnxStack supports DirectML, CPU, Cuda, and CoreML, but no OpenVINO.

I try to add OpenVINO EP in the C# implementation of Stable Diffusion by Cassie Breviu here: Stable Diffusion with C# and ONNX Runtime.

But it encountered an unknown exception. Similar to this issue: microsoft/onnxruntime#18152
Have you encountered this issue, or do you have any suggestions?
Thank you so much!!

saddam213 · 2023-12-11T17:47:45Z

saddam213
Dec 11, 2023
Maintainer

If you can link me to an OpenVino converted model I can give it a shot

0 replies

kimi0230 · 2023-12-12T03:00:05Z

kimi0230
Dec 12, 2023
Author

Hi saddam213,

I appreciate your message.
You can download OpenVINO nuget package from my google drive: https://drive.google.com/drive/folders/1VkzSIg6x6fVS6A0BcCY34YPMe9iqZ9my?usp=drive_link Build from Step 2

Below is my steps:

Use stablediffusion1.5 onnx model, https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/onnx
Build C# package, got Microsoft.ML.OnnxRuntime.OpenVino and Microsoft.ML.OnnxRuntime.Managed from microsoft/onnxruntime
I had try commit id : ccf3b2054b47c3a48001bd9305957d430ac02f0e and tag v1.16.0 both can build well, but can't run well on https://github.com/cassiebreviu/StableDiffusion

git clone https://github.com/microsoft/onnxruntime.git
cd onnxruntime
.\build.bat --config RelWithDebInfo --use_openvino CPU_FP16 --build_shared_lib --build_nuget --skip_tests

Will generate two file like as Microsoft.ML.OnnxRuntime.Managed.1.17.0-dev-20231212-0151-ccf3b2054b.nupkg and Microsoft.ML.OnnxRuntime.OpenVino.1.17.0-dev-20231212-0151-ccf3b2054b.nupkg

You can download those from my google drive: https://drive.google.com/drive/folders/1VkzSIg6x6fVS6A0BcCY34YPMe9iqZ9my?usp=drive_link

Clone C# Stable Diffusion sample code : https://github.com/cassiebreviu/StableDiffusion

git clone https://github.com/cassiebreviu/StableDiffusion.git
cd StableDiffusion

Add nuget package from Step 2 : Microsoft.ML.OnnxRuntime.OpenVino and Microsoft.ML.OnnxRuntime.Managed

Example

# In StableDiffusion folder
nuget add C:\Users\kimi0\Desktop\ONNX\onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo\Microsoft.ML.OnnxRuntime.OpenVino.1.17.0-dev-20231212-0151-ccf3b2054b.nupkg -Source C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion

nuget add C:\Users\kimi0\Desktop\ONNX\onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo\Microsoft.ML.OnnxRuntime.Managed.1.17.0-dev-20231212-0151-ccf3b2054b.nupkg -Source C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion

dotnet add package Microsoft.ML.OnnxRuntime.Managed -v 1.17.0-dev-20231212-0151-ccf3b2054b -s C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion

dotnet add package Microsoft.ML.OnnxRuntime.OpenVino -v 1.17.0-dev-20231212-0151-ccf3b2054b -s C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion

# In StableDiffusion.ML.OnnxRuntime folder
nuget add C:\Users\kimi0\Desktop\ONNX\onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo\Microsoft.ML.OnnxRuntime.OpenVino.1.17.0-dev-20231212-0151-ccf3b2054b.nupkg -Source C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion.ML.OnnxRuntime

nuget add C:\Users\kimi0\Desktop\ONNX\onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo\Microsoft.ML.OnnxRuntime.Managed.1.17.0-dev-20231212-0151-ccf3b2054b.nupkg -Source C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion.ML.OnnxRuntime

dotnet add package Microsoft.ML.OnnxRuntime.Managed -v 1.17.0-dev-20231212-0151-ccf3b2054b -s C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion.ML.OnnxRuntime

dotnet add package Microsoft.ML.OnnxRuntime.OpenVino -v 1.17.0-dev-20231212-0151-ccf3b2054b -s C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion.ML.OnnxRuntime

Register OpenVINO execution provider in cassiebreviu/StableDiffusion

sessionOptions.AppendExecutionProvider_OpenVINO(@"CPU_FP16");
sessionOptions.AppendExecutionProvider_CPU();

Build and Run cassiebreviu/StableDiffusion
I get error message:

2023-12-12 10:08:12.9542620 [E:onnxruntime:, inference_session.cc:1887 onnxruntime::InferenceSession::Initialize] Encountered unknown exception in Initialize()
Unhandled exception. Microsoft.ML.OnnxRuntime.OnnxRuntimeException: [ErrorCode:RuntimeException] Encountered unknown exception in Initialize()
   at Microsoft.ML.OnnxRuntime.InferenceSession.Init(String modelPath, SessionOptions options, PrePackedWeightsContainer prepackedWeightsContainer)
   at Microsoft.ML.OnnxRuntime.InferenceSession..ctor(String modelPath, SessionOptions options)
   at StableDiffusion.ML.OnnxRuntime.UNet.Inference(String prompt, StableDiffusionConfig config) in C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion.ML.OnnxRuntime\UNet.cs:line 92
   at StableDiffusion.Program.Main(String[] args) in C:\Users\kimi0\Desktop\ONNX\stablediffusiononnx\StableDiffusion\Program.cs:line 90

OS Version: Windows 11
ONNX Runtime Installation: Built from Source (commit id : ccf3b2054b47c3a48001bd9305957d430ac02f0e and tag v1.16.0)
ONNX Runtime API: C#
Architecture: X64
Execution Provider: OpenVINO
OpenVINO version: openvino_2023.1.0

Thank you so much!!

0 replies

saddam213 · 2023-12-12T05:23:40Z

saddam213
Dec 12, 2023
Maintainer

Ok, managed to get some progress

So using the nuget packages you built I get the same error, but I think this is because you have compiled them for CPU_FP16, cassiebreviu/StableDiffusion only supports Float32

Using the model here (Float16) the exception does not throw
https://huggingface.co/nmkd/stable-diffusion-1.5-onnx-fp16

However Run will fail due to cassiebreviu/StableDiffusion not supporting Float16

I suspect if you recompile the nuget for CPU_FP32 it should all work fine
.\build.bat --config RelWithDebInfo --use_openvino CPU_FP32 --build_shared_lib --build_nuget --skip_tests

0 replies

kimi0230 · 2023-12-14T05:43:24Z

kimi0230
Dec 14, 2023
Author

Thank you!!
I will rebuild for CPU_FP32 and try it.

0 replies

AshD · 2023-12-20T15:16:27Z

AshD
Dec 20, 2023

Hi @kimi0230 - Wondering if you made progress on this.

There are lot of new laptops and desktops without a GPU and OpenVino should make Image Gen faster on them.

Regards,
Ash

2 replies

saddam213 Dec 20, 2023
Maintainer

OpenVino seems to work for OnnxStack backend, Hopefully its part of the production OnnxRuntime.Managed package soon so I can include it

kimi0230 Dec 27, 2023
Author

I use CPU_FP32 still has same error,
I found some warnings when build onnxruntime. Maybe I missing some libraries
I will try your new release later.
Thanks!!

AshD · 2023-12-20T15:22:15Z

AshD
Dec 20, 2023

Also, Take a look at this https://github.com/rupeshs/fastsdcpu
He gets images in 1.7 seconds on a CPU.

0 replies

AshD · 2023-12-20T18:30:06Z

AshD
Dec 20, 2023

I wonder if the CPU optimized Olive models will be accelerated with Open Vino

…

On Wed, Dec 20, 2023 at 10:11 AM Adam Clark ***@***.***> wrote: OpenVino seems to work for OnnxStack backend, Hopefully its part of the production OnnxRuntime.Managed package so so I can include it — Reply to this email directly, view it on GitHub <#67 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABY4ODELFOIO4J7PXYZBQ3YKMS37AVCNFSM6AAAAABAPPUSA2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TSMJRGA4DA> . You are receiving this because you commented.Message ID: ***@***.***>

1 reply

saddam213 Dec 20, 2023
Maintainer

I cant run olive F16 models at all on my CPU (i7 -12700KF)

But the 32 models should work, I can whip up a console app to test with this weekend

AshD · 2023-12-20T20:50:18Z

AshD
Dec 20, 2023

I found out later that the models I created work only if you have DirectML and a GPU due to the specific optimizations in the graph. I can create specific Olive optimized CPU based F16 models but the user will need some way load the correct model based on if they have a GPU or not.

…

On Wed, Dec 20, 2023 at 11:16 AM Adam Clark ***@***.***> wrote: I cant run olive F16 models at all on my CPU (i7 -12700KF) But the 32 models should work, I can whip up a console app to test with this weekend — Reply to this email directly, view it on GitHub <#67 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABY4OC5NV5PEMZD445TEBLYKM2SFAVCNFSM6AAAAABAPPUSA2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TSMJRGU2TE> . You are receiving this because you commented.Message ID: ***@***.***>

6 replies

AshD Dec 21, 2023

Looking at https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html
It looks like you can turn off the graph optimizations
onnxruntime.GraphOptimizationLevel.ORT_DISABLE_ALL

I wonder if disabling optimizations will make Olive DirectML models work on CPU only systems.

saddam213 Dec 21, 2023
Maintainer

Graph optimization is set to ORT_DISABLE_ALL for CPU in OnnxStack so I don't think its that

Im pretty sure CPUs just don't support Float16, I think some of the new CPUs support BFloat16

However OpenVino EP might handle the datatypes differently, wont know until I test some models

saddam213 Dec 21, 2023
Maintainer

Found this, seems to support my ramblings

https://stackoverflow.com/questions/49995594/half-precision-floating-point-arithmetic-on-intel-chips

AshD Dec 21, 2023

I just tried that flag in Python. Got this error.

onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for GroupNorm(1) node with name 'GroupNorm_22'

Sigh. It would be nice to have a ONE model with common optimizations work.

I can do FP16 models on my CPU with Hugging Face Transformers. But I don't know if it is internally converting the FP16 model to FP32 at runtime.

saddam213 Dec 21, 2023
Maintainer

Same error I get on my CPU

There are different types of F16, the Onnx specific types are Float16 and BFloat16, I think the pytorch based F16 are just half precision F32's, but they are just handled as float in the backend

I have opened a PR to add OpenVino to OnnxStack.Core, so at least we can test a bit easier now

Unfortunately the OpenVino DeviceId is a string, all others are an int, so it ,may take a bit of re-work to pass that down, but its set to auto for now

saddam213 · 2023-12-29T05:07:07Z

saddam213
Dec 29, 2023
Maintainer

Made a UI build using the OpenVino nuget you posted
OpenVino Build: OpenVino.F16.zip

Using this model stable-diffusion-1.5-onnx-fp16 I can successfully generate images, however its painfully slow 200 seconds each

I do not have the OpenVino toolkit installed so that could be the issue, so posting a link to the binaries in case you want to test further

OpenVino Branch:
https://github.com/saddam213/OnnxStack/tree/OpenVino
Use build configuration Debug-OpenVino or Release-OpenVino

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is OpenVINO EP work? #67

{{title}}

Replies: 9 comments 9 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Is OpenVINO EP work? #67

kimi0230 Dec 11, 2023

Replies: 9 comments · 9 replies

saddam213 Dec 11, 2023 Maintainer

kimi0230 Dec 12, 2023 Author

saddam213 Dec 12, 2023 Maintainer

kimi0230 Dec 14, 2023 Author

AshD Dec 20, 2023

saddam213 Dec 20, 2023 Maintainer

kimi0230 Dec 27, 2023 Author

AshD Dec 20, 2023

AshD Dec 20, 2023

saddam213 Dec 20, 2023 Maintainer

AshD Dec 20, 2023

AshD Dec 21, 2023

saddam213 Dec 21, 2023 Maintainer

saddam213 Dec 21, 2023 Maintainer

AshD Dec 21, 2023

saddam213 Dec 21, 2023 Maintainer

saddam213 Dec 29, 2023 Maintainer

kimi0230
Dec 11, 2023

Replies: 9 comments 9 replies

saddam213
Dec 11, 2023
Maintainer

kimi0230
Dec 12, 2023
Author

saddam213
Dec 12, 2023
Maintainer

kimi0230
Dec 14, 2023
Author

AshD
Dec 20, 2023

saddam213 Dec 20, 2023
Maintainer

kimi0230 Dec 27, 2023
Author

AshD
Dec 20, 2023

AshD
Dec 20, 2023

saddam213 Dec 20, 2023
Maintainer

AshD
Dec 20, 2023

saddam213 Dec 21, 2023
Maintainer

saddam213 Dec 21, 2023
Maintainer

saddam213 Dec 21, 2023
Maintainer

saddam213
Dec 29, 2023
Maintainer