Learning ONNX for trading - page 7

[MetaQuotes]

Deploy Machine Learning anywhere with ONNX. Python SKLearn Model running in an Azure ml.net Function

Deploy Machine Learning anywhere with ONNX. Python SKLearn Model running in an Azure ml.net Function

The video showcases how the ONNX runtime simplifies and standardizes the deployment of machine learning models built in different languages and frameworks. It demonstrates the process of packaging a Python scikit-learn model into an ONNX model and deploying it in an Azure ML .NET function. The video highlights that the Azure function can be easily triggered through an HTTP POST request, making it easy to call from any application or website, and regardless of the language used to build the machine learning model, it can be converted to an ONNX model and deployed through ML.NET to run consistently.

• 00:00:00 In this section, the video introduces the ONNX runtime and how it simplifies the deployment process of machine learning models built in various languages and frameworks. The ONNX runtime allows models to be encapsulated in a way that can be easily deployed into a different environment. It replaces the process of pickling and provides a standard runtime that works in Python, R, .NET, Java, and other languages. The video goes on to demonstrate an end-to-end example of building a simple model in Python, packaging it up into an ONNX model, and running it in an ML .NET function. The code used in this example is available on the Advancing LLS YouTube channel's public GitHub.
  
  
• 00:05:00 In this section, the speaker demonstrates how to deploy a machine learning model using ONNX in an Azure function. The demo begins by showing a basic Python model that uses scikit-learn to train a linear regression model using San Francisco house price data. The model is trained using a training set that consists of numerical and categorical values, and once the model is trained, it is pickled out to be persisted and deployed in a container. Finally, the speaker tests the model by calling predict on the training set and gets some values back. This process can be used to deploy and run machine learning models anywhere using ONNX.
  
  
• 00:10:00 In this section of the video, the presenter sets up the model to run in ONNX so that Python is not required to use it. The number of features going into the model's training is specified, and the ONNX model is initialized to know what to expect. The initial input is named the feature input, which is needed for scoring. To score the model, a session is created and a line of code is responsible for predicting based off of the training data. The predictions are then printed.
  
  
• 00:15:00 In this section, the speaker discusses running a Python model in ONNX and how it can be run consistently in Python regardless of the packages used. The speaker shows how to convert a Python model to ONNX and run it in an Azure function using an HTTP POST with a JSON object. The function pulls out important data such as the year, the greater living area, and the sale condition, and passes it through to the model, which uses ONNX to score the data and return the result. The speaker explains that the Azure function is a standard HTTP trigger with a REST API off the back of it, making it easy to call from any application or website.
  
  
• 00:20:00 In this section, the speaker explains the steps involved to create a variable called input and use it to shape the input tensor so that it is acceptable for session to run. The session is launched and kickstarted to run the new session on the path and score with the designated input. The score result is then pulled out and packaged as a result object and passed back to the function. The speaker demonstrates how to test the deployed model in Postman by passing through raw JSON object model parameters, such as the year built and living area, to get a score back. The advantage of deploying machine learning models through ONNX and ML.NET is that it does not matter which language people build their models in, as long as it can be converted into an ONNX model, then it can be deployed and run consistently.

[MetaQuotes]

Deploy Machine Learning Models (TensorFlow/Caffe2/ONNX) - Fast and Easy

Deploy Machine Learning Models (TensorFlow/Caffe2/ONNX) - Fast and Easy

The video demonstrates how transfer learning can be used to classify images and how to integrate the image classification model into an end-user application using Python and TensorFlow. The presenter uses a car trading application example to illustrate the challenges faced when photos are not uploaded from the required perspective and labels need to be checked manually, leading to boredom and inefficiency. He explains how to overcome these challenges by training an existing neural network to recognize photo perspectives using the transfer learning technique. He then shows how to test and deploy the model in the Oracle cloud using the GraphPipe open-source project. Finally, the presenter emphasizes the importance of taking machine learning models from the laboratory phase to the production phase.

• 00:00:00 In this section, Jeroen Kloosterman explains how machine learning can be used for image classification and how to integrate the image classification model into an end-user application. He highlights the challenges faced by a car trading application where photos are not uploaded from the required perspective and labels have to be checked manually, leading to boredom and inefficiency. To overcome these challenges, Jeroen uses transfer learning technique by training an existing neural network to recognize photo perspectives using Python and Tensorflow. By retracing only the last layers of the neural network to recognize different perspectives, Jeroen Kloosterman successfully trains the network to classify the photos automatically.
  
  
• 00:05:00 In this section Jeroen shows how to test the model, which has been trained in the previous section, and then deploy it in the Oracle cloud using the GraphPipe open-source project. To start, the retrain Python script is called, which uses TensorFlow to retrain the model. After setting up the environment on the Oracle cloud, the presenter writes an example client using Python to call the image classifier. The classification returned by the model can then be used by the front-end to show a message to the end-user of the car sales app. Finally, the presenter emphasizes the importance of taking machine learning models from the laboratory phase to the production phase.

[MetaQuotes]

Deploy ML Models with Azure Functions and ONNX Runtime

Deploy ML Models with Azure Functions and ONNX Runtime

The video demonstrates how to deploy a machine learning model using ONNX Runtime and Azure Functions in VS Code. The process includes creating an Azure Function project, updating the code with the score script, loading the model from the model path, creating an inference session with ONNX Runtime, and returning the output. The video also shows how to deploy the function to Azure and test it there. This method enables efficient deployment of models through Azure Functions and ONNX runtime, allowing easy access to results.

• 00:00:00 In this section, the video shows how to deploy a machine learning model using ONNX Runtime and Azure Functions in VS Code. The process involves creating an Azure Function project with Python and HTTP trigger, updating the code with the score script, resizing and reshaping the image and pre-processing it, loading the model from the model path, creating an inference session with ONNX Runtime, and returning the output. After verifying the code's functionality, the video shows how to deploy the function to Azure and test it there.
  
  
• 00:05:00 In this section, the speaker demonstrates how to test the deployed function by obtaining the function URL and pasting it on the test. The speaker demonstrates how the deployed function can be used to get a result easily through Azure Functions and ONNX runtime, enabling efficient deployment of models.

[MetaQuotes]

Deploying on Desktop with ONNX

Deploying on Desktop with ONNX

In the video "Deploying on Desktop with ONNX", Alexander Zhang discusses the challenges of deploying on desktop and the solutions offered by ONNX. Supporting desktops has its challenges as there is less control over system restrictions on the GPU or operating system, as well as significant diversity in desktop GPUs. To address these challenges, Alexander relies on different inference libraries for each of the hardware vendors Topaz labs supports. ONNX is used to specify the same model to all of these libraries, providing relatively consistent results on different hardware while saving manual work on each model. However, ONNX conversions may create various issues, such as ambiguity, inconsistency, and quality discrepancies, requiring developers to perform test conversions and use the latest ONNX offsets explicitly. To maximize throughput through batching and potentially run on multiple devices and libraries in parallel, they split images into blocks and select an appropriate size based on VRAM, and then run the blocks through inference.

• 00:00:00 In this section, Alexander Zhang discusses the challenges of deploying on desktop and the solutions offered by ONNX. Fitting into existing workflows while meeting performance expectations and keeping up with advances requires delivering the latest and highest-quality image models available. Supporting desktops has its challenges as there is less control over system restrictions on the GPU or operating system, as well as significant diversity in desktop GPUs. To address these challenges, Alexander relies on different inference libraries for each of the hardware vendors Topaz labs supports. ONNX is used to specify the same model to all of these libraries, providing relatively consistent results on different hardware while saving manual work on each model. However, ONNX conversions may create various issues, such as ambiguity, inconsistency, and quality discrepancies, requiring developers to perform test conversions and use the latest ONNX offsets explicitly.
  
  
• 00:05:00 In this section, the speaker explains that one reason they do many conversions themselves rather than using existing libraries or wrappers is performance. They also emphasize the importance of flexibility to optimize for their own models and performance needs if desired without being obligated to write model-specific code for every model. To maximize throughput through batching and potentially run on multiple devices and libraries in parallel, they split images into blocks and select an appropriate size based on VRAM, and then run the blocks through inference. However, they conclude that there are continuing difficulties in ensuring new model architectures behave well across all libraries, but they remain hopeful that their strategy will overcome these challenges and deliver consistent image quality improvements.

[MetaQuotes]

Deploying ONNX models on Flink - Isaac Mckillen-Godfried

Deploying ONNX models on Flink - Isaac Mckillen-Godfried

Isaac McKillen-Godfried discusses the challenges of incorporating state-of-the-art machine learning models from research environments into production for effective utilization. The goal of the talk is to make it easier to move models from research environments to production and to enable the incorporation of state-of-the-art models into different platforms. He explains the advantages of ONNX format and the different options for integrating deep learning models in Java. Additionally, he discusses deploying ONNX models on Flink using Jep, a Python interpreter written in Java, and explains an open-source project that allows data to be consumed from the Flink Twitter connector and then filter non-English tweets. The talk also highlights the current CPU-only implementation of deploying ONNX models on Flink and the potential for future GPU or hybrid implementations.

• 00:00:00 In this section of the video, the speaker discusses the challenges of incorporating state-of-the-art machine learning models from research environments into production for effective utilization. He mentions that most of the popular frameworks are written in Java and Scala, while the majority of the code and papers are written in Python. The goal of the talk is to make it easier to move models from research environments to production and to enable the incorporation of state-of-the-art models into different platforms. The speaker also talks about the challenges, including the poor Python support in Flink and the difficulty of incorporating ONNX into Java. He also mentions the popularity of PyTorch in the research community and its increasing implementation in machine learning frameworks.
  
  
• 00:05:00 In this section, the speaker discusses the ONNX format and its advantages. ONNX is an open neural network exchange format that allows for easy export and import of models from various frameworks. The goal of ONNX is to enable running models in different languages and frameworks, making it a valuable tool for developers. Additionally, the speaker talks about various ONNX frameworks and tools available for exporting and importing models. They also introduce a scorecard that measures the operations support in ONNX, with TensorFlow and Caffe2 having considerable support. The speaker then discusses the different options for integrating deep learning models in Java, including creating a microservice, Java embedded Python, and running ONNX back-end JVM-based frameworks.
  
  
• 00:10:00 In this section, the limitations of using Java with ONNX models are discussed, including the limited support of neural network operations in frameworks such as Meno and Vespa. The export process can also be difficult and time-consuming and there may be a need to entirely retrain models. One solution is to use async and microservices, but this approach requires scaling and maintaining a separate service. Another approach discussed is using Java embedded Python (JEP), which allows for the use of any Python library and has been found to run fast with frameworks such as Keras. However, there may be issues with shared libraries that need to be addressed.
  
  
• 00:15:00 In this section of the video, the speaker discusses deploying ONNX models on Flink and the potential setup issues that may arise. While it is possible to transfer Java primitives into Python and vice versa, there can be problems with setting up dependencies. The speaker recommends creating a custom Docker image that includes both Flink and Python packages to make setup easier. The speaker also highlights Flair, a PyTorch framework for NLP tasks, and explains how to integrate it into Flink using JEP. The example code uses a rich map function to return the results from Flair as a string.
  
  
• 00:20:00 In this section, the speaker talks about deploying ONNX models on Flink using Jep, a Python interpreter written in Java. The speaker demonstrates an example of performing sentiment analysis on Twitter data with the Flink Twitter connector, and explains the importance of loading the model in the open part of the function to prevent it from reloading every iteration. They also show how to set variables in Python using Jep, and how to return the result to Java as a string. The speaker highlights the use of a shared interpreter in Jep to avoid errors while using Python modules, and suggests converting the result to JSON for easier processing in Java.
  
  
• 00:25:00 In this section, Isaac McKillen-Godfried talks about an open-source project that allows data to be consumed from the Flink Twitter connector and then filter non-English tweets. The data would then be processed through multitask named entity recognition models, which can handle several languages, and specific language models. Named entity recognition and sentiment analysis would occur before being converted to tables and using queries to group them by their entity and sentiment. By incorporating deep learning and other models into Flink, a real-time view of named entities and their sentiment on Twitter text can be seen. Although onnx backends lack maturity, it saves time from having to convert code and rewrites, and running the model on a cluster is fast. McKillen-Godfried plans to do benchmarks to measure the latency increase in the near future.
  
  
• 00:30:00 In this section, Isaac Mckillen-Godfried discusses the current CPU-only implementation of deploying ONNX models on Flink, and the potential for future GPU or hybrid implementations that could further speed up the process. He notes that he has only tested the model on CPUs and has not yet explored the possibilities of increasing efficiency through GPU usage.

[MetaQuotes]

Deploying Tiny YOLOv2 ONNX model on Jetson Nano using DeepStream

Deploying Tiny YOLOv2 ONNX model on Jetson Nano using DeepStream

This video showcases the efficiency of utilizing a pre-trained Tiny YOLOv2 model in the ONNX format to process four video streams simultaneously.
The streams come from four distinct files and are processed on Jetson Nano using the DeepStream SDK. The system achieved an FPS of approximately 6.7 while processing all four videos in parallel.

https://github.com/thatbrguy/Deep-Stream-ONNX

[MetaQuotes]

ONNX Runtime inference engine is capable of executing Machine Learning Models in different environments

ONNX Runtime

The ONNX Runtime is an open source inference engine optimized for performance, scalability, and extensibility, capable of running new operators before they are standardized. The ONNX format allows for easy representation and deployment of models developed on preferred tools in a common way. Microsoft has partnered with Xilinx to build the execution provider for the Vitis AI software library, which allows for AI inferencing and acceleration on Xilinx hardware platforms. The Vitis AI toolkit consists of IP tools, libraries, models, and examples designs for FPGA developers, with benchmark numbers showing peak acceleration for geospatial imaging solutions. The Vitis AI execution provider can be built from source or deployed through a pre-built software library soon to be released in the Azure Marketplace.

• 00:00:00 In this section, Manash Goswami, Principal Program Manager for AI Frameworks at Microsoft, introduces the ONNX Runtime, which is an open source inference engine used to execute ONNX models. The ONNX format allows for data science teams to use their preferred tools for model development while ensuring that the model can be represented and deployed in a common and easily executable way. The ONNX Runtime is optimized for performance, extensibility, and scalability, and it supports custom operators, making it capable of running new operators before they are standardized. The runtime is backwards and forwards compatible, and its execution provider interface allows for ML model execution on different hardware platforms. Microsoft has partnered with Xilinx to build the execution provider for the Vitis AI software library, which executes ONNX models on the Xilinx U250 FPGA platform.
  
  
• 00:05:00 In this section, we learn about the Vitis AI software library, the Xilinx development platform specialized for AI inference on Xilinx hardware platforms. The U250 FPGA is available for use with the Vitis AI software stack in private preview in Azure, as the NP VMs queue for users. The Vitis AI toolkit consists of optimized IP tools, libraries, models, and example designs for developers to use with FPGAs, allowing them to combine AI inferencing and acceleration. Peakspeed, a startup that provides geospatial analytic solutions, integrates ONNX Runtime and the Vitis AI stack with Esri's ArcGIS Pro application to build the world's fastest geospatial imaging solutions. Peakspeed successfully accelerated the geospatial correction or orthorectification process on CPUs, recording benchmark numbers comparing TrueView running on an Azure NP hosting Xilinx U250 FPGA against the same algorithm running on a Xeon platinum CPU.
  
  
• 00:10:00 In this section, Manash explains how developers and customers can infuse their applications with deep learning using ONNX Runtime and the Vitis AI stack to accelerate on FPGA endpoints in Azure, as well as on-prem with Xilinx U250 hardware. He also highlights that developers can build the Vitis AI execution provider with ONNX Runtime from source, and Xilinx will soon release a VM image in Azure Marketplace with all the pre-built software libraries integrated in one place for easy deployment into the Azure NP VM.

[MetaQuotes]

Deploy Transformer Models in the Browser with #ONNXRuntime

Deploy Transformer Models in the Browser with #ONNXRuntime

The video demonstrates how to fine-tune and deploy an optimized BERT model on a browser using ONNXRuntime. The presenter shows how to convert the PyTorch model to ONNX format using the Transformers API, use ONNXRuntime to quantize the model for size reduction, and create an inference session. The video also covers the necessary steps to import packages into JavaScript using WebAssembly and how to run text inputs through the transformed model for emotion classification. Despite a reduction in prediction accuracy, the smaller model size is ideal for deployment on a browser. Links to the model, data sets, source code, and a blog post are provided.

• 00:00:00 In this section, the video presenter demonstrates how to operationalize transformer models and shows the end project that includes a transformer model that has been optimized and deployed to a browser. The model used in the project is an optimized bert model that has been distilled by Microsoft to reduce its size and made test agnostic. The emotions data set used to fine-tune the model is available on the hugging face hub. The presenter walks through the fine-tuning process using the transformers API and shows how to convert the PyTorch model into the ONNX format using transformers' built-in conversion tool. Finally, the ONNX Runtime Web package is used for inferencing in JavaScript, where different operation sets can be chosen depending on the required operators.
  
  
• 00:05:00 In this section, the video discusses how to deploy transformer models in the browser with ONNXRuntime. First, the video explains how to use ONNXRuntime to quantize the model to reduce its size, after which an inference session is created for both the unquantized and quantized models. The video then demonstrates how to import the necessary packages into JavaScript using WebAssembly and how to encode text inputs before running them through the ONNXRuntime model. The demo shows how the transformed model can be used to predict different emotions given an input text. Despite a decline in prediction accuracy, the video concludes that the reduced model size makes it ideal for deployment in the web.
  
  
• 00:10:00 In this section, the presenter explains how they were able to take a large transformer model, distill and quantize it, and use ONNX Runtime to perform inference on the edge. They also provide links to the model and data sets used as well as the source code and a blog post on the demo.

[MetaQuotes]

Open Neural Network Exchange (ONNX) in the enterprise: how Microsoft scales Machine Learning

Open Neural Network Exchange (ONNX) in the enterprise: how Microsoft scales ML - BRK3012

The Open Neural Network Exchange (ONNX) is introduced as a solution to challenges in deploying machine learning models to production, including managing multiple training frameworks and deployment targets, with Microsoft already widely adopting ONNX for products such as Bing, Bing ads, and Office 365. ONNX allows for scalability and maintenance of machine learning models, as well as significant performance improvements and cost savings attributed to the use of hardware accelerators such as GPUs. Additionally, the ONNX ecosystem includes partners such as Intel for runtime optimization, with readily available dev kits and quantization techniques available to convert FP32 models to lower precision data types, resulting in increased efficiency. Speakers also highlight the benefits of utilizing ONNX for edge computing, as the runtime is flexible and can deploy models to different hardware platforms.

• 00:00:00 In this section, the presenters discuss the scale of Microsoft's machine learning initiatives, including over 180 million monthly active users in Office 365 and machine learning technology deployed on hundreds of millions of Windows devices. They also mention that Microsoft uses over six machine learning frameworks and that there are challenges in deploying machine learning models to production. They introduce ONNX and ONNX Runtime as solutions to these challenges, which can be used with hardware accelerators such as Intel and NVIDIA and on Azure machine learning.
  
  
• 00:05:00 In this section of the video, the speaker discusses the challenges that arise when training machine learning models and deploying them to production. With so many different training frameworks and deployment targets, it becomes difficult to manage and maintain efficient application performance. To address this issue, Microsoft introduces ONNX (Open Neural Network Exchange), an industry standard that allows for the conversion of machine learning models to ONNX format, regardless of the framework used for training. This enables the deployment of ONNX models to any supported framework, making for a more flexible and scalable solution. Additionally, Microsoft is building a strong ecosystem of partners to support ONNX and ensure its success as an industry standard.
  
  
• 00:10:00 In this section, the speaker discusses real production use cases of Open Neural Network Exchange (ONNX) and ONNX runtime to show how they bring business value to Microsoft's products and customers. Some of Microsoft's products, such as Bing, Bing ads, and Office 365, have already widely adopted ONNX and ONNX runtime, and have seen significant improvements in model performance and reduced latency. For instance, with ONNX and ONNX runtime, Office 365's grammar check feature has seen a 14.6x improvement in performance, resulting in reduced cost and latency. Another use case, OCR, has also significantly benefited from ONNX and ONNX runtime.
  
  
• 00:15:00 In this section, the speaker discusses how Microsoft is using Open Neural Network Exchange (ONNX) to scale machine learning in various scenarios, including improving the quality and performance of their Azure Cognitive Services OCR community service, as well as improving their search quality and enabling new scenarios like question and answering with personal assistants. The speaker also mentions how ONNX and ONNX Runtime have improved the speed of machine learning models by 3.5 times and 2.8 times, respectively, bringing great value to their product teams. They also highlight the importance of training machine learning models to really understand the semantic meaning of images for improved multimedia search.
  
  
• 00:20:00 In this section, the speakers discuss the use of Open Neural Network Exchange (ONNX) models in Microsoft products, specifically the Bing visual search feature. ONNX allows for the scaling and maintenance of machine learning models, as well as significant performance improvements and cost savings, achieved through the use of hardware accelerators such as GPUs. The speakers also highlight the versatility of ONNX, which can run on a variety of platforms, including x64 and ARM-based architectures, and is an open-source inference engine for ONNX models, available across Windows, Mac, and Linux. The use of ONNX allows for the optimization of execution on specific target hardware without changing the model's interface, making it a valuable tool for scaling and maintaining production deployments.
  
  
• 00:25:00 In this section, the speaker discusses the execution provider interface used to run ONNX models on different hardware platforms, including CPUs, GPUs, and FPGAs. The ecosystem of partners includes Intel, with whom Microsoft has collaborated on runtime optimization using open VINO-based execution providers. They also offer readily available dev kits, including the neural compute stick, to accelerate AI workloads for various verticals such as manufacturing, retail, and transportation. Microsoft and Intel have also collaborated on quantization to convert FP32 models to a lower precision data type, resulting in lower memory bandwidth, reduced memory footprint of the model, and increased number of four-tops per watt with minimum loss of accuracy.
  
  
• 00:30:00 In this section, the speaker discusses how they were able to use vector processing instructions and graph fusion to improve the performance and memory requirements of integer gem kernels for convolutional neural networks. They were able to achieve 4x more compute and 4x less memory requirement. They showcased the benefits of using n-graph as an execution provider along with their hardware processing capability by showing the scaling in performance for various batch sizes. The accuracy loss was very small and met the motivational benefits of quantization. They also discussed the different ways to generate ONNX models such as using ONNX Model Zoo, Azure Machine Learning Experimentation, and custom vision services from Microsoft.
  
  
• 00:35:00 In this section, the speaker explains how to run inference sessions using the ONNX runtime within an application. After converting the model, the user loads it into the ONNX runtime and starts parsing the graph. The runtime identifies available optimizations and queries underlying hardware to identify what kind of ops are supported by hardware libraries. The ONNX runtime is designed to be consistent, so code snippets for Python and C-sharp are very similar. The speaker also mentions AutoML, which allows the user to input data to receive an application proposal that automatically converts code in Python and C-sharp. Additionally, the speaker describes a docker image that includes converters for different frameworks and allows the user to quickly get started with the ONNX runtime. The workflow is demonstrated through the use of Azure Notebooks.
  
  
• 00:40:00 In this section, the speaker discusses how to utilize the Open Neural Network Exchange (ONNX) and Microsoft's machine learning tools to scale models in the enterprise. The process involves using an ML workspace to create a scoring file that includes both pre-processing and inference steps. The speaker then demonstrates how to create container images for the target computer and environment, including base images such as an ONNX runtime base image. Finally, the images are deployed to a cloud IKS service with the ability to send test images for inferencing on the CPU and GPU endpoints.
  
  
• 00:45:00 In this section, the speaker demonstrates the flexibility of the Open Neural Network Exchange (ONNX) runtime by showing a demo where the same code is used to point an application towards different hardware platforms, such as CPUs versus GPUs and x86 versus ARM. The speaker also showcases a demo of deploying ONNX runtime and models on edge devices, specifically on the Intel up squared, to detect safety scenarios in a factory worker scenario using pre-recorded video feed and post-processing of bounding boxes. The code used in the demo is identical, but different hardware accelerators are used to optimize the application. The speaker summarizes that ONNX becomes the common format to represent neural network models and that ONNX runtime enables deployment both in the cloud and on edge devices.
  
  
• 00:50:00 In this section, the presenters discuss the benefits of using Open Neural Network Exchange (ONNX) for building and deploying machine learning applications with Azure. They also address audience questions about machine learning on different processors, using ONNX with existing pipelines, and the possibility of going back from ONNX to previous frameworks. Additionally, they mention their plans for expanding the ONNX Model Zoo with targeted scenarios and data.
  
  
• 00:55:00 In this section, the speakers discuss the benefits of ONNX in terms of framework flexibility, as ONNX models can be used with a variety of frameworks for serving. They also mention the integration of ONNX with Azure machine learning, which allows users to upload inferencing telemetry for retraining or experimenting. The session also discusses the possibility of ONNX integration with Excel natively, though this is still in development. They also address the question of creating custom algorithms and converting them to ONNX, with the option to use Python to manipulate the ONNX file format. Finally, the session mentions the need for an approach to signing ONNX models for distribution, which will be taken as feedback for future improvements.

[MetaQuotes]

#OpenVINO Execution Provider For #ONNX Runtime - #OpenCV Weekly #Webinar Ep. 68

#OpenVINO Execution Provider For #ONNX Runtime - #OpenCV Weekly #Webinar Ep. 68

The OpenVINO Execution Provider for ONNX Runtime was the main topic of discussion in this OpenCV Weekly Webinar. The product aims to accelerate performance for ONNX models on Intel hardware while requiring minimal effort on the user's end. The webinar discussed the challenges of deploying deep learning models in the real world, with OpenVINO presented as the solution to these challenges. OpenVINO can optimize AI models for efficient performance on various devices and hardware. The ONNX runtime, an open source project designed to accelerate machine learning inference, was discussed at length. The webinar also presented a demonstration of the performance improvement achieved with the OpenVINO Execution Provider for ONNX Runtime, as well as its features such as multi-threaded inference, full support for various plugins, and model caching. The integration between OpenVINO and PyTorch through the OpenVINO Execution Provider was also discussed. The presenters addressed questions from the audience on topics such as compatibility with ARM devices and potential loss of performance or accuracy when using ONNX interchange formats.

• 00:00:00 In this section, the hosts introduce their guests, Devon Agarwal and Prita Veeramalai, who are technical product manager and AI framework engineer respectively at OpenVINO. They provide a brief introduction to the show and discuss some upcoming giveaways for viewers. The main topic of discussion revolves around the OpenVINO execution provider for ONNX runtime, which aims to accelerate performance for ONNX models on Intel hardware with minimal effort required on the user's end. The hosts also outline the agenda for the session, which includes an overview of deep learning and its challenges, followed by an introduction to OpenVINO and its capabilities.
  
  
• 00:05:00 In this section of the video, the speaker introduces the OpenVINO Execution Provider for ONNX Runtime and explains its purpose. They also give an overview of OpenVINO, ONNX, and ONNX Runtime, followed by a guide on how to get started with the OpenVINO Execution Provider for ONNX Runtime and integrate it into ONNX applications. The speaker also discusses the product's feature set and future plans, including a sneak peek at a beta preview product for developers. The conversation then moves to the importance of deep learning in today's world, the need for AI on edge devices, and computing demands for AI. The video also covers the challenges associated with the development and deployment of deep learning models, including unique inference needs, integration challenges, and the absence of a one-size-fits-all solution.
  
  
• 00:10:00 In this section, the challenges of deploying deep learning models in the real world are discussed. There are technical challenges stemming from the disconnect between deep learning network training and inference happening on embedded platforms. There are also programming language and hardware variations, which require a dedicated API for software and hardware communication. Intel's OpenVINO toolkit is introduced as a solution to these challenges. The toolkit streamlines development workflow and allows developers to write an application once and deploy it across Intel architecture, providing a write once deploy anywhere approach. The toolkit is capable of deploying applications targeting CPU, iGPU, Movidius VPU, and GNA. It can be valuable in various industries, including industrial, health and life sciences, retail, safety, and security, offering faster, more accurate, and efficient results for real-world deployment.
  
  
• 00:15:00 In this section, the speaker explains the concept of OpenVino and how it optimizes AI models for edge devices. They explain how, once built in frameworks like PyTorch or TensorFlow, AI models must be optimized for efficient performance on specific devices such as CPUs, GPUs, or VPUs. OpenVino automates this conversion process for various devices and hardware, ensuring that the models will be optimized for efficient performance on the device it is deployed on. The speaker then moves on to explain what ONNX is, an open format for representing machine learning models, defining a common set of operators as the building blocks of machine learning and deep learning models. Overall, ONNX enables AI developers to use a variety of frameworks, tools, runtimes, and compilers without worrying about downstream inferencing implications.
  
  
• 00:20:00 In this section, the speakers discuss the ONNX format, an open intermediate model that allows for the conversion of models produced by different tools to be read using a standardized format. The ONNX runtime is an open source project designed to accelerate machine learning inference across various operating systems and hardware platforms. It automatically identifies optimization opportunities and provides access to the best hardware acceleration available. The OpenVINO Execution Provider for ONNX enables the power of the OpenVINO toolkit to accelerate the inference of ONNX models on Intel CPUs, GPUs, and VPUs. It allows users to run inference using ONNX runtime APIs while easily integrating the OpenVINO toolkit as the backend.
  
  
• 00:25:00 In this section, the integration between ONNX runtime and OpenVINO execution provider is discussed. This integration allows for the efficient execution of deep learning models on Intel devices. OpenVINO toolkit provides optimized libraries for running models on Intel devices. When the OpenVINO execution provider is enabled, it intelligently selects which operators in the model should be run on the OpenVINO back-end for maximum efficiency. The remaining operators are executed using the native ONNX runtime framework. The user can install the OpenVINO execution provider through building from source, pulling the Docker image, or using pip install. The ONNX runtime OpenVINO package can be found on PyPI.
  
  
• 00:30:00 In this section of the webinar, the presenters discuss how to use the OpenVINO Execution Provider for ONNX Runtime and demonstrate its capabilities. They explain how to install the product using pip and provide code snippets to show how to import the ONNX runtime library and start an inference session. They also show how easy it is to use the OpenVINO Execution Provider with a simple modification to an existing line of code. The presenters then invite viewers to scan QR codes to access demos and samples to try out for themselves.
  
  
• 00:35:00 In this section, the video presents a demonstration of the performance improvement achieved with the OpenVINO Execution Provider (EP) for ONNX Runtime. The video shows a comparison between the 5 fps achieved with the CPU execution provider and the 8 fps achieved with the OpenVINO EP. Additionally, the video presents a demo of quantization using the OpenVINO EP for ONNX Runtime, which resulted in a 2x performance gain with negligible loss in accuracy. The demo notebook is available on the Intel DevCloud, which provides remote access to real Intel hardware for benchmarking and analysis.
  
  
• 00:40:00 done with the demonstration? In this section, the presenter demonstrates how to launch the OpenVINO Execution Provider for ONNX Runtime and select the hardware, such as i3, i5,xenon, or Core i9, and graphics division. They also showcase the Jupyter notebook and the object detection sample, which takes input video and an ONNX model to run interference on the device. The presenter explains that there is support for CPU, GPU, and FP16 VADM media, and they also mention the CPU execution provider, which is the native ONNX runtime backend. Finally, the presenter discusses the usefulness of the OpenVINO Execution Provider for testing different hardware without buying or renting each device.
  
  
• 00:45:00 In this section, the features of the OpenVINO Execution Provider are discussed. It offers multi-threaded inference, full support for various plugins, and model caching. Model quantization and graph partitioning are also available, as well as APIs for multiple languages. IO buffer optimizations are present for enhanced performance, and external file saving is available for ONNX models. A question is asked for the audience to win a prize, and a sneak peek into OpenVINO integration for PyTorch models is shared.
  
  
• 00:50:00 In this section, they discuss OpenVINO integration with PyTorch through the OpenVINO execution provider. This product can accelerate the performance of PyTorch models on Intel hardware using just two additional lines of code. The user can wrap their nn.module in the torch orp/ort inference module which prepares the module for inference using OpenVINO execution provider and exports the module to an in-memory graph through onnx.export. The ONNXruntime session then partitions the graph into subgraphs with supported and unsupported operators for subgraph partitioning, and OpenVINO-compatible nodes will be executed by the provider and can execute on intel CPUs, GPUs, or VPUs, while all other nodes will fall back onto the default CPU ML execution provider. The installation process is simple, with the option to build from source, do a simple pip install, and get access to the full range of Python APIs.
  
  
• 00:55:00 In this section, the hosts of the OpenCV Weekly Webinar read out questions from the audience on topics such as OpenVINO's compatibility with ARM devices and potential loss of performance or accuracy when using ONNX interchange formats. The hosts and presenters explain that OpenVINO can indeed run and give a boost on ARM CPU devices, and ONNX can even improve performance for most models. However, it is always advisable to test accuracy on a test set when converting from one format to another. The hosts also clarify that ONNX does support dynamic shape models, contrary to a question from the audience. Finally, the hosts and presenters thank the audience and Phil, the organizer, for a great presentation and informative session.