Learning ONNX for trading - page 13

[MetaQuotes]

Machine Learning Community Standup - Deep Learning with PyTorch ONNX

Machine Learning Community Standup - Deep Learning with PyTorch & ONNX

The "Machine Learning Community Standup - Deep Learning with PyTorch & ONNX" video covers various topics related to machine learning, PyTorch, and ONNX. One section covers overfitting and how to prevent it in neural networks by using dropout and cross-validation. The hosts also highlight various community-based machine learning projects and their upcoming events on using .NET with machine learning. The video also introduces PyTorch, a popular machine learning library used for computer vision and natural language processing, with various built-in modules such as torch vision and transforms. The speakers explain the ONNX format for representing machine learning models and its runtime for running inference and training in multiple languages. The tutorial also discusses how to use pre-built models in PyTorch's model zoo and covers debugging and managing Python packages and environments using Jupyter Notebooks and Anaconda. Additionally, the tutorial covers the details of training and exporting a PyTorch model using ONNX, which can be used with ONNX runtime to improve models' performance.

The video also discusses various topics related to machine learning and deep learning. The speakers talk about using SkiaSharp for image processing in Xamarin and the limitations of on-device models due to their size, but note the benefits of having on-device models. They also suggest various resources for learning the theory of machine learning such as the Andrew Ng Coursera class, and an applied machine learning class which gives high-level information on using tools and libraries to create machine learning models. The importance of having a goal when learning about machine learning and incorporating learning into one's job is also mentioned. Lastly, the speaker hints at upcoming content that may be of interest to the audience.

• 00:00:00 In this section, the hosts introduce their guest, Cassie Breviu, a senior program manager at Microsoft on the ONNX runtime team, who will be discussing PyTorch and ONNX. They briefly recap the previous stream's topic, which was about using .NET with libraries like PyTorch. Before delving into the main topic, the hosts introduce a new segment where they talk about machine learning concepts, starting with dropout. Dropout is a method to prevent overfitting in neural networks, which is achieved by randomly dropping out some of the nodes during training.
  
  
• 00:05:00 In this section of the video, the hosts discuss overfitting and how the use of dropout and cross-validation can help solve the issue in machine learning models. They mention that getting 100% accuracy on a training set is not always a good thing and can lead to poor performance on real data. They make an analogy with studying for a test and memorizing answers without understanding the concepts behind them. The hosts then move on to community links and mention several projects and presentations from within the community, including building serverless machine learning solutions, sentiment analysis applications, using AutoML and Model Builder, and training an image classification model using Azure. These projects aim to help users build and deploy AI solutions more efficiently.
  
  
• 00:10:00 In this section, the speaker talks about a tutorial on how to use Azure to train image classification models, as well as a demonstration of using the ONNX runtime and object detection with the YOLO v5 model. They also mention several upcoming events on machine learning and .NET, including the Let's Learn .NET Focus on Machine Learning event, where participants will go through a learn module coming out later this month. They then discuss the new feature in Visual Studio that allows the use of notebooks, similar to Jupiter notebooks, and how this enables users to reverse engineer from a low-code UI to coding, machine learning, and deeper learning. Finally, they express their hope that notebooks become the standard format for user guides and examples, enabling users to download notebook files to run code cells and learn how to use an API instead of reading through wiki pages or other markdown files.
  
  
• 00:15:00 In this section, the speakers discuss various updates and improvements being made to Microsoft Learn and ML.NET. They mention the interactive notebook support that can be accessed through the web UI, making it easier for those who don't know how to configure Python local environments. They also highlight the Torch Sharp library, which allows users to create custom neural networks and integrate them closely with the ONNX runtime, improving the experience of consuming deep learning models. Additionally, there is a focus on ethical and responsible AI within ML.NET, with plans for name recognition, dynamic data input, and output and time-series forecasting, among other things. Overall, the speakers emphasize the importance of feedback from users and encourage them to provide suggestions for improvements.
  
  
• 00:20:00 In this section of the video, the speaker discusses the ways in which the development flow can change when building machine learning models, particularly in terms of the different tools and libraries that are used. They compare traditional programming to machine learning, explaining that with the latter, data is fed into an algorithm to produce a model, which is then used for inference. The speaker then goes on to introduce PyTorch, an open-source machine learning library, and its various components, including the neural network function and data utilities. They also touch on other libraries like TensorFlow and scikit-learn, as well as tools like Jupyter notebooks and VS Code.
  
  
• 00:25:00 In this section, the speaker provides an overview of PyTorch, which is a popular machine learning library commonly used in computer vision and natural language processing. The library includes several modules such as torch vision for computer vision, torch audio for audio data, and torch text for natural language processing. As data is the backbone for machine learning, the modules also provide pre-processing functionalities such as data sets, data loaders, and transforms to prepare data for training by a model. The speaker recommends using an interactive tutorial on PyTorch to learn more about the library and its functionalities. They also introduce ONNX, an open format built to represent machine learning models, and ONNX runtime, a cross-platform deployment tool that enables inference and training in multiple languages.
  
  
• 00:30:00 In this section, the speakers discuss the use of ONNX models in Blazor, with Cassie admitting that she has not tried it yet. They also talk about how to convert models into the ONNX format using libraries like PyTorch, Scikit-learn, or TensorFlow. Using ONNX models is simple, as you just need to import the ONNX runtime library, create an inference session, and pass your input data to get a result. The difficulty comes with data manipulation in different languages, as Python is considered powerful in this regard compared to other languages. The speakers also mention the model zoo, which has pre-built models that can be used for inference with sample code available to help.
  
  
• 00:35:00 In this section, the speaker discusses how to use pre-built models in PyTorch's model zoo, with a focus on object segmentation using fast rcnn, a type of convolutional neural network (CNN). The speaker also briefly touches on working with tensors and manipulating them using NumPy, noting that PyTorch provides an interactive notebook for learning how to manipulate tensors efficiently. Additionally, the speaker notes that PyTorch's data sets are obtained using `torchvision.datasets`, which downloads them from an open source repository.
  
  
• 00:40:00 In this section of the video, the speaker discusses the open-source nature of PyTorch, allowing access to the code and the different blob storage locations that are being used. They highlight how PyTorch's documentation list all the available datasets that can be accessed, with the data sets being just a wrapper around different open-source data sets. The speaker highlights the legal implications of using open-source datasets and what is okay to pull from the source and what should not be hosted. The video also discusses data sets and data loaders, test data for validation, and creating custom data sets. Finally, they briefly touch on transforms and their importance in manipulating data to turn them into tensors and process through the algorithm.
  
  
• 00:45:00 In this section, the speaker talks about the tooling he uses for managing Python packages and environments. He highlights Anaconda, a tool that he loves and has added to his Windows terminal where he can quickly activate and use different pre-configured packages for different machine learning libraries. He goes on to showcase how he uses Jupyter Notebooks in VS Code, which offers more tooling, Intellisense, and the ability to use a cloud computer to run models that need a larger GPU. Finally, he points out that for those who prefer the .NET environment, the Jupiter Notebook experience is directly installable, thanks to running Anaconda.
  
  
• 00:50:00 In this section, the speaker discusses the Jupyter Notebook experience with PyTorch in Azure Machine Learning Studio, where users can install any necessary packages and use whatever tool they prefer. The speaker briefly covers the Quick Start guide for PyTorch, which is available on Azure Machine Learning Studio and PyTorch's website, and then shifts the discussion to ONNX. Without delving too deeply, the speaker provides a brief overview of the model architecture, loss function, and optimizer needed to train a model. The speaker also gives a quick mention of Copilot and its ability to write comments for users.
  
  
• 00:55:00 In this section, the speaker explains how to train and export a PyTorch deep learning model using ONNX. To train the model, users must set the number of epochs and the validation data, ensuring it is different from the test data to avoid overfitting. The speaker shows a graph of the model's loss at each epoch as it loops through each batch, and once training is complete, the model can be exported in ONNX format using a function that sends a tensor to the device. The exported model can be used to run inference using ONNX runtime with input tensors and optimized graph optimizations for better performance. The speaker also demonstrates a soon-to-be-released Xamarin support for ONNX runtime.
  
  
• 01:00:00 In this section, the speakers discuss using SkiaSharp for image processing in Xamarin and how ONNX runtime is used to process the data. They mention the limitations of using on-device models due to their size, but note the benefits of having on-device models. The speakers also briefly touch upon the architecture of Xamarin and how it creates scaffolding. The discussion ends with a mention of wanting more collaboration between C# developers and the AI world and how anyone with machine learning or deep learning questions can reach out to Cassidy Williams on Twitter or the ONNX runtime Twitter handle for answers.
  
  
• 01:05:00 In this section, the presenters discuss resources for those interested in learning the theory of machine learning. They suggest taking a coursera class, such as the one taught by Andrew Ng, which delves into the behind-the-scenes workings of deep learning models. They give the caveat that there are two ways to approach machine learning: the applied method and the theory method. Other resources include an applied machine learning class on coursera, which gives high level information on using tools and libraries to create machine learning models, and the PyTorch learning path, which is more in-depth and covers some of the math. Additionally, the presenters mention a hackathon for ML.Net that will be occurring in the coming weeks.
  
  
• 01:10:00 In this section, the speaker discusses the importance of having a goal when learning about machine learning and deep learning. There are many free resources available online, but it can be overwhelming to decide where to start. The speaker suggests setting a goal, such as training a computer vision model, and finding resources that align with that goal. It can also be helpful to incorporate learning into one's job, such as using hackathons or project sprints to try out new things. The speaker also mentions that it's important to keep pushing through the hard parts and not quit until the goal is accomplished. Lastly, the speaker hints at upcoming content that may be of interest to the audience.

[MetaQuotes]

Object-Detection Yolov7, ML.NET onnx model

Object-Detection Yolov7, ML.NET onnx model

https://github.com/ptiszai/Object-Detection-yolov7-ML.NET-onnx

[MetaQuotes]

Implement Yolo3 Real-time using C#

Implement Yolo3 Real-time using C#

https://github.com/duonghb53/YoloOnCSharpGPU

[MetaQuotes]

Face Detection Using C# and OpenCVSharp - Practical ML.NET User Group 01/19/2022

Face Detection Using C# and OpenCVSharp - Practical ML.NET User Group 01/19/2022

The video tutorial on face detection using OpenCVSharp in C# started with the speaker introducing the OpenCVSharp library, an open-source library for computer vision tasks, with a .NET wrapper. The video discussed using different classifiers for detection, including for eyes, and the importance of experimentation in classifer selection. The tutorial assisted the listener in building a program for face and eye detection using webcams, with the aid of code snippets, visual studio, and .NET interactive notebooks. Different aspects, including how to overlay transparent images and handle mat objects properly, were also elaborated on. The speaker acknowledged the ease of use, speed, and compatibility of OpenCVSharp with .NET but also noted the lack of examples and uncertain long-term support.

• 00:00:00 In this section, the speaker introduces the topic of face detection using OpenCVSharp in C# and gives a brief overview of what OpenCVSharp is, which is a .NET wrapper for OpenCV, an open-source library for computer vision tasks. The speaker discusses the different features of OpenCVSharp, including image manipulation, face detection, and face recognition models. The speaker also notes that while there are not many C# examples for OpenCVSharp, there are many Python examples that can be easily converted to C# due to the similarities in the APIs. The speaker then goes on to list some examples of how to perform specific tasks in both Python and C# using OpenCVSharp.
  
  
• 00:05:00 In this section, the speaker discusses the different flavors of OpenCVSharp and the importance of pairing managed libraries with native binding libraries. They then demonstrate some examples using Visual Studio Code and .NET interactive notebooks, loading NuGet packages and using the hard cascade XML files to perform face detection. The code walks through bringing in the hard cascade classifiers and using them to detect faces and draw boxes around them in images.
  
  
• 00:10:00 In this section, the presenter explains the importance of properly handling and disposing of "mat objects" in OpenCVSharp to avoid potential memory leaks. They demonstrate how to load in a source image and create a new empty "gray image" for face detection, as the classifier works best on black and white images. They then explain how to use "cb2 equalized histogram" to normalize the brightness and contrast of the image, which leads to improved cascade classification. The presenter also discusses how to use minimum size requirements for face detection to avoid false positives and artifacts. Finally, they show how to use "cv2 rectangle" to draw a box around detected faces and open a window to show the resulting image, with the weight key being used as a way for the user to close the window.
  
  
• 00:15:00 In this section of the video, the speaker discusses how they don't expect a weight key to come in since they don't need it for showing a static image. However, it will come more into play when they look at the webcam demo. They also answer a question about default namespaces and go on to explain how they're going to detect not only the face, but also the eyes in the face. They discuss bringing in two cascade files - the frontal face and eye cascade - and how they loop through the face rectangles to try and do an additional detection of the eyes. They explain the process involved in creating a region of interest (ROI) and drawing a rectangle around the face.
  
  
• 00:20:00 In this section, the instructor goes through the process of detecting eyes in a face image using nested cascades in OpenCVSharp. They take the previously detected face region of interest, convert it to gray, and then apply a nested cascade specifically to detect eyes. By doing this, they are able to draw circles around the detected eyes, with the center point of the circle being the location of the detected eyes. They note that the size of the detected eyes should be smaller than the size of the detected face, and through some trial and error, they adjust the parameters of the nested cascade to better detect eyes in different sized faces. Overall, while the process isn't perfect, it is effective in detecting eyes in most cases.
  
  
• 00:25:00 In this section, the speaker discusses different classifiers that can be used for face detection and the need for experimentation to determine which classifier works best for specific situations. The speaker also acknowledges that there may be artifacts in the detection process and suggests using logic to remove them if productionalizing the process. The speaker then moves on to discuss the use of webcams and how the code for webcam face and eye detection is similar to previous examples, with the only difference being the introduction of video capture and the creation of a window object. Finally, the importance of waiting for one millisecond between frames to see if a key gets pressed is highlighted, as it greatly affects the frames per second captured in the webcam.
  
  
• 00:30:00 In this section, the presenter demonstrates the face detection feature in OpenCVSharp using C#. The code captures webcam data and detects the subject's face by placing a box around it. The presenter explains the weight key, a distinguishing feature of OpenCVSharp that must be included for the window to show an image. They go on to answer a question about when the weight key method is executed and describe how it is used. The section also includes an example of adding frames per second to the webcam data capture code. The presenter notes the importance of including the weight key, as forgetting it can cause issues, and demonstrates how to destroy all windows in the code.
  
  
• 00:35:00 In this section, the presenter discusses how they are able to put text on the screen and control the color using C# and OpenCVSharp. They explain how they define a color for the box around faces and eyes, and define another color called fps color, which is red if the frame rate is less than 20 frames per second, and green if it's greater than or equal to 20. They also demonstrate the effect of the delay in the weight key and how it affects frame rates. They mention that with all the variables that go into it, they are happy if they can get 20 or more frames per second, which allows them to achieve their goals for the moment.
  
  
• 00:40:00 In this section, the presenter demonstrates how to overlay images and explains the importance of using IamReadModes unchanged to retain the transparency element of an image. They use sunglasses as an example and admit that the code is not optimized, as they didn't have the time to make everything perfect. They also cheat by hard-coding the coordinates of where they want the sunglasses to land instead of using the x and y coordinates of the face rectangle. The presenter explains that in real-life scenarios, one would need to pass x and y coordinates to the MakeOverlay method to know where to put the overlay. Additionally, they mention the Resize method that would need to be used to resize the overlay image based on how big the rectangle is, but this is not necessary in their cheating example.
  
  
• 00:45:00 In this section, the speaker explains the process of merging a transparent overlay image onto a base image. First, the transparent overlay image is converted to RGB while preserving the transparent channel. Then, the RGB channels are extracted from the original overlay and merged into a new map. After that, the transparency channel is merged with the new map by duplicating it three times into a new map called "overlay alpha 3 channel" and performing a bitwise operation. Finally, the region of interest where the sunglasses will be placed is defined, and the face rectangle is extracted to serve as the face region of interest for merging the sunglasses.
  
  
• 00:50:00 In this section, the speaker briefly explains the process of adding sunglasses to a detected face using OpenCVSharp. They use bitwise operations to place the sunglasses on top of the detected face, maintain transparency, and place the face back in its original location. The speaker also highlights some pros and cons of OpenCVSharp, such as its ease of use, speed, and compatibility with .NET, but the lack of examples and uncertain long-term support. The speaker provides various resources and encourages users to sponsor the OpenCVSharp project on GitHub. A participant asks if OpenCVSharp can run on a microcontroller such as Raspberry Pi, and the speaker responds that it should be possible with the Linux Arm flavor.

[MetaQuotes]

Predicting on a Custom Vision ONNX Model with ML.NET

Predicting on a Custom Vision ONNX Model with ML.NET

In this YouTube video, the presenter discusses using ML.NET to predict on a custom vision ONNX model. This involves exporting the model from the custom vision service and importing it into the ML.NET project. The implementation includes resizing images, extracting image pixels, creating a data context and an empty data list to load the image data, using the ML.NET framework to make predictions on the model, and outputting the results. The video also demonstrates how to get the output name of a model using a tool called Neuron and how to get bounding box information from the model for a given test image. The presenter also shows how to draw a rectangle around the bounding box and display the predicted labels using the Graphics API. The implementation of the ONNX model using the ML.NET API and image resizing is emphasized as the most significant part of the implementation.

• 00:00:00 We will use the ML.NET framework to predict on a custom vision ONNX model. To do this, we need to first export the model from the custom vision service and import it into our ML.NET project. Once the model is in our project, we can create a pipeline that resizes the images and extracts the image pixels before making the prediction on the model. To do this, we need to download and add several NuGet packages to our project, as well as create a new data context and empty data list to load the image data. Finally, we use the ML.NET framework to make predictions on the model and output the results.
  
  
• 00:05:00 In this section, the speaker discusses using a tool called Neuron to get the output name of a model downloaded from Custom Vision. Neuron is a little desktop app that provides information on a model, including inputs and outputs. Once the output name is obtained, it can be used as the output column name when calling the transform to apply the model. The speaker also demonstrates how to create a prediction engine and use helper methods to get bounding box information from the model for a given test image.
  
  
• 00:10:00 In this section, the speaker continues with the implementation of the object detection model by demonstrating how to get the bounding boxes, draw a rectangle around it, and display the predicted labels for the images using the Graphics API. The max confidence is obtained to get the top bounding box where only one bounding box with the max confidence is considered. The code also uses helper methods to draw the rectangle, and the string of the description is returned to indicate the confidence percentage. Finally, the test image is saved to the file system for future reference. The speaker emphasizes that the biggest part of the implementation is the resizing of images and the application of the ONNX model using the ML.NET API.

[MetaQuotes]

Making neural networks portable with ONNX

Making neural networks portable with ONNX

In this YouTube video, Ron Dagdag explains how to make neural networks portable with ONNX, focusing on the inferencing side of machine learning. ONNX is an open-source framework that allows for the portability of machine learning models across various processing units and devices. The speaker discusses the process of converting models to ONNX, deploying and integrating the model with applications, and using it for cloud and edge deployment. They also demonstrate how to load an ONNX model in Node.js and integrate image classification models into web and mobile applications using ONNX Runtime. ONNX models are an open standard that can be created from various frameworks to be deployed efficiently on the target platform.

• 00:00:00 In this section, Ron Dagdag introduces the concept of making neural networks portable through ONNX, focusing on the inferencing side of machine learning. He explains the typical process of programming versus machine learning and the need for training data, a training framework, a model, and an inferencing runtime to create algorithms. He also mentions that the focus of the presentation will be on the inferencing side, using ONNX to incorporate the model into an application.
  
  
• 00:05:00 In this section, the speaker introduces ONNX as a framework that allows for the portability of machine learning models across various processing units such as CPUs, GPUs, and FPGAs, as well as different devices like the cloud or IoT. They compare it to a PDF in that it is a format that can be consumed anywhere and is open source on GitHub, having garnered over 11,000 stars and 200 contributors. The speaker recommends using ONNX when dealing with models trained in Python that need to be deployed with other applications or require high inferencing latency, such as running on edge devices like Azure Percept.
  
  
• 00:10:00 In this section, the speaker discusses scenarios where it becomes necessary to convert an application programmed on a specific operating system (OS) or hardware to run on a different OS or hardware. The solution is to convert the program to ONNX format. ONNX is used when running models that are created from different frameworks and when training takes too long. The speaker then goes on to detail four ways to get an ONNX model, the first of which is the ONNX model zoo on GitHub. This is followed by custom vision service, converting from an existing model, and trained using Azure machine learning or automated machine learning.
  
  
• 00:15:00 In this section, the speaker discusses the process of converting models to ONNX. The process is not limited to models written in Python and can also be done with ML.NET. The speaker demonstrates how to convert an ML.NET model and then save it to ONNX. There are many models available that have been converted to ONNX, including image classification and object detection models. Custom Vision Service can also be used to upload images and tag them to classify different types of objects. GitHub provides an app called Netron that allows software engineers to visualize what an ONNX model looks like, including its input and output. The speaker explains that one can also generate an ONNX model by converting it from PyTorch or TensorFlow.
  
  
• 00:20:00 In this section, the speaker demonstrates how to generate a regression model and export it to ONNX, an open-source format that allows for portability when using different machine learning frameworks. Using C# and ml.net, the speaker loads salary data into a data frame and trains a regression model, specifying the label column and the input (years of experience) using a pipeline. After evaluating the model, they convert it to ONNX by passing the model, sample data, and stream to context, resulting in a file that can be viewed using Netron. The speaker notes that ONNX can also be used as an intermediary format to convert machine learning models between frameworks, such as PyTorch and TensorFlow.
  
  
• 00:25:00 In this section, the speaker discusses the process of deploying and integrating an ONNX model into an application. Converting models to ONNX allows for compatibility across different frameworks, and users can also use ONNX models for transfer learning and fine-tuning. The speaker emphasizes the importance of versioning and having a model registry, just like a container registry, for easy tracking and deployment. Additionally, the speaker highlights the different skill sets required for data scientists building the "recipe" versus those needed for engineers starting a "bakery" and integrating AI into applications. The talk covers various frameworks and platforms that support the deployment of ONNX models, such as Azure Machine Learning, Ubuntu or Windows Server, and IoT edge devices.
  
  
• 00:30:00 In this section, the speaker discusses the deployment of machine learning models, covering both cloud and edge deployment. According to the speaker, edge deployment is about how close the data processing is to the users, and it is much closer than the cloud. Running AI on the edge is ideal for low latency and saving costs, as it can process data locally without uploading to the cloud. Additionally, running locally is more flexible and can ensure privacy compliance, particularly important when dealing with rules that forbid sharing images outside the network. To facilitate the deployment of machine learning models on the edge, Microsoft has developed an open-source ONNX runtime that supports not only neural networks but also traditional machine learning spec, can perform high-performance inference, and has an extensible architecture for hardware accelerators.
  
  
• 00:35:00 In this section, the speaker discusses how to make neural networks portable with ONNX. ONNX runtime allows users to run models on different processing units, including GPUs, CPUs, and VPUs, and it can be used on various platforms, such as Windows, web browsers, Node.js, and React Native applications. The speaker shows a demo on how to load an ONNX model in a Node.js application and how to pass in input values to receive the desired output. By using ONNX runtime, developers can easily incorporate machine learning models into their applications without worrying about hardware compatibility.
  
  
• 00:40:00 In this section, the video discusses the ONNX Runtime Web, a JavaScript library designed for running ONNX models directly in a browser using optimized WebAssembly and WebGL technologies. This method of inferencing in the browser is faster, safer, and cheaper while also allowing for offline use. However, it may not be efficient for larger models that require specific hardware specifications and could result in longer download times. The video provides an example of using the ONNX Runtime Web to analyze an image of a crying baby, demonstrating how the model is integrated into an HTML file and executed using the runtime.
  
  
• 00:45:00 In this section of the video, the presenter demonstrates how to use the ONNX model zoo and the ONNX runtime web to integrate an image classification model into a web application. He uploads an image to the web app and resizes and converts it to grayscale and a tensor to prepare it for the pre-trained model. Using the ONNX runtime web, the model is loaded and fed the processed image, resulting in a prediction for the emotion expressed in the image. The presenter then demonstrates scoring a pre-trained ONNX model in a React Native app.
  
  
• 00:50:00 In this section, the speaker discusses the use of ONNX runtime mobile as a lightweight inference solution for running optimized ONNX models on Android and iOS platforms. To integrate ONNX runtime mobile into the application, one can use npm install ONNX runtime react native, which minimizes the binary size to a pre-optimized ONNX model called ort format. The tutorial available at onnxruntime.ai explains step-by-step how to deploy ONNX models using various platforms like node, web browser, react mobile, etc. The compatibility chart shows that Chrome supports web assembly and webGL on Windows 10, macOS, Ubuntu, iOS, and Android. ONNX models are an open standard that can be created from various frameworks to deploy using the right tools efficiently on the target platform.

[MetaQuotes]

On .NET Live - AI Everywhere: Azure ML and ONNX Runtime

On .NET Live - AI Everywhere: Azure ML and ONNX Runtime

The video "On .NET Live - AI Everywhere: Azure ML and ONNX Runtime" focuses on using Azure ML and ONNX Runtime for machine learning with C#. The speakers discuss the benefits of using ONNX format for exporting models across programming languages, the ONNX runtime's optimization for hardware acceleration and inference, and its compatibility with specific versions of the framework. They also showcase how to use ONNX Runtime with Azure ML in Python and .NET, create and train a neural network model, and explain inferencing and its final step in machine learning. The video concludes with the introduction of a new provider for the ONNX runtime that allows for the use of OpenVINO for the ARM CPU, providing debugging capabilities.

In this section of the video, the hosts discuss the flexibility and configurability of the ONNX runtime and its ability to run on various hardware and software platforms. The ONNX runtime is seen as a great wrapper for different platforms as customers can use it on a cloud, Android, iOS or the Snapdragon CPU, and it allows for faster inferencing.

• 00:00:00 In this section, the hosts introduce the guest, Maro Benici, an Italian .NET developer with experience in artificial intelligence (AI). The episode will focus on how to use Azure ML and ONNX Runtime with C# to continue studying machine learning. In the related bits segment, they discuss a document on the Azure Machine Learning site about making predictions with AutoML ONNX models in .NET, which is closely related to the day's topic.
  
  
• 00:05:00 In this section, the speaker discusses the option to export machine learning models into ONNX format for sharing models across frameworks using components in Automl inside Azure Machine Learning. To perform inference in .NET, the ONNX runtime package is used, with ml.net performing the prediction pipeline using the ONNX model. Netron, a tool for understanding inputs and outputs of a model is introduced, and the creation of classes for inputs and outputs is demonstrated. Finally, the pipeline including the model is defined, and an instance of input is run through the engine to make predicts, with the resulting output being the predicted fare. The speaker then briefly introduces a related article about using the ONNX runtime with Stable Difussion to generate AI images.
  
  
• 00:10:00 In this section, the speaker talks about how technology has evolved in the last 15 years, from custom hardware to data centers and cloud computing. He gives an example of his first project in 2006, which involved handmade sensors to catch minerals in a vineyard, and how the data was stored on portable hard disks and shipped to a laboratory to be analyzed. Today, the same project could be done with a mobile phone, with data collected on the cloud and analyzed in real-time, enabling instantaneous alerts and adjustments to be made. The speaker also discusses how the move to the cloud has opened up opportunities for businesses, with cloud-based services now available to customers who were previously restricted to on-premise computing.
  
  
• 00:15:00 In this section, the speaker discusses the challenges faced by customers who use different programming languages to build and deploy their machine learning models. He explains that the ONNX format was created as a solution to this problem, allowing customers to save their models in a language that can be reloaded in a different programming language. He goes on to explain that ONNX can export the mathematical functions and values of the layers in the neural network, allowing for easy conversion between different languages. The speaker emphasizes that ONNX is being used by many companies, making it possible to outsource code and use different companies that know different programming languages.
  
  
• 00:20:00 In this section, the video discusses the ONNX framework and its use for exporting and running machine learning models on different platforms and programming languages. The ONNX framework allows users to export models and run them without having to reload libraries or frameworks, making it easier for customers who may not know about Python and TensorFlow. The ONNX runtime is open source and available on GitHub, and users can select their preferred language and architecture to run the model. The video also touches on the op set compatibility and how it is being influenced, as it is essential to ensure compatibility with specific versions of the framework.
  
  
• 00:25:00 In this section, the speakers discuss how to ensure that mathematical functions used in AI models are compatible with the ONNX runtime library. They recommend using the most recent version of the library, which supports all the operations in the model. They also mention that there is a recent update to the ONNX documentation that provides better visibility into which apps are supported and which are not. If a mathematical function is not supported, an alert is generated, and the user is informed which offset is not supported. The speakers also discuss the benefits of the ONNX runtime library, such as hardware acceleration for GPUs and optimization for AI models.
  
  
• 00:30:00 In this section, the speaker discusses optimization for ONNX runtime and the ability to use it for both training and inference. With ONNX , one can optimize the training process and accommodate custom hardware without starting from scratch. The speaker also confirms that ONNX runtime can be used for optimizing training hardware, while another hardware can be chosen for inference. Overall, the speaker stresses the ability to create and train models for any hardware environment and the flexibility of ONNX runtime.
  
  
• 00:35:00 In this section, the speaker talks about using Azure ML and ONNX Runtime to create a neural network for classifying flowers. They start by selecting the application running inside a standard S11 machine for measuring. Then, they use Tensorflow as their company generally uses it, and choose pandas, NumPy, and other preprocessing tools. They discuss encoding strings as matrices and splitting the data set between training and testing. They finally create the neural network, which they encode in ONNX format so that it can be used in their customer’s data center.
  
  
• 00:40:00 In this section of the video, the speaker explains the process of training a neural network model using Azure Machine Learning and TensorFlow. He demonstrates how to compile the model and optimize for accuracy before evaluating the model's performance. Once the model is trained, he shows how to import the model into TensorFlow to create a new model for the customer. To make it easy for the customer, he defines the input specification and names the input and output. He also explains the benefits of converting the model to ONNX format for testing before delivering it to the customer.
  
  
• 00:45:00 In this section of the video, the speaker showcases how to use ONNX Runtime with Azure ML on Python and .NET. They import ONNX once in Python, select a provider, and load the ONNX runtime to use while ignoring the TensorFlow model. They create input and feed it with array values before running the model to get the output. Additionally, the speaker shows how to create an endpoint to expose ONNX directly on Azure and how to use ONNX on an IoT device or smartphone using a console application on .NET Core.
  
  
• 00:50:00 In this section, the hosts explain the concept of inferencing, which is the final step in machine learning where the trained neural network is used to predict outcomes based on new data it has never seen before. They also discuss the ONNX format, which can convert any custom tensor models into a format supported by the ONNX Runtime. They mention that the pre and post-processing of the model can also be serialized into the ONNX model itself, making it easier to run on each platform. The hosts demonstrate that the process is similar across different languages once the model is loaded.
  
  
• 00:55:00 In this section, the speaker discusses the new provider that has arrived for the ONNX runtime, making it possible to use OpenVINO for the ARM CPU, enabling debugging. The speaker emphasizes the power of this, stating that when the model is supported and possible to run, there is no need to change the code, and it is possible to run the same model without constantly converting it. The speaker notes that while choosing the hardware for the model is essential, it is not something that the ONNX runtime controls, and the decisions made in operationalizing the model need to consider the hardware requirements. Furthermore, the speaker mentions that it is possible to use ONNX runtime on a cluster, and in inference, the model does not need to be converted every time.

[MetaQuotes]

Berlin Buzzwords 2019: Lester Solbakken – Scaling ONNX and TensorFlow model evaluation in search

Berlin Buzzwords 2019: Lester Solbakken – Scaling ONNX and TensorFlow model evaluation in search

Lester Solbakken discusses the challenges of scaling machine learning for search applications and proposes an alternative solution to using external model servers. He suggests evaluating machine learning models on content nodes, rather than sending data to external model servers, to improve scalability and control latency and throughput. Solbakken highlights Vespa's use of its own ranking language and tensor API extension to make it easy to create a declarative package of state for an application, and the ongoing effort to support machine learning models in Vespa. He emphasizes the importance of understanding the correlation between different phases of ranking to avoid system-level retrieval issues and encourages people to contribute to the open-source project.

• 00:00:00 In this section of the video, Lester Solbakken discusses the challenges of scaling machine learning production specifically for search-based applications. He explains how model servers have become increasingly popular to solve performance and scalability issues, but for search applications, evaluating the model for each of the search results quickly leads to a bottleneck in network capacity. The solution is to either send less data for results or find a way to evaluate the model locally for each server. Solbakken touches on his team's work with the Vespa search platform to address these issues.
  
  
• 00:05:00 In this section, Lester Solbakken discusses the challenge of scaling ONNX and TensorFlow model evaluation for search. One potential solution is to send features to model servers, but the problem is that content and model servers may be evaluating ranking based on different sets of features, which can lessen the probability of getting globally best results. Instead, Solbakken suggests evaluating models on content servers to achieve better results, something that has been worked on at Vespa. Vespa, which is heavily used across hundreds of thousands of queries per second worldwide with hundreds of U.S. publications running, has rich core features, including elasticity, scalability, and capacity for advanced relevant scoring.
  
  
• 00:10:00 In this section, Lester Solbakken discusses how Vespa integrates ONNX and TensorFlow to make it easy to create a declarative package of state for an application, making it easy to use machine learning models. Vespa uses its own ranking language to execute these models, and the tensor API extension provides a small set of core features that can represent different types of computation. The benefit of this is that different models from different sources can be translated into this format, making it easier to optimize them. Solbakken also highlights a benchmark they set up to test the hypothesis that sending data around is not efficient, emulating a blog recommendation system where they used a dot product in the first phase and a neural network in the second phase.
  
  
• 00:15:00 In this section, the speaker discusses the scalability limitations of external model servers for machine learning in search and proposes an alternative solution. He explains that adding additional content nodes to distribute the workload can improve upon latency and suggests that by having these nodes rewrite a larger number of documents, you can increase the number of results with machine learning models, resulting in potentially higher quality results. Through multi-phase ranking, where the evaluation is done on the content node and not just pushed through a network, it is possible to scale and control latency and throughput. The speaker concludes that external model servers are not a good solution for scaling and emphasizes the importance of evaluating on the content node instead.
  
  
• 00:20:00 In this section, Lester Solbakken talks about multi-phase ranking in Vespa and the importance of understanding the correlation between the different phases to avoid system-level retrieval issues. He also mentions that model support in machine learning and Vespa is an ongoing effort and encourages people to contribute to the open-source project. For those interested in playing around with Vespa, they can find many resources on the Vespa AI homepage, including sample applications, use cases, and tutorials.

[MetaQuotes]

Assimilate ONNX

Assimilate ONNX

In this video, the presenter introduces ONNX as an open standard for machine learning interoperability that can work on all different platforms. They go through the process of creating an ONNX project from scratch, tweaking an example from the Microsoft repo, troubleshooting issues, and exploring other ONNX-related Github projects. They then test an ONNX binding using GPT2 and CUDA, expressing interest in exploring ONNX runtime Rust bindings further in the future. The presenter notes the versatility and portability of ONNX and sees it as a good tool for experimentation and building more substantial projects in the future.

• 00:00:00 In this section, the speaker introduces ONNX , an open standard for machine learning interoperability that allows the use of many different frameworks and accelerators for inference on cloud services. They mention the availability of a safe wrapper around ONNX runtime with a cross-platform high performance ML inference and training accelerator. They also highlight the portability associated with ONNX , as it can work on all different platforms, and demonstrate how to create a new ONNX project from scratch using examples from the Microsoft Branch. The speaker notes that with ONNX being actively worked on and continuously integrated, it provides a good tool to play around with in machine learning.
  
  
• 00:05:00 In this section of the video, the presenter navigates to the ONNX website to get the latest version and create an ONNX runtime. They also show how to copy and tweak an example from the Microsoft repo and add a makefile. In addition, they integrate the tracing subscriber and gbt in the code and add the squeeze net model. The presenter encounters some errors but resolves them by checking a previous example and ensuring that the correct models are available.
  
  
• 00:10:00 In this section, the speaker is troubleshooting an issue where they are trying to use a specific version of ONNX, but are running into errors. They try various methods to fix the issue, such as copying a working example and rebuilding the project, but none of them seem to work. Eventually, they find an older example in the ONNX documentation that works with their desired version. They swap out the model and are able to compile and run the program successfully. Overall, this section highlights the importance of thoroughly checking dependencies and being willing to try different solutions when debugging.
  
  
• 00:15:00 In this section of the video, the speaker discusses the limitations of the basic, "Hello world" ONNX example and suggests that it would be more useful to have an example that loads and manipulates an image. They highlight some other ONNX -related Github projects, such as Rust Code Search and Track ONNX , which have more advanced functionalities, and show interest in trying them out. Additionally, they explore the GPT-RS project, which enables the use of GPT2 models with CUDA, and explain how it could potentially be used to manipulate image data.
  
  
• 00:20:00 In this section, the user tests out an ONNX binding using GPT2 and CUDA, which generates test outputs for code completion or text completion. The user finds this example to be promising and expresses interest in exploring ONNX runtime Rust bindings further in the future. They summarize their experience as an interesting experiment and express a desire to build something more substantial in their next attempt with ONNX.

[MetaQuotes]

HITNET Vs. ACVNet Neural Stereo Depth Estimation Comparison (ONNX)

HITNET Vs. ACVNet Neural Stereo Depth Estimation Comparison (ONNX)

Comparison of the HITNET and ACVNet stereo depth estimation models in the Driving Stereo dataset.

Model Inference details (NVIDIA 1660 SUPER):
HITNET (640X480): 220 ms
ACVNet (640x384): 480 ms
References: [HITNET Inference]  https://github.com/ibaiGorordo/ONNX-HITNET-Stereo-Depth-estimation
[ACVNet Inference]                     https://github.com/ibaiGorordo/ONNX-ACVNet-Stereo-Depth-Estimation
[Driving Stereo dataset]              https://drivingstereo-dataset.github.io/