Learning ONNX for trading - page 3

[MetaQuotes]

ONNX Model Zoo Demo | Tutorial-10 | Open Neural Network Exchange | ONNX

ONNX Model Zoo Demo | Tutorial-10 | Open Neural Network Exchange | ONNX

The video tutorial showcases how to use ONNX Model Zoo to perform inferencing on an ONNX model using the ONNX runtime. The presenter guides viewers through the process of creating a virtual environment, installing necessary packages, downloading the MNIST handwritten model from the ONNX Model Zoo, and writing a Python script for inference. The demo shows that the prediction time is fast and encourages users to download models directly from the ONNX Model Zoo. The video teases the next tutorial, which will cover converting a Python model to TensorFlow.

• 00:00:00 In this section, the presenter demonstrates how to download the MNIST handwritten model from the ONNX Model Zoo and perform inferencing on top of the ONNX runtime. The user needs to create a virtual environment and install the required packages such as ONNX Runtime, OpenCV, and NumPy. The presenter then shows how to download the model directly from the ONNX Model Zoo or by copying the link to the CNTK.ai website. Once the model is downloaded, the presenter explains how to write a Python script for inference, including loading the ONNX model, preprocessing the image, and running the session to obtain the output. Finally, the presenter displays the prediction by painting the results based on the argmax operation.
  
  
• 00:05:00 In this section, the speaker discusses inferencing with ONNX models using the ONNX runtime. They demonstrate using a pre-trained ONNX model to predict handwritten digits and show that the prediction time is quite fast. The speaker also mentions that users can download models from the ONNX Model Zoo and start inferencing without the need to convert them. They tease the next video, where they plan to convert a Python model to TensorFlow, giving users a more in-depth understanding of the process of model conversion.

[MetaQuotes]

PyTorch to Tensorflow Demo | Tutorial-11 | Open Neural Network Exchange | ONNX

PyTorch to Tensorflow Demo | Tutorial-11 | Open Neural Network Exchange | ONNX

The video demonstrates how to use ONNX to convert a PyTorch model to TensorFlow format. The process involves training the model in PyTorch, saving it in .pth format, and then converting it to ONNX format before finally converting it to TensorFlow format. The conversion process is shown in detail through the use of a handwritten digit classification model using the MNIST dataset, and the resulting TensorFlow model is tested with sample images. The video also briefly touches upon converting a model from Caffe2 to ONNX and suggests that users explore ONNX further.

• 00:00:00 In this section of the video, the speaker demonstrates how to convert a Python model to TensorFlow using ONNX. They explain that when converting models, there are two steps that need to be followed: first, train the model in the desired framework, and then convert it to the ONNX file format. From there, it can be converted to the desired format, such as TensorFlow or PyTorch. The speaker then shows how to use ONNX and TensorFlow packages to convert a handwritten digit classification model from PyTorch to TensorFlow using the MNIST dataset. They explain each step of the process, including installing necessary packages, importing libraries, defining the model, and creating training and test
  functions. The notebook code is provided in the resources section for users to follow along.
  
  
• 00:05:00 In this section of the video, the presenter trains a PyTorch model and saves it in a .pth file format. Next, the model is loaded and converted into an ONNX file format. The converted ONNX model is then loaded into TensorFlow to test its functionality on three.png and seven.png images. The model predicts the correct values for both images. Finally, the ONNX model is converted to a TensorFlow model and saved in a .pb file format, which can be used for further predictions. Overall, the presenter demonstrates how to convert PyTorch models to TensorFlow models with the help of ONNX.
  
  
• 00:10:00 In this section of the video, the speaker discusses how one can convert their model from a caffe2 model to ONNX. The speaker has given a link to the notebook where the code is already written and all the required packages are available. The speaker explains that all possible conversions, such as PyTorch to ONNX, PyTorch to Caffe2, and TensorFlow to ONNX, are available on the notebook. The speaker advises the viewers to explore ONNX further and try real-time examples for a better learning experience. Finally, the speaker ends the video and thanks the viewers for watching the series.

[MetaQuotes]

Netron is a tool for viewing neural network, deep learning, and machine learning models

[MetaQuotes]

Quick look into Netron

Quick look into Netron

In the video, the presenter provides an overview of Netron, a tool for viewing and analyzing machine learning models. Netron supports various formats and can be installed on multiple platforms. The presenter demonstrates how to start Netron and navigate through several example models, highlighting the tool's capabilities and limitations. While Netron is useful for exploring simpler network architectures, the presenter suggests that it could benefit from additional features for visualizing more complex models. Overall, the presenter recommends Netron as a helpful tool for examining and understanding machine learning models.

[MetaQuotes]

Netron - Network Visualization Tool | Machine Learning | Data Magic

Netron - Network Visualization Tool | Machine Learning | Data Magic

Netron is a Python library that helps users to visually explore and examine the structure and parameters of deep learning models. It is an open-source library that provides sample models for analysis and has a simple installation process. With just two lines of code, users can install Netron and use it to visualize the neural network structure, activation functions, pooling layers, convolutional layers, and all attributes passed at each layer of a given machine learning model. Netron provides an easy-to-use interface that allows users to export visualizations as PNG files and explore different features and options.

• 00:00:00 In this section, we learn about Netron, a python library that helps us visualize the internal structure of deep learning models. We can use Netron to examine the specific neural network layers and parameters used within a given model. Netron is an open-source library that provides sample models for analysis and has a simple installation process. Once installed, users can import the Netron library and use the "start" method to pass in a machine learning model file. Netron will then create a visual representation of the model structure, allowing users to visually explore each layer and its parameters.
  
  
• 00:05:00 In this section of the video, the presenter demonstrates how to use Netron, a network visualization tool for machine learning models. The tool can visualize the neural network structure, activation functions, pooling layers, convolutional layers, and all attributes passed at each layer of a given machine learning model. With just two lines of code, the tool can be installed on a local machine or accessed online at the netron.app website. Users can export the visualizations as PNG files and explore different features and options available in the tool's interface.

[MetaQuotes]

PyTorch, TensorFlow, Keras, ONNX, TensorRT, OpenVINO, AI Model File Conversion

[Educational Video] PyTorch, TensorFlow, Keras, ONNX, TensorRT, OpenVINO, AI Model File Conversion

The speaker in the video discusses the advantages and trade-offs of different AI frameworks, such as PyTorch, TensorFlow, Keras, ONNX, TensorRT, and OpenVINO, and recommends PyTorch as the preferred framework for training and data conversion. The speaker explains the conversion process, including converting PyTorch models to ONNX and then to TensorRT or OpenVINO, and cautions against using TensorFlow PB file and Cafe. The speaker also discusses the importance of setting the floating point format properly and recommends using FP 32 for most models. The video provides examples of model conversion and encourages viewers to visit the official website for more educational videos.

• 00:00:00 In this section, the speaker discusses the different AI frameworks, including PyTorch, TensorFlow, Keras, and Cafe, and explains their company's preferred method of using PyTorch to save the model as an ONNX file, which they double-check with their data set using ONNX Runtime. If the test is passed, they convert the ONNX model to TensorRT and OpenVINO formats. The speaker cautions against using TensorFlow PB file and Cafe and recommends using Python with original data and PyTorch for training instead. Lastly, the speaker mentions that sometimes the inference engines need the floating point to be set properly.
  
  
• 00:05:00 In this section, the speaker discusses how the choice of floating point format affects the speed and precision of model conversion. He explains that while using FP 16 can increase speed, it can lead to less precision, while using FP 64 results in slower speed but higher precision. The speaker recommends using FP 32 for most models and discusses how different formats can be used for specific types of data sets, like FP 32 and FP 64 for medical image analysis. He also explains that converting a model from FP 32 to FP 16 can lead to precision loss, which can be mitigated by using calibration, deletion or retraining with FP 16 to improve the model's precision.
  
  
• 00:10:00 In this section of the video, the speaker discusses the trade-offs between speed, precision, and data information when using different AI frameworks such as PyTorch, TensorFlow, Keras, ONNX, TensorRT, and OpenVINO. The speaker recommends using PyTorch and converting the models from PyTorch to ONNX using a provided solution. Then, they explain how to convert the models from ONNX to TensorRT using another provided solution. The speaker demonstrates the conversion process by running the code in Jupiter Lab and shows how to locate the converted model files.
  
  
• 00:15:00 In this section of the video, the speaker discusses the ease of converting AI models from PyTorch to ONNX and then to TensorRT or OpenVINO, emphasizing that it is a simple process. However, for those using TensorFlow or Keras, the speaker recommends using PyTorch to retrain the data set as it will be easier to do model conversions. The speaker warns that there may be problems using Keras as the model file data contains only parameters, and there is a need to build the network architecture first before importing the H5 parameters file. The speaker suggests that the ultimate solution to such problems is Café, but with the developers of Café migrated to Café 2 and no one to maintain it, the speaker recommends using PyTorch as the main AI framework.
  
  
• 00:20:00 In this section, the speaker discusses the advantages of using PyTorch and the ease of migration due to its fast speed and improved architecture. The speaker also provides an example of using a solution to convert a v3 weight model to an OpenVINO AIA model and the most powerful solution for object detection, Euro v4. To use the example, two files are required for conversion, a Euro v4 CPP weights file and a Euro v4 CFG network configuration file. After the conversion, a Haitatsu PTH file is generated for use in inference image to verify the results. The speaker recommends using Python to do the AI training and then converting to ONNX and then TensorRT or OpenVINO. Finally, the speaker encourages viewers to visit the official website for more educational videos and to become a free member to receive the videos list every week.

[MetaQuotes]

How we use ONNX in Zetane to complete machine learning projects faster with less trial-and-error

How we use ONNX in Zetane to complete machine learning projects faster with less trial-and-error

Patrick Saitama, co-founder and CTO of Zetane Systems, discusses the value of using ONNX in his company's new product to address issues related to the black box problem of AI. Zetane's engine allows for the exploration and inspection of the ONNX models, providing insights into the model's interaction with data and leading to more decisive strategies for improving its quality. The example given shows how Zetane's engine helped debug an autonomous train model by inspecting the radio layer and adding more images of tunnels labeled as non-obstacles. Zetane also includes tools for dynamically inspecting internal tensors and taking snapshots of the model for later investigation. Additionally, Zetane's new engine allows for larger models such as YOLOv3 to be installed.

• 00:00:00 Patrick Saitama, the co-founder and CTO of Zetane Systems discusses how to extract greater value from ONNX to shorten your development cycle time and reduce guesswork. His company, Zetane Systems, based in Montreal, has recently released a new product, which is industry-agnostic, and aims to address some of the issues related to the black box problem of AI. By passing input data to the ONNX models, and then projecting the models in the Zetane engine, the models can be explored and inspected, including its architecture and computation graph, as well as all the tensors contained in each operator node, in order to debug and optimize the model.
  
  
• 00:05:00 The speaker discusses how using ONNX in Zetane allows for deeper insights into the model and its interaction with data, which in turn leads to more decisive strategies for improving the model's quality. The example given is of an autonomous train model and how looking at the radio layer in the sustained engine showed that the model was detecting the tunnel as an obstacle, leading to the addition of more images of tunnels labeled as non-obstacles. Zetane also includes tools for inspecting internal tensors dynamically and taking snapshots of the model at certain moments to later investigate and improve upon. Additionally, the detain engine has recently been launched, allowing for the installation of larger models such as YOLOv3.

[MetaQuotes]

What's New in ONNX Runtime

What's New in ONNX Runtime

This talk will share highlights of the ONNX Runtime 1.10-1.12 releases, including details on notable performance improvements, features, and platforms including mobile and web. Ryan Hill has been with the AI Frameworks team for the past 4 years, where he has mostly worked on operator kernels, C APIs, and dynamically loading execution providers. Prior to this he worked on the Office PowerPoint team, where his most widely seen work is many of the slideshow slide transitions. For fun he likes trying to use the latest C++ features and hitting internal compiler errors.

In the video, software engineer Ryan Hill discusses the various features and updates of ONNX Runtime, a widely used cross-platform runtime that can target multiple CPU architectures. He highlights the latest features added to ONNX Runtime, such as the ability to call op kernels directly and performance improvements like transpose optimizer and small size optimization. Hill also talks about ONNX Runtime's execution providers, which enable optimal performance on various hardware, and the release of mobile packages that support NHWC conversion at runtime. The video also covers layout-sensitive operator support, Xamarin support for cross-platform apps, ONNX Runtime web, and the ONNX Runtime extensions library that focuses on model pre-post-processing work, including text conversions and mathematical operations, and currently focuses on NLP, vision, and text domains.

• 00:00:00 In this section, Ryan Hill, a software engineer working on ONNX Runtime for about four years, discusses the features of ONNX Runtime and its new releases. He highlights that ONNX Runtime is a cross-platform, runtime that can target multiple CPU architectures and has language bindings for several programming languages. It is widely used by multiple industries, including Microsoft, having over 160 models in production. Ryan also discusses the new features added in the latest releases, such as being able to call op kernels directly from outside a model run call and the ability to feed external initializers as byte arrays for model inferencing. Additionally, Ryan talks about performance improvements of the latest version, such as a transpose optimizer and a small size optimization feature. Lastly, he highlights ONNX Runtime's execution providers that enable it to perform optimally on various hardware and its mobile packages' release, which now supports NHWC conversion at runtime.
  
  
• 00:05:00 In this section, the video covers the new features and updates in the ONNX Runtime, including layout-sensitive operator support and Xamarin support for cross-platform apps on Android and iOS. Additionally, the ONNX Runtime web offers a single C++ code base, compiled into web assembly, that is faster and uses less memory, and there is now an ONNX Runtime extensions library that focuses on model pre-post-processing work, allowing users to do this work entirely inside the model run call. The library includes text conversions, mathematical operations, and currently focuses on NLP, vision, and text domains. Microsoft Office team are currently using this extension library.

[MetaQuotes]

v1.12.0 ONNX Runtime - Release Review

v1.12.0 ONNX Runtime - Release Review

The v1.12.0 release of the ONNX Runtime (ORT) focuses on inferencing but also includes continued investments in training, with integration with Hugging Face Optimum resulting in the acceleration of several Hugging Face models. New features include the ability to use native ORT ops in custom ops and call directly into a native or runtime operator without building a graph. The release also includes support for .NET 6 and the Multi-platform App UI (MAUI) and execution providers for specific platforms like the Neural Processing Unit on Android and Core ML on iOS. Performance improvements were made by reducing memory allocations during inferencing and eliminating unnecessary logging. Future improvements to enhance cache locality and thread pool utilization are planned.

• 00:00:00 In this section, the new features and updates of version 1.12 of ONNX Runtime are discussed. These include the deprecation of .net standard 1.1 support and the addition of support for ONNX offset 17 and on on xml offset 3. One new feature is the capability to invoke individual ops without creating a separate graph, and support for feeding external initializers for inferencing was also added. Other updates include support for python 310 and enabling mac m1 support in the python and java libraries, as well as adding .net 6 maui support in the c-sharp package. improvements to performance and quantization were also made, and new execution providers were introduced, including Qualcomm Snappy and general infrastructure for the accident pack ep, with ongoing work to add more kernels for mobile and web scenarios.
  
  
• 00:05:00 In this section, the speaker discusses the updates made to the ONNX Runtime (ORT) and mentions that the focus has been primarily on inferencing. However, there have been continued investments in ORT training to accelerate the training of large models. The recent integration with Hugging Face Optimum has resulted in the acceleration of several Hugging Face models. The speaker then introduces Randy, who discusses a new feature that allows users to use native ONNX Runtime ops in custom ops. Randy explains that this feature came about from customer requests to make custom operators more performant and versatile by utilizing the powerful matrix computation capabilities of the ONNX Runtime.
  
  
• 00:10:00 In this section, the speaker discusses a new feature that allows customers to call directly into a native or runtime operator without building a graph or something similar, making it much easier to execute matrix computation functions. This feature was proposed to the community because another group of customers were working on audio processing and wanted to achieve statement management, meaning they wanted to cache some past inputs or outputs, combine them with the latest input, and feed the operator with a combined altered input. This was previously difficult to achieve, but with the new feature, customers can add a wrapper around the native operator of the ONNX Runtime to do the statement management, making their lives easier and achieving their purpose. Samples of how to use this feature are available on the community website.
  
  
• 00:15:00 In this section, Scott McKay, the lead for ONNX Runtime Mobile, discusses the new features added in v1.12.0 of ONNX Runtime. The release includes support for .NET 6 and the Multi-platform App UI (MAUI), allowing developers to create an app using one shared code base that can run on Android, iOS, macOS, Windows, and Linux. ONNX Runtime also includes execution providers for specific platforms such as the Neural Processing Unit on Android and Core ML on iOS, which can optimize model execution speed and power efficiency. McKay explains that developers can use the same C# bindings for interacting with the ONNX Runtime library across all these frameworks, but there may be some platform-specific code required for handling differences in device screen sizes and processing images. To add ONNX Runtime into a .NET 6 project, developers can use the microsoft.ml.onnxruntime package and the microsoft.ml.onnxruntime.managed package, which provide the C++ implementation for executing the model and the C# bindings to interact with the native library.
  
  
• 00:20:00 In this section, the speaker discusses the availability of examples for users to learn how to use the new libraries. There is a Github repository that has an example Xamarin app to demonstrate the functionality of the new release. Additionally, the team will be updating the app to include a Maui app that will be similar in design. Finally, the audience expresses interest in a deep-dive tutorial since the new Maui support will be super useful, and the examples would be excellent. The following speaker explains the concept behind the extensibility of the ONNX Runtime team and provides updates on execution providers. The update for this section focuses on the integration with hardware via the execution provider interface. In this release, the team focused on inferencing workloads, and collaboration with vendors such as Nvidia, Intel, and Qualcomm yielded many improvements. One improvement is an option to share execution context memory that reduces the overhead of accessing multiple subgraphs with TensorRT. Another optimization relates to engine caching support, where building engines in advance reduces the time to rebuild engines at inference time.
  
  
• 00:25:00 In this section, the release review of v1.12.0 ONNX Runtime is discussed, specifically the execution provider called Snappy, which stands for Snapdragon Neural Processing Engine by Qualcomm. This is Qualcomm's runtime that is used to accelerate AI workloads on mobile Snapdragon SOCs. The support for Snappy is brand new and was announced at the last month's Microsoft Build conference. Alongside the support for Snappy, Intel has also started building Python packages that are being hosted on PyPI with ONNX Runtime with OpenVINO enabled. This makes it easier for developers to set up and enables better support for models with dynamic input shapes. Links to documentation and examples are also provided in the release notes.
  
  
• 00:30:00 In this section, Dmitry Smith, a principal software engineer at Microsoft, discusses the performance improvements made in version 1.12.0 of the ONNX Runtime. Customers had approached Microsoft requesting lower CPU latency and usage for inferencing, which prompted the improvements. The team focused on reducing memory allocations during inferencing and eliminating unnecessary logging, with changes made to the way code was written. The improvements resulted in reduced latency in some scenarios by a factor of two or more, and further improvements such as enhancing cache locality and thread pool utilization are planned for future releases.

[MetaQuotes]

v1.13 ONNX Runtime - Release Review

v1.13 ONNX Runtime - Release Review

Version 1.13 of the ONNX runtime was recently released with security patches, bug fixes, and performance enhancements. The update focuses on optimizing Transformer models for GPU quantization and adds support for direct ML execution providers that are device agnostic and support over 150 operators. Additionally, the release includes updates to the ORT mobile infrastructure for compatibility with new EPS, such as the XNN pack. The use of quantization to improve the performance of Transformer-based models is also discussed, with optimization of the CUDA execution provider to run the quantized BERT model and the use of quantized aware training to maximize accuracy while optimizing the ONNX runtime execution engine.

• 00:00:00 In this section, the speaker discusses the recently released version 1.13 of the ONNX runtime, which includes security patches, bug fixes, and improvements in performance. The update focused on optimizing Transformer models for GPU quantization and added support for direct ML execution providers. The latter is a machine learning API that supports over 150 different operators and is device agnostic. The speaker also mentions the new can EP execution provider, which was contributed by Huawei to support their Ascend 310 hardware. Additionally, updates were made to the ORT mobile infrastructure, which allowed for compatibility with new EPS such as the XNN pack.
  
  
• 00:05:00 In this section, the speakers discuss the ONNX runtime release v1.13 and how it works with any GPU that supports up to DirectX 12, making it easier to optimize for Windows machines. They also discuss the new operators and updates to the ONNX offsets in version 1.12. The speakers highlight how the new release has expanded support for different model architectures to make it easier to leverage the execution provider within ONNX runtime. They also discuss the new execution provider, Excellent Impact, and how it fills in performance gaps on mobile devices where handwritten kernels aren't available. The feature is currently enabled for Android, but the team is looking to add support for iOS and Xamarin or Maui builds in the following release. Lastly, they discuss the new feature in the release called optimizations for BERT model quantization.
  
  
• 00:10:00 In this section, the speaker discusses the use of quantization to improve the performance of Transformer-based models such as BERT. They explain how they optimized the CUDA execution provider to run the quantized BERT model and the use of quantized aware training to maximize accuracy while optimizing the ONNX runtime execution engine. The speaker provides examples of how to do BERT model quantized aware training and export the model with the tools available in ONNX runtime for further optimization. By better supporting the quantized aware training, the ONNX runtime can deliver further performance optimization while maintaining maximum accuracy. They mention that after the user follows the examples to export the model, the offline tools available in the new version of ONNX runtime can optimize the model for better speed.