**Optimizing Mobile Models with PyTorch**

One of the most significant challenges in developing mobile applications is optimizing models to run smoothly on devices with limited resources. In this article, we will explore how to optimize a mobile model using PyTorch, a popular open-source machine learning framework.

To achieve optimal performance, it's essential to understand the importance of layer fusion and quantization in mobile models. Layer fusion involves combining multiple layers together into a single operation, which improves performance in memory footprint. This technique is particularly useful when working with convolutional neural networks (CNNs), as shown in the documentation for the latest PyTorch version.

When we fuse two or more layers together, they can be combined into a single neural network layer. However, there are certain combinations of layers that cannot be fused together. In this example, we will use convolutional-batch normalization-Darrell (CBND) modules, which are commonly used in CNNs. By fusing these modules together, we can reduce the number of parameters and improve the overall performance of our model.

Another critical aspect of optimizing mobile models is quantization. Quantization involves reducing the precision of the model's weights and activations from 32-bit floating-point numbers to 8-bit integers. This technique reduces memory usage and improves inference speed. When we quantize a PyTorch model, we need to ensure that it can handle the reduced precision without compromising its accuracy.

To demonstrate these techniques, let's consider an example project that includes an optimized version of MobileNet trained on the ImageNet dataset. We will also include a text file containing human-readable labels for the thousand categories in the ImageNet dataset and an image file used as input for our classifier. By following these steps, we can create a PyTorch model that runs efficiently on mobile devices.

**Setting up the Project**

To begin optimizing our MobileNet model, we need to set up the project by adding necessary files to our Xcode project. We will add two files: `torch_module.h` and `torch_module.mm`. These files contain the C++ wrapper for PyTorch's LibTorch library, which allows us to call PyTorch functions from our Objective-C code.

The `torch_module.h` file contains a header that defines the `PyTorchModule` class, which is used to initialize the PyTorch runtime with a model. The `torch_module.mm` file contains the implementation of this class, including methods for initializing the model and performing inference.

**Adding the Model to the Project**

To add our optimized MobileNet model to the project, we need to create an instance of the `PyTorchModule` class and initialize it with the model. We will also load the image resource used as input for our classifier and resize it to the required size (224x224 pixels).

**Loading the Model and Image**

We will use a lazy-loaded instance of our PyTorchModel to load the optimized MobileNet model from a file. The `torch_module` object is initialized with the model, which allows us to call its methods from our Objective-C code.

We also need to define an array of strings containing the human-readable labels for the thousand categories in the ImageNet dataset. These labels will be used as input for our classifier when making predictions.

**Inference and Prediction**

To make a prediction using our PyTorch model, we need to create a buffer containing the input image data, resize it to the required size, and normalize it according to the model's requirements. We then pass this buffer to the `predict` method of the `PyTorchModule` object, which returns the predicted label or category.

Finally, we display the predicted label in our text view using the `label` property of the `PyTorchModule` object. By following these steps, we can create a PyTorch model that runs efficiently on mobile devices and provides accurate predictions for image classification tasks.

**Running the App**

To test our optimized MobileNet model, we will run our app on an iOS device or simulator. We expect the app to start up quickly and display the predicted label for the input image. Note that this may take a minute to run the first time due to the initialization of the PyTorch runtime.

By following these steps and techniques, developers can optimize their mobile models using PyTorch and create efficient, accurate, and reliable image classification applications for iOS devices.

"WEBVTTKind: captionsLanguage: enhello and welcome to the pike torch summer hack 2020 I'm Brad Heights and I'm a partner engineer with the PI torch team if you're not sure what a partner engineer is it means that my day job is making sure that developers like yourselves get the most they can out of PI torch and related products in this video I'm going to give you a walk-through of setting up the PI torch mobile runtime in an iOS project to follow along you'll need Xcode version 11 or higher on Mac OS you'll need to have cocoa pods installed and if you want to deploy to a device you'll need an Apple Developer account in an iOS device running at least iOS 12 you should also be using PI torch 1.5 or higher PI torch offers native runtimes for iOS and Android allowing you to bring your machine learning app to mobile devices including the library in your project as a one-liner but there is more to do getting set up for best performance on mobile in this video I'm going to show you how to set up your project to include the PI torch runtime how to export your model to torch script PI torches optimized model representation how to further optimize your model for best performance how to add your model to your Xcode project and how to call the model from your code let's dive in for this demonstration we're going to build an image classifier into an iOS app first we'll create the project I'm going to create a single page app with UIKit UI now close Xcode and switch to your command line so here's the pod file we're going to attach to this project you can see it specifies iOS 12 and PI torch 1.5 so now pod install that and we'll open the workspace now instead of the project it's important when you're using cocoapods and then we'll go ahead and run the project just to make sure all as well and there's the sim nothing there yet no surprise we'll stop that the next thing we'll need is a UI I'm going to have one image of you one button and one textview in my app the image view and text view will have outlets in the view controller and the button will have an attached action rather than making you watch me set up a bunch of storyboard constraints we'll do a little movie magic Yola now I'll show you how to set up and optimize your model for use with pi towards mobile so first thing first we'll import PI torch and torch vision and this is a good place to point out that you should be using PI torch 1.5 or higher for this example in the next cell we're going to create a PI torch model object now for the actual app I have a pre trained optimized model ready to go but for the optimization process I wanted to show you this on a custom model so that you could duplicate the process with your own models this model happens to contain some very common layer types A to D convolutional layer a to D batch norming layer and a rectified linear unit for activation and the forward function just strings those three operations together so now that we have our model how do we optimize it first thing let's get an instance of the model in my get model helper you'll notice that besides just instantiating the model I call MD Val so eval turns off things in the model that you don't want on during inference time training only layers like drop out automated gradient tracking all this training related stuff meets up CPU cycles and we don't need it for inference so we're gonna make sure the model is in eval mode the second thing we're going to do is some layer fusion fusing layers means taking multiple operations and combining them together into a single operation this improves performance in memory footprint now with diffuse modules method that I'm going to show you there are only certain combinations of layers that you can fuse together I'm going to refer you to the documentation for the latest information on that but here we're going to try to use together convolution bachelor-man Darrell you once modules are fused the next one we're going to do is quantize the model PI torch tensors default to using 32-bit floating-point numbers as their underlying type when we quantize we're going to change that to an 8-bit integer this will perform faster and reduce the models footprint both on disk and in memory the I don't think we're going to do is save the model as torch script torch script is an optimized format for your model including both your computation graph and you're learning weights it's meant to be consumed by the PI torch in just-in-time compiler or JIT which is included with the PI torch mobile runtime so once it's exported will save now there are subtleties to layer fusion into quantizing your model that you'll want to be familiar with when you're optimizing your own model for use with PI torch mobile all of this is covered in the PI torch documentation for quantization which I encourage you to check out now I'm going to pull in the resources for this project they include the model so in my case that's an optimized version of mobile nephew trained on the thousand categories of the image net data set we also have a text file that contains the human readable labels of those thousand categories and finally we have an image file for our classifier to work on there should be links to a sample project that includes these resources wherever you got this video now we can flip back to Xcode and I'll add the three files to my project now how do we call pi torch from our code because pi torch is implemented under the hood in C++ with Lib torch we need a wrapper for that C++ library so we're going to go ahead and create a new group in the project we're going to flip back to the command line and I'm going to add these two files torch module dot H and torch module dot mmm again these should be available wherever you've got this video and then Xcode is going to offer to make a bridging header for me and I'm going to let it do that so you can see here in the header there are only two methods that are really interesting one which initializes the PI torch runtime with a model which is store as a file and it's our model of resource in our project and the second one that does the actual prediction when you pass it in a buffer full of image information and you can see the C++ bridge code and torch module mmm and now add my image resource to my image view so we can see what it is we're classifying and speaking of images I'm going to add one more source file and this is going to be in addition to the UI image framework class that allows us to resize it more easily the particular model I'm using because it was trained on image net expects three color images size 224 by 224 pixels so this helper role helped me achieve that we have some stuff to fill in the my viewcontroller first I'm going to add a lazy loaded instance of our module our model you can see I'm loading it from the file and initializing the torch module object with it also I have lazily loaded array of strings the labels those thousand human readable labels for the categories that the model recognizes I'm also setting that up here now you notice we've got an error on the torch module type it suppose we're to go back to the bridging header and make sure we include torch module page if for whatever reason your project isn't recognizing the type go check the bridging header make sure you've included a header for torch module dot H now we're going to fill in the body of our inference buttons action and you can see here we're going to load the image we're going to resize it to do 224 by 224 we're also going to normalize it so just as with the size of the image our model expects the image to be normalized in a certain way and we have a method to do that and finally we take our image buffer and we pass it to module dot predict after that we retrieve the label or the predicted category and display it in our text view so now at last let's run our app yeah you'll see it starts up and there's our kitten we press the infer button and be aware that even if you run this on a laptop it may take a minute to run the first time but there we are our model thinks that this cat is a cat and that is how you get a pi torch model onto iOS thanks for participating in the hackathon and thanks for watching this videohello and welcome to the pike torch summer hack 2020 I'm Brad Heights and I'm a partner engineer with the PI torch team if you're not sure what a partner engineer is it means that my day job is making sure that developers like yourselves get the most they can out of PI torch and related products in this video I'm going to give you a walk-through of setting up the PI torch mobile runtime in an iOS project to follow along you'll need Xcode version 11 or higher on Mac OS you'll need to have cocoa pods installed and if you want to deploy to a device you'll need an Apple Developer account in an iOS device running at least iOS 12 you should also be using PI torch 1.5 or higher PI torch offers native runtimes for iOS and Android allowing you to bring your machine learning app to mobile devices including the library in your project as a one-liner but there is more to do getting set up for best performance on mobile in this video I'm going to show you how to set up your project to include the PI torch runtime how to export your model to torch script PI torches optimized model representation how to further optimize your model for best performance how to add your model to your Xcode project and how to call the model from your code let's dive in for this demonstration we're going to build an image classifier into an iOS app first we'll create the project I'm going to create a single page app with UIKit UI now close Xcode and switch to your command line so here's the pod file we're going to attach to this project you can see it specifies iOS 12 and PI torch 1.5 so now pod install that and we'll open the workspace now instead of the project it's important when you're using cocoapods and then we'll go ahead and run the project just to make sure all as well and there's the sim nothing there yet no surprise we'll stop that the next thing we'll need is a UI I'm going to have one image of you one button and one textview in my app the image view and text view will have outlets in the view controller and the button will have an attached action rather than making you watch me set up a bunch of storyboard constraints we'll do a little movie magic Yola now I'll show you how to set up and optimize your model for use with pi towards mobile so first thing first we'll import PI torch and torch vision and this is a good place to point out that you should be using PI torch 1.5 or higher for this example in the next cell we're going to create a PI torch model object now for the actual app I have a pre trained optimized model ready to go but for the optimization process I wanted to show you this on a custom model so that you could duplicate the process with your own models this model happens to contain some very common layer types A to D convolutional layer a to D batch norming layer and a rectified linear unit for activation and the forward function just strings those three operations together so now that we have our model how do we optimize it first thing let's get an instance of the model in my get model helper you'll notice that besides just instantiating the model I call MD Val so eval turns off things in the model that you don't want on during inference time training only layers like drop out automated gradient tracking all this training related stuff meets up CPU cycles and we don't need it for inference so we're gonna make sure the model is in eval mode the second thing we're going to do is some layer fusion fusing layers means taking multiple operations and combining them together into a single operation this improves performance in memory footprint now with diffuse modules method that I'm going to show you there are only certain combinations of layers that you can fuse together I'm going to refer you to the documentation for the latest information on that but here we're going to try to use together convolution bachelor-man Darrell you once modules are fused the next one we're going to do is quantize the model PI torch tensors default to using 32-bit floating-point numbers as their underlying type when we quantize we're going to change that to an 8-bit integer this will perform faster and reduce the models footprint both on disk and in memory the I don't think we're going to do is save the model as torch script torch script is an optimized format for your model including both your computation graph and you're learning weights it's meant to be consumed by the PI torch in just-in-time compiler or JIT which is included with the PI torch mobile runtime so once it's exported will save now there are subtleties to layer fusion into quantizing your model that you'll want to be familiar with when you're optimizing your own model for use with PI torch mobile all of this is covered in the PI torch documentation for quantization which I encourage you to check out now I'm going to pull in the resources for this project they include the model so in my case that's an optimized version of mobile nephew trained on the thousand categories of the image net data set we also have a text file that contains the human readable labels of those thousand categories and finally we have an image file for our classifier to work on there should be links to a sample project that includes these resources wherever you got this video now we can flip back to Xcode and I'll add the three files to my project now how do we call pi torch from our code because pi torch is implemented under the hood in C++ with Lib torch we need a wrapper for that C++ library so we're going to go ahead and create a new group in the project we're going to flip back to the command line and I'm going to add these two files torch module dot H and torch module dot mmm again these should be available wherever you've got this video and then Xcode is going to offer to make a bridging header for me and I'm going to let it do that so you can see here in the header there are only two methods that are really interesting one which initializes the PI torch runtime with a model which is store as a file and it's our model of resource in our project and the second one that does the actual prediction when you pass it in a buffer full of image information and you can see the C++ bridge code and torch module mmm and now add my image resource to my image view so we can see what it is we're classifying and speaking of images I'm going to add one more source file and this is going to be in addition to the UI image framework class that allows us to resize it more easily the particular model I'm using because it was trained on image net expects three color images size 224 by 224 pixels so this helper role helped me achieve that we have some stuff to fill in the my viewcontroller first I'm going to add a lazy loaded instance of our module our model you can see I'm loading it from the file and initializing the torch module object with it also I have lazily loaded array of strings the labels those thousand human readable labels for the categories that the model recognizes I'm also setting that up here now you notice we've got an error on the torch module type it suppose we're to go back to the bridging header and make sure we include torch module page if for whatever reason your project isn't recognizing the type go check the bridging header make sure you've included a header for torch module dot H now we're going to fill in the body of our inference buttons action and you can see here we're going to load the image we're going to resize it to do 224 by 224 we're also going to normalize it so just as with the size of the image our model expects the image to be normalized in a certain way and we have a method to do that and finally we take our image buffer and we pass it to module dot predict after that we retrieve the label or the predicted category and display it in our text view so now at last let's run our app yeah you'll see it starts up and there's our kitten we press the infer button and be aware that even if you run this on a laptop it may take a minute to run the first time but there we are our model thinks that this cat is a cat and that is how you get a pi torch model onto iOS thanks for participating in the hackathon and thanks for watching this video\n"