Introducing PyTorch Live: Revolutionizing Mobile AI Development

By Raziel, Tech Lead at PyTorch

As the mission of PyTorch continues to evolve, we are excited to introduce a new initiative that will revolutionize mobile AI development. With the increasing demand for AI-powered applications on various platforms, it's time to simplify and streamline the process of building these apps. That's why we've created PyTorch Live, an innovative platform designed to make it easier for developers, researchers, and enthusiasts to build and showcase their AI models on multiple platforms.

At its core, PyTorch Live is built on top of the popular deep learning framework, PyTorch. This means that you can leverage the same powerful tools and libraries used in research settings to develop your mobile app. With PyTorch Live, you can take advantage of state-of-the-art AI models, such as MobileNet, while enjoying a seamless integration with the React Native ecosystem.

Our goal is to provide an end-to-end solution for building and deploying mobile apps that incorporate AI models. This means that developers will no longer need to worry about the complexities of porting their models across different platforms. With PyTorch Live, you can focus on what matters most – developing innovative and engaging applications that showcase the power of AI.

PyTorch Live offers a range of features designed to simplify the process of building mobile apps with AI. First and foremost, we've created an easy-to-use interface for integrating PyTorch models into your React Native app. This includes pre-built components for tasks such as image classification, object detection, and natural language processing. Our platform also provides a cross-platform data processing specification that abstracts away the complexities of processing input data on different platforms.

This means that developers can focus on adding their AI model to PyTorch Live without worrying about the intricacies of data processing. We've also made it easy for users to share and reuse models, thanks to our flexible data processing API. This API will be highly inspired by the PyTorch C++ and Python APIs, so you can transfer your learnings from previous uses of PyTorch.

In addition to simplifying the process of building mobile apps with AI, we're also committed to fostering a thriving community around PyTorch Live. We believe that collaboration and knowledge-sharing are essential for advancing the field of AI research and development. To this end, we've created a dedicated Discord channel where developers can connect with each other, share their projects, and get feedback on their work.

To get started with PyTorch Live, simply visit our website for tutorials and API documentation, check out the PyTorch Live GitHub repository for the source code, and join our Discord channel to connect with the community. We're excited to see what you'll create with PyTorch Live and look forward to sharing your projects on social media using the hashtag #PyTorchLive.

What's Next for PyTorch Live?

As we continue to develop and refine PyTorch Live, there are several exciting features and improvements on the horizon. First and foremost, we're committed to supporting a wide range of ML model domains, including those that work with audio and video data. We believe that this will enable developers to build more comprehensive applications that take advantage of multiple AI models.

We'll also be releasing a flexible data processing API that will allow users to customize their data processing pipeline. This means that you can choose to include the data processing as part of your ML model or have it be part of the app logic. Our team is working hard to ensure that this API is highly inspired by the PyTorch C++ and Python APIs, so you can transfer your learnings from previous uses of PyTorch.

In addition to these technical advancements, we're also committed to building a strong community around PyTorch Live. We believe that collaboration and knowledge-sharing are essential for advancing the field of AI research and development. To this end, we'll be hosting regular meetups and workshops where developers can share their projects, get feedback on their work, and learn from industry experts.

Overall, PyTorch Live is an exciting new initiative that's designed to revolutionize mobile AI development. With its innovative features, seamless integration with React Native, and commitment to community building, we're confident that this platform will become a go-to destination for developers, researchers, and enthusiasts looking to build and showcase their AI models on multiple platforms.

"WEBVTTKind: captionsLanguage: en- Hi, everyone.My name is Raziel,and I am a tech lead in PyTorch,and today I havesome exciting newson the developerproductivity front.As you probably already know,PyTorch's mission isto accelerate the pathfrom research prototypingto production deployment.This is important not onlyto reduce the costsfrom taking a modelinto product,but because we know bothresearch and product developmentare highly iterative,and being able to look backat product feedback to researchin quick iterationstranslates into better modelsfor the target use case.In order to make this happen,we operate under fourguiding principles.First, we always enablecutting edge research,and we want to codevelopwith many stakeholders.We are also modular,to allow users to only usewhat they care about,and we are performantand production grade.This has never beenmore important thanin the already big and growingon device ML ecosystem,where on one hand, being ableto easily go from ideato prototypeon the actual target deviceis key to meet the moreastringent resourcerestrictions of mobile devices,and on the other hand,we want to enable the largecommunity ofmobile developers .With the goalof reducing the frictionto create novelML based mobile solutions,we introduce PyTorch live.During the rest of the talk,Roman will cover the detailsof why exactly PyTorch live isand how it works.- Thanks, Raziel.If you're an ML researcheror mobile developer,this is for you.I'm Roman, and I'ma software engineer on the AIExperiences team at Meta AI.Today I will be talking aboutwhat PyTorch Live is,what PyTorch Livecurrently supports,what you can expectfrom PyTorch Live in the future,and how you canget started with it.Currently, PyTorch Liveis a set of toolsto build mobile AIpowered experiences,but it will continueto grow over time.These tools supportbackground mobile AIpowered apps that run on bothAndroid and iOS platforms.Now, instead of havingto write the same apptwice in two differentprogramming languages,PyTorch Live uses JavaScriptas the unified languageto write appsfor both platforms.To achieve this,PyTorch Live is poweredby PyTorch Mobileand React Native.First, PyTorch Mobile powersthe on device inference.Second, Reactive Nativeis a best in classJavaScript library for buildingvisual interactiveuser interfaces for Android,iOS, and other platforms.Both are active and flourishingopen sourceprojects with tons of examplesand tutorials.Here are three highlightsof PyTorch Live.First, a command line interfacewhich quickly sets up a mobileapp development environment,and bootstraps mobileapp projects.Second, a dataprocessing API,which preparesand integrates custom modelsto be used with PyTorch Live.Third, a cross platform APIwhich builds mobile AI poweredapps for Android and iOS.For example,if you want to showcasea mobile AI modelthat runs on android and iOS,it would have taken daysto configure the projectand build the user interface.With PyTorch Live,it cuts the costs in half,and you don't needto have Androidand iOS developer experience,and here's how it works.Let's start withthe command line interface,or CLI for short.The CLI reduces the setup costsfor mobile app developmentand helps you to setupa mobile ML project for PyTorchand mobile models in just a fewminutes using simple commands.The setup dev commandwill install developmentdependencies,includinglike Yarn,Watch Man and CocoaPods,SDKs like OpenJDK and AndroidSDKand an Android emulator.And this is a one-time install,so it takes a few minutesdepending on your Internetspeed and computer.Once the developmentdependencies are installed,the init commandallows you to initializea new PyTorchLive AI project,and this project containsall native code necessaryto run Android and iOS.It also includesworking examplesfor image classification,object detection andnatural language processing.Lastly, the run Androidand run iOS commands buildand run the app either on theemulator on a physical device.This is a greatstarting point for youto adaptany of these examples,and of course you can bring inyour own PyTorch mobile models.Now that we've setupthe mobile AI project,let's look at how to integrateyour own modelsinto PyTorch Live.One challenge when buildingAI-powered appsis integration with datawith from device sensorsor user input.The data processing stepsare often not part of the model,and need to beprogrammed additionally.For example, integratingan image classification modelto work with the camerainput requires custom datapreprocessing stepsto convert a three-channelRBG image to a tensor.In reverse, the model outputneeds to be post-processed,returning the consumable dataformat, such as an image class,and this would need to be donefor both Android and iOS.PyTorch Live offersa cross-platform dataprocessing specificationthat abstracts processing stepsand makes it possibleto integrate ML modelson different platforms.It allows you to moreeasily share modelswith others in the community.To add an ML modelin PyTorch Live,we first need to include sucha data processing specification.And this specificationis in a JSONformat that describes the dataprocessing steps for our model.Let me show youwhat that looks likefor an image classificationmodel such as mobile net.A specificationconsists of two parts,a pack definitionand an unpack definition.The pack converts input datasuch as an image to a tensor,the unpack converts the modeloutput into consumable data.The tensor from imagewhich forms an image to a tensorusing three transforms,a center crop transform,a scale transform,and an RBG nominalization.The unpack definitionwill compute the argmaxon the model outputand return it as a result.The specificationcan also define variables,prefixed with a dollar sign,which allows you passingin params with the model call.The processing specificationneeds to be bundledas an extra filewith the model,and this can be done duringthe model export in Python.PyTorch Live comes witha selection of transformationsimplementing best practicesfor image classification,object detection andnatural language processing.We plan on supporting a widearrange of ML model domainsand also releasinga flexible data processing APIexposing functionslike torch.argmaxor torchvision.transformsfrom the C++APIs directly to JavaScript.And this way, you can chooseto include the data processingas part of the ML model,or have it be partof the app logic.Now that we've setup the mobileAI project and prepared a model,let's see how to usethe model in an app.Building apps that workacross different platformsrequires expertise in multipleprogramming languages,and therefore increasesthe cost to showcase modelson different devices.PyTorch Live uses JavaScript,and it can be used to build appsfor Android and iOS.PyTorch Live comeswith a set of examplesfor image classification,object detection and NLP.If your model is inany of these categories,you can plugand play your modelwithout the need to programa user interface.But you can also buildyour own user interfaceto showcase your model.On top of basic user interfacecomponents from React Native,PyTorch Live providesa camera componentfor image-basedML model use cases,a canvas componentto give you the flexibilityto draw primitives,for example,drawing bounding boxes arounddetected objects in an imageor creating free-formdrawing canvasesto feed drawings intoa handwritten digit classifierand lastly a mobile APIto run ML inference.Let's have a lookat an exampleof how to call the mobilenet model from before.First, put the model anywherein the AI project folderwhich allows you to requirethe model using a relative path.Second, createa React Native componentto showcase your model.Third, add the PyTorchLive camera componentwith onCapture call back.Fourth, call the model with thecaptured image from the camera.You can pass in valuesfor variables to find earlierin the dataprocessing specification.Last, get the top image classbased on the model resultand use it in your app logic,and that's it.We'd love to seewhat you all create.If you publish your apps,tweet with the hashtagPyTorch Live,and we'll retweet some of themfrom the PyTorch live account.And here's how you can getinvolved with PyTorch Live.Visit the PyTorch Live websitefor tutorialsand API documentation, check outthe PyTorch Live GitHubrepository for the source codeto make suggestions,report issuesand contribute pull requests,and join the discord channelto connect with the communityand start discussions.So what's next for PyTorch Live?This is our initial approachof making it easierfor you to build mobile appsand showcase the ML modelsto the community.It is also an opportunityto take this a step further,by building a thriving communityfor ML researchersand mobile developers,and share and utilizePyTorch mobile models,and engage in conversationswith each other.There are also short-termtactical improvementsthat will be comingto PyTorch Live in the future.As mentioned earlier,we're working ona flexible data processing APIto support custom dataprocessing steps.The data processing APIwill be highly inspiredby the PyTorch C++and Python APIs,so you can transferyour learningsfrom previoususes of PyTorch.We will also be adding supportfor other ML domainsthat work with audioand video data.We hope you have a great timeusing PyTorch Live.Thanks for watching.\n"