PROJECTIONS, Episode 21 - Valve's SteamVR 'Knuckles' Controllers

The Art of VR Character Animation: Creating Believable Interactions

We're doing some fairly complex math and creating different levels of interest in the environment to make people's eyes behave naturally, even though our current model is using an old version. Normally, normal eyes are actually misbehaving because we're using this system. It looks like his eyelids are flipping up and so he blinks up and turns down, which can be very disconcerting. However, I thought it was cool when he expressed himself in a different way, even though it was not what people would naturally do.

When it comes to gestures, we're trying to identify things that people are naturally drawn to, such as other people or interesting movements. But, thinking of taking gestures and also volume is crucial to making people believable and avoiding the uncanny valley effect. The latter can occur when something is almost, but not quite, like what we'd expect from a real person.

We're using 2D avatars, which are still pretty cool, especially with Jeremy's help. But, taking things like saccades – those little tiny infinitesimal movements of the eyes – into consideration is important for creating realistic interactions. People naturally glance around even when they're looking at someone, and we need to mimic that or else people will feel uncomfortable.

Another aspect of VR animation is movement. We're not just tracking the controllers, but also adding a little bit of idle animation on the fingers so that users don't have straight paddle hands. This can be a problem, as it's something that freaks people out when they see unnatural movements in real life. However, if we talk and our eyebrows don't move, that's even weirder.

We're also experiencing tracking issues with the lighthouses in the test room. It's not ideal to have three people in the same space without proper planning to avoid occlusion issues. If one person's hand is between two others, it can get blocked by either of them. We've had some problems with this in the past.

Jeremy recently used the knuckle controller for a bit and found it cool. However, he wishes we could see his hands as they were actually moving in VR. This was something we didn't do with the wands because we couldn't get the tracking to line up perfectly. It's an area we might revisit if possible.

As developers, we're constantly experimenting with these new controllers and pushing the boundaries of what's possible. Our platform is allowing us to create realistic virtual environments that were previously impossible without VR technology. We're even making cartoons that people have never been able to make before without VR.

If you're interested in learning more about our work or how to get involved, you can contact me at info@foodforVR.com or check out our website at foodforVR.com. There's plenty of information there for anyone looking to start creating their own virtual animations and cartoons with us.

"WEBVTTKind: captionsLanguage: enyou hey everybody its norm from test is Jeremy from tested today we have a special treat for you guys we have a whole sub here with the vive headset but we're not gonna be talking about the headset or these controller or even the pro audio strap we're your probably shop which is pretty good we have a special guest that's gonna bring a new piece of hardboard let's bring him right now it's foo BR Will Smith alright that's it I will confer well here because one of the reasons we want to learn more about food what you've been doing the past year and also oh my god is it that this is so 2016 as well it's this old technology Jeremy and put these down we don't talk about those any actually those are still really good we use them all the time barely but the function valve were kind enough to send us some duxelles prototypes to use for an upcoming project which i terribly can't talk about right now but we'll be talking about in the two and a half three weeks maybe around September 6th perhaps if you want to follow me on twitter at twitter.com/zeromd ZAR the new valve knuckles controller yes that they unveiled last year actually at a talk and then they started to show up in developers hands yeah what do you think I'm really impressed like they solved the problems that I had with the one like I loved oculus touch controls because a definite way to hold them they have the grip button and the trigger button separately the grip buttons actually usable on the touch controllers which is important and and like it's just it's just a better controller design than the lungs for what we do like we need a definite hand position like grip ergonomics it's one thing one of the reasons like you said I think the ocular touch controls are more ergonomic than the the wide controllers maybe less versatile and how you can hold them is because your resting position of your hand is basically that grip it's almost like the it's a neutral prepare a neutral poses right all right but the knuckle controller which isn't this is not HTC this is still valve experimentation yes let's you have a controller in your hand without gripping it isn't at all that's kind of the magic so yeah there's a neoprene there's a knee opening of sleeve that you kind of put your hand between that and the controller and then the other magic part about this is the controller the back of the surface of the controller is capacitive so that means it can detect when your fingers are on it with a great deal of sensitivity the same way that if you put your palm on your iPad screen your phone screen it'll be able to tell there's a bunch of fingers then so as far as the strap goes I know there's a tensioning that's just a tighten it until it's comfortable and you don't ever have to do that again I mean if you put on something with tiny hands maybe it's elastic though so yeah I usually go in and off at once and have it adjusted it that's interesting it's interesting because whenever now I'm not gripping at all it really is attention I feel most of that tension on the bottom part by my by my pinkie so you might need a little too tight you want it to be kind of even across the top of your knuckles if you want to ride the at least what I found is the best is to ride the strap across the top of your knuckles and you said they're capacitive sensors along my grip here along the inside edge here yeah which they in the software divided believe in the three segments for each through your fingers and and so it calibrates that each time you put it on so when you put those controllers in your hand you grip them and you release your grip any release in the gripping at least three or four times until the fingers start tracking correctly or close to it you first put them on you'll see gnarled fingers representations and they're all over the place and then it dials it in as you as you did that tap and release and it figures out which fingers are which so with the touch controller with the active session feel of these two fingers would never be tracted no no way shape or form right but they are analog because it is a button that you depress well this one would be a button right I'm right about it there's a little thinking you're right you're right the reason fingers not analog on oculus touch controllers so there's nothing it's just a rental finger is just you know that long yeah but post capacitive when you touch it and then this analog going forward oh it's with these three it's it's almost made a lot of the entire travel see I thought was just capacitive when you touch and then it moves in when you pull the trigger but oh no you can have them grab yeah so um but then also the top is a little different like there is button parity between this and your standard vibe controller you know there is a touchpad here it's a little bit different shape so the way the way the buttons work is the round one is the steam button the one that pops up the steam menu the one on the inside edge you know on your right hand it to your left left button on your left your right button is the like grip button I believe and then the one on the outside edge is the is the button that's above the pad it's like the one that I'm reviews right users weapon select whatever make it in but then the circle pad is totally redesigned so it it's the same more or less the same surface area but much deeper it's a much deeper curve than the it's a bowl rather than like a disc is what I would say on the traditional one mhm and of course these are prototypes the the final design may change they're not committing to anything yet they're just putting these out there I think one of the things that they learn from the initial long design is that how they think people are going to use them and and this is also new that how people use this stuff in the real world versus how you think it's you can't design for how people are going to use it because nobody knows how they're going to use it yet you have to try stuff I can say I like that this is now a unique button the actual home the steamvr happen because I would always forget is at the top of the bottom one in game and always always flip-flop on yeah there's much better button place when you get one additional button also of course there are plenty of the sensors around the controller so you're not going to get occlusion I like that it's rather something I mean you never really hit the back of your hand in VR it's mostly the poem and then and then there's of course the trigger which which is organ AMA canned as I do today is your stage trigger and it's capacitive as well and the touching on the front there buttons I believe are as well not sure about the face surface but I would expect that to be in the final speaking of someone who literally has a gash in my drywall about that big from a friend who was playing something and yeah out with the vibe controller across the room this is the killer it's order for this room like I knocked my watch off a while ago and I was shadowboxing yeah I know yeah I'm not the man fell off it's not great I'm like throwing rope not coming up I love how crazy you are with my one-of-a-kind prototype years old but you've been using and you can cemented this in a version of food so what we're going to do is want to put on some headsets and jump into a food environment you've created talk about what you've been working on for the past two years I guess ready to jump in yeah all right one two three whoa wait a minute we're a virtual world now something's wrong with my model oh wait your hair Jenner I look a little funny dude you look awesome okay you look so cool so will is on the far right I'm only wearing using wills model is in the middle and I'm here on the right with a pixelated tessellated version of me a low poly version this is this is food all right yeah this is well this is one of the things we've built so for the last year but we started out doing this VR recorded shows right shows that happened in be odds you can watch in VR turns out there's not a lot of headsets on the market we like making money is nice so we've started doing 2d animation we did a the first thing with adults waiver earlier this year in January whose some really exciting stuff to come up coming up in the next couple weeks so I can't really talk about now this is so weird or maybe not you make live cartoons that's the takeaway we're using the flu animation to make guys that look like me and look like Jeremy and looks like Carl from Aqua Teen Hunger Force and we can animate them live so put a performer in instead of having to do a bunch of 2d animation I think you lost an arm there Jeremy then we can we can go in and do things they're possible in traditional animation like live call-in shows or you know putting characters in places that you wouldn't expect to see them and all sorts of other interesting things like that so some of interest saying people should know because we're going to switch between this virtual camera on the live camera is that our relative space it like you are actually right there in the real world and Jeremy is actually right there on TV I yeah there you go I just touch Jeremy is that like part of the foo formula is is that what you're going for it could could it be done remotely well so you can do it remotely we've recorded some pilots where one of the performers was in LA and one of them was on the East Coast we if you want to do something like live on stage or in a live context like we're doing right now then then doing it where we're all in the same physical space and virtual space makes a lot of sense but there are complications because for example I get over close to Jeremy without his lips will start flapping or maybe not because you're hitting my microphone hitting his mic alrightlet's flapping is my other action going on it Jeremy's isn't suicide might be Laurie sure checking one two I don't see you moving right now but your eyes are tracking nicely but we do all the animation procedurally so that this is the interesting thing about what we built is all of the data that we're getting to animate these bodies it comes off of two hands and the head that you get from the vibe right we can add other things like we have some HCC tracker pucks that we can put on feet and back and stuff like that to get more data and do a little bit better job below the waist but for the most part we with machine learning and AI K and all sorts of terrifying that we can we can animate a believable human out of just three three count them three can I do it three points of data three so can we talk about that for a second yeah we are using the knuckles controllers so yeah in foo what was it like to actually incorporate this hardware into your game well valve set some controllers to us and then I went to FedEx and sent one of the controllers that he sent to us to my lead dev in New Mexico and then like two days later it was working so it takes two days no it was um so Andre Infante who's our lead dev spent a couple of hours working on it and got it up relatively quickly ago we have to change like you'll notice with my avatar especially there's some stuff that changes and norm norm is using the old ones as you can see Jeremy and I are both using knuckles and basically because there's a definite position that you hold the knuckles you know where it's where it's always in the same orientation on your hand then then we have to change the way the math works that defines what the angle on the pans is but other than that it's all more or less the same and this is a more consistent angle than you were getting from the original ones yeah with the world if you look how norms holding the ones or you don't put your hands on your back also for whatever you do map mask rise when you put your hands behind your back but yeah you could hold the ones in different ways you could hold them like a baseball bat you can hold them like a golf club you can hold them I eat a Harry Potter style magic wand and this there's only really one correct way to hold the controller's and that matters because you're doing what's called inverse kinematics in order to determine where all the body parts are yeah it's on the only three tracking point so we well I mean we've inverse kinematics we use some machine learning we use a bunch of other stuff we did a GDC talked about it there BRDC talked earlier this year that explains a little bit more about how it works but the upshot is that yeah the the the relationship between the position of your hand the orientation of your wrist the Orient have your elbow and shoulder is much easier to predict if you know how people are holding the controller interesting so with these knuckles controllers I can actually pull one finger out two fingers out I can make a big gesture again I already did that Wow I love you guys I don't really mean this so because the what you're getting is capacitive sensors I can repoint them on CapSense your middle finger your pinky a ring finger and your pinky how are you getting analog them yeah that's insane okay so the back of the controller is capacitive so that means that it can detect when your finger first hits it at the lowest knuckle to your palm as well as when the tip of your finger wraps all the way around and that means that you can infer the position of the hand you're not measuring directly that you're you know you that's why when you first put the controllers on or when you change your your when you wiggle them around a little bit you have to kind of grip in and out a few times yeah and you just keep doing that until the fingers look right I have to say the amount of granularity that you get about where fingers are is better than I expected it would be right like look you can nut without calibration on your end why I've open my hands at least three times and that's all it takes right yeah it constantly is calibrating and refining and inferring and a lot of that stuff happens on the driver level well what we've built here is our quickest dirtiest implementation you know it's literally Andre working on it for a few hours over a couple of days and there's definitely room for refinement there's more information we can infer from what we're getting from the controllers and frankly these models my model wasn't really built to be rigged I mean we put bones in the fingers so that we can animate them individually but but there are things that we would change if we were doing it from scratch right now Jeremy when you release the knuckle controller so you're not actively gripping what is your natural pose they put on your side is it like that okay and you're not holding on right now you know gripping at all nope you're just it looks pretty natural yeah pretty much I mean that's pretty much how I stand I would guess especially if I was in an Old West oh yeah yeah yeah so of the our characters are all bowlegged we do that so that we avoid can collision of the legs when they cross over in front of each other it's just it was it was an artistic choice kind of directed by necessities of the engine so what you've built into your tool will including like the models the your I K but also with the way we're filming this is a camera system yeah so that's the other part of this if you want to make 2d content you need a way to get to the content back out and the easiest way to do that is to give people a camera right so the version of the camera that Gunther is controlling right now is actually a drone camera at models a drone camera and he's flying it around within a month one controller and it works difficult off the Pam to control so it's left stick is up and down a pan left and right and then right stick is forward backwards left right straight right and then you can press a to switch from up and down on the left stick to tilt mmm I would help to do roll which is a discomforting Emily a zoom on this because you know it turns out there's no optics it's all virtual so we just changed that will be on the camera dynamically when you zoom in and out so obviously that gives you camera control tip a gamepad no tracking require but you also have the ability to track using a track to tour yeah so we also built a rig that lets us put like a vibe control or HCC tracker on just a shoulder-mounted rig and we can mount that so that a camera person can get in this space and and shoot directly up in the face you get handheld handheld tracking and all the stuff that comes with that we can motion smooth that out we could build a one of the people we've been talking to what's a dolly so that you can dolly across oh right there's something else we laid on tracks that's cool um I mean we wouldn't actually put the real tracks there we would place them in the world and plan out those shots and event better than real world photography exactly we've done things like put a green screen in so we did the carl thing the aquitine it's a pigskin puck nuptse prognostications from the pigskin wizard wy z ze ard if you want to check that out and you can see we basically built a green screen for him so they can host him on anything so like it would be a virtual avatar yeah instead of this cartoonish living room environment it's flat frame walls that's exactly right Gunther if you press G on your keyboard that might just pop up I'm not sure no did it it made the camera green made the camera I feel that's the other thing that's worth matching we could put as many cameras in as you want so we can do like a traditional we can shoot like a traditional 4 camera sitcom-style where we have four cameras you know main conversations secondary conversation wide two-thirds all that kind of stuff is possible but then you can actually do the post later right you can choose what angles you want later you don't have to do all that you can do that as well you can record an engine just like we do the are you ply the cameras around you can record off of that and post if you want you can also do like a live mix so one of the things that we've had is that you like one of the keys to making this work for people is fitting it into their existing content pipeline so so for example we let you hook up to say a TriCaster you hook up your capture computers whether it's one that's a 1080p machine or two that are 1080 machines or one that's a 4k machine that splits into four to any Pequod's leave a virtual camera in each of the quads and then they pipe into a normal video mixer they can handle it just like they would you know an episode of The Tonight Show or you're live 30 rocks or that's year you can whatever virtual performance capture pretty much with these versatile cars and now with the precision of individual finger movement well and deposit into any type of video and the benefit of doing the virtual performers in this context instead of say putting them in a mocap suit with the ping-pong balls and all that stuff is that here the performers can actually see the other performers as they're going to be you know if we scale somebody up or down they'll be the right size if they're picking up a prop they're interacting with something then they can you know use it in the space of a storm button a lot which button is if it's this one so you know we and we can do things like that are impossible in real world like right in space you can't grab it here I'm sorry um but you know you can you know rung up I just out board it again donation has left the building yeah I'm over here yeah I'm the man in the money this is my favorite thing to do when you put new people in VR by the way just tell them to draw a box yeah and watch them draw a 2d projection of a box right drop the chalk II I did so yeah that's pretty much it that's what we've done knuckles is another part of what we're doing I mean it obviously gives us a lot more fidelity in control over the animation lets us do things i mean we can programmatically pick this up I think this chalk looks okay you can grab it for my hands if you want Jeremy to squeeze the it's not not to him how do I draw you draw by squeeze pressing the left but the it's your right hand the left button there you go cool if you find that using knuckles it's more comfortable and you you're using it for longer periods and then you would be grabbing a holding you know one of these wand controller um it's uh I haven't noticed that the battery life on the knuckles prototypes is a little bit lower than the ones um but I mean it's nice to not have to worry about dropping the controller and not have to always happens in your hand and this way you can leave your hands-free and like once you get used to having your hand three you can make more natural gestures there's a fall somewhere in here I don't know where it went how do I remove my robot pick up the chalk again and then press the circle pad peacock has left the building I'll go get the truck oh okay I knocked the camera down oh I just rode up again I did I'm going to clip through you guys for a sec I got it early it wasn't real I mean the the stock that it looks seriously is like actually looking at the camera the tessellation sort of blends into a real photographic image and I feel like I'm watching television norm over there right I mean that's the thing is you get you dwell in that cameras far away so we're dealing with resolution limits but like but the human motion is there which is enough to create a sense of realism exactly right and what we're seeing in the headset which you guys can't see is a box that looks like a preview of it's a basically a preview what Gunther sees on his monitor so you know we're able to look at that and we can tell when the animations getting edgy when you do something nightmarish like put your hands behind you and like weird melty things happen or if you're clipping through your body or whatever you can correct that the performance can crack down on the fly which we very quickly learned was an important thing one of the things that you first demoed with foo was incorporating a fire watch game aesthetic into your engine is that something you're still doing is that a imagine glorious process yes I mean obviously we're definitely still doing that we're still we still have episodes of the Kickstarter to fulfill we're super late and I'm sorry it turns out we we didn't really anticipate how long it was going to take us to do things like track down horrible bugs and and things like that but we've recorded we have two more episodes of the foo show in the can that are ready to go one is features total war war or two and we go inside we get human scale with Warhammer dragons and wizard men and characters it's really awesome the other one is even more incredible we go and talk to a biochemist biophysical chemist who builds nano machines out of DNA named Sean Douglas and we go full powers of 10 we start out in his lab and we get smaller and get smaller and gets smaller and gets smaller until we're standing someplace that people have never been before so I think you'll be really excited by that and you get to a little bit better understanding of how and why you might want to build nano machines out of DNA there's an example where I would love to see eye tracking you know yeah it's not so much for Vogue elated rendering or cope we know words for my purposes it's for your purposes that I can actually feel like I'm with you even more well so we eye track now you see I'm watching you nor might be more interesting than you than me too but it's all simulated well it turns out simulated is good enough when you can't really look around inside the headset that much so we do we do some fairly complex math and we add we make different things in the environment different levels of interesting so that people their eyes behave naturally normal eyes are actually misbehaving right now because he's using an old version of the model I thought I was getting his eyelids flip up and so he blinks up and turns down which is very disconcerting oh I thought that was cool because it looked like he was expressing himself well so when he but he if he gets loud his eyebrows will go up I think yeah my exact yuge I like that you've cheated the system by identifying things in your world you know people are going to naturally gravitate toward of other people obviously but thinking of taking things like gestures and also volume well there's a lot of things that are interested that you have to mimic in order to make people believable and make people not squeezed out by them and we're much better off but better with it now when we're using say the 2d avatar that Jeremy and I are wearing then you're older tessellated one but things like saccade saccade are those little tiny infinitesimal movements of your eyes we're kind of always glancing around even if we're looking at somebody more eye contact and you have to mimic that or else people get squeezed out it's uncanny valley territory well they've all rubbery they're if you're what's mold watcher but but things like the the movements of the fingers you need to be able to move the fingers a little bit even if they're even if we're not tracking them with the controllers we do just a little bit of idle animation on the fingers so that you don't have straight paddle hands because that's something that again freaks people out yet if you talk and your eyebrows don't move that's really weird so we had a little bit of eyebrow movement that bounces with the cadence of your voice just basic stuff like that Oh tracking issues okay is that a problem is it tracking an issue we get the tracking issues are the problem we're having here the your lighthouses and the tested office are a little bit lower than I would normally go to put three people in the same space if you want to get three people in the same States you really want to shoot down on the on the play area just so that you don't have occlusion issues because if say your right hand which is between Norman the lighthouse then it'll it'll get blocked when it doesn't see either lighthouse it flies off to the origin but one of you better than ever clean my controller in the lighthouse exactly Oh Jeremy how you feeling after using the knuckle controller for a bit oh it's cool it's cool yeah I wish that I could see my hands as but that's just a matter of the software so the scene your hands is something we didn't do with the wands because we could get it to line up one two one and then normal I found when we did the the vikon motion tracking thing like three four years ago now yeah it was that was really discomforting when you can't when you're when you see your hands in VR and they don't line up with the real hand I think with the knuckles we're probably going to go back and retry it because it seems like it's a little bit of a better another opportunity to show real hands I am not about to drop them that's one thing yeah I think even not a time we've been in here I'm aware I'm holding something I'm never forgetting the fact that I'm holding these controllers and you are you more relaxed so your hand position more or less because you know that actively grow you know I'm so I'm so used to having Digger controllers that half the time I probably have been gripping in yeah it takes a while to break yourself of that have exact route but uh you know it's nice to be able to let go and I can imagine playing a game or if I'm going to grab something and physically actually grabbing it rather than just pushing a trigger would have a real big effect on the immersion factor Wow well we're just scratching the surface here both in how you as developer or experimenting with these dumb controllers and how your platform is allowing us to create these these virtual videos of virtual actors winces inside virtual environments super cool we're making cartoons that people have never been able to make before without VR without the game engine technology that we're using if you want to find out more you want to make some cartoons with us give us a shout you can email me at info food to VR comm or just go to Poovey are calm and there's information there typical will hey guys thanks for excellent guy and we will see you next time whoa wait a minute we're in a household world now something's wrong with my modelyou hey everybody its norm from test is Jeremy from tested today we have a special treat for you guys we have a whole sub here with the vive headset but we're not gonna be talking about the headset or these controller or even the pro audio strap we're your probably shop which is pretty good we have a special guest that's gonna bring a new piece of hardboard let's bring him right now it's foo BR Will Smith alright that's it I will confer well here because one of the reasons we want to learn more about food what you've been doing the past year and also oh my god is it that this is so 2016 as well it's this old technology Jeremy and put these down we don't talk about those any actually those are still really good we use them all the time barely but the function valve were kind enough to send us some duxelles prototypes to use for an upcoming project which i terribly can't talk about right now but we'll be talking about in the two and a half three weeks maybe around September 6th perhaps if you want to follow me on twitter at twitter.com/zeromd ZAR the new valve knuckles controller yes that they unveiled last year actually at a talk and then they started to show up in developers hands yeah what do you think I'm really impressed like they solved the problems that I had with the one like I loved oculus touch controls because a definite way to hold them they have the grip button and the trigger button separately the grip buttons actually usable on the touch controllers which is important and and like it's just it's just a better controller design than the lungs for what we do like we need a definite hand position like grip ergonomics it's one thing one of the reasons like you said I think the ocular touch controls are more ergonomic than the the wide controllers maybe less versatile and how you can hold them is because your resting position of your hand is basically that grip it's almost like the it's a neutral prepare a neutral poses right all right but the knuckle controller which isn't this is not HTC this is still valve experimentation yes let's you have a controller in your hand without gripping it isn't at all that's kind of the magic so yeah there's a neoprene there's a knee opening of sleeve that you kind of put your hand between that and the controller and then the other magic part about this is the controller the back of the surface of the controller is capacitive so that means it can detect when your fingers are on it with a great deal of sensitivity the same way that if you put your palm on your iPad screen your phone screen it'll be able to tell there's a bunch of fingers then so as far as the strap goes I know there's a tensioning that's just a tighten it until it's comfortable and you don't ever have to do that again I mean if you put on something with tiny hands maybe it's elastic though so yeah I usually go in and off at once and have it adjusted it that's interesting it's interesting because whenever now I'm not gripping at all it really is attention I feel most of that tension on the bottom part by my by my pinkie so you might need a little too tight you want it to be kind of even across the top of your knuckles if you want to ride the at least what I found is the best is to ride the strap across the top of your knuckles and you said they're capacitive sensors along my grip here along the inside edge here yeah which they in the software divided believe in the three segments for each through your fingers and and so it calibrates that each time you put it on so when you put those controllers in your hand you grip them and you release your grip any release in the gripping at least three or four times until the fingers start tracking correctly or close to it you first put them on you'll see gnarled fingers representations and they're all over the place and then it dials it in as you as you did that tap and release and it figures out which fingers are which so with the touch controller with the active session feel of these two fingers would never be tracted no no way shape or form right but they are analog because it is a button that you depress well this one would be a button right I'm right about it there's a little thinking you're right you're right the reason fingers not analog on oculus touch controllers so there's nothing it's just a rental finger is just you know that long yeah but post capacitive when you touch it and then this analog going forward oh it's with these three it's it's almost made a lot of the entire travel see I thought was just capacitive when you touch and then it moves in when you pull the trigger but oh no you can have them grab yeah so um but then also the top is a little different like there is button parity between this and your standard vibe controller you know there is a touchpad here it's a little bit different shape so the way the way the buttons work is the round one is the steam button the one that pops up the steam menu the one on the inside edge you know on your right hand it to your left left button on your left your right button is the like grip button I believe and then the one on the outside edge is the is the button that's above the pad it's like the one that I'm reviews right users weapon select whatever make it in but then the circle pad is totally redesigned so it it's the same more or less the same surface area but much deeper it's a much deeper curve than the it's a bowl rather than like a disc is what I would say on the traditional one mhm and of course these are prototypes the the final design may change they're not committing to anything yet they're just putting these out there I think one of the things that they learn from the initial long design is that how they think people are going to use them and and this is also new that how people use this stuff in the real world versus how you think it's you can't design for how people are going to use it because nobody knows how they're going to use it yet you have to try stuff I can say I like that this is now a unique button the actual home the steamvr happen because I would always forget is at the top of the bottom one in game and always always flip-flop on yeah there's much better button place when you get one additional button also of course there are plenty of the sensors around the controller so you're not going to get occlusion I like that it's rather something I mean you never really hit the back of your hand in VR it's mostly the poem and then and then there's of course the trigger which which is organ AMA canned as I do today is your stage trigger and it's capacitive as well and the touching on the front there buttons I believe are as well not sure about the face surface but I would expect that to be in the final speaking of someone who literally has a gash in my drywall about that big from a friend who was playing something and yeah out with the vibe controller across the room this is the killer it's order for this room like I knocked my watch off a while ago and I was shadowboxing yeah I know yeah I'm not the man fell off it's not great I'm like throwing rope not coming up I love how crazy you are with my one-of-a-kind prototype years old but you've been using and you can cemented this in a version of food so what we're going to do is want to put on some headsets and jump into a food environment you've created talk about what you've been working on for the past two years I guess ready to jump in yeah all right one two three whoa wait a minute we're a virtual world now something's wrong with my model oh wait your hair Jenner I look a little funny dude you look awesome okay you look so cool so will is on the far right I'm only wearing using wills model is in the middle and I'm here on the right with a pixelated tessellated version of me a low poly version this is this is food all right yeah this is well this is one of the things we've built so for the last year but we started out doing this VR recorded shows right shows that happened in be odds you can watch in VR turns out there's not a lot of headsets on the market we like making money is nice so we've started doing 2d animation we did a the first thing with adults waiver earlier this year in January whose some really exciting stuff to come up coming up in the next couple weeks so I can't really talk about now this is so weird or maybe not you make live cartoons that's the takeaway we're using the flu animation to make guys that look like me and look like Jeremy and looks like Carl from Aqua Teen Hunger Force and we can animate them live so put a performer in instead of having to do a bunch of 2d animation I think you lost an arm there Jeremy then we can we can go in and do things they're possible in traditional animation like live call-in shows or you know putting characters in places that you wouldn't expect to see them and all sorts of other interesting things like that so some of interest saying people should know because we're going to switch between this virtual camera on the live camera is that our relative space it like you are actually right there in the real world and Jeremy is actually right there on TV I yeah there you go I just touch Jeremy is that like part of the foo formula is is that what you're going for it could could it be done remotely well so you can do it remotely we've recorded some pilots where one of the performers was in LA and one of them was on the East Coast we if you want to do something like live on stage or in a live context like we're doing right now then then doing it where we're all in the same physical space and virtual space makes a lot of sense but there are complications because for example I get over close to Jeremy without his lips will start flapping or maybe not because you're hitting my microphone hitting his mic alrightlet's flapping is my other action going on it Jeremy's isn't suicide might be Laurie sure checking one two I don't see you moving right now but your eyes are tracking nicely but we do all the animation procedurally so that this is the interesting thing about what we built is all of the data that we're getting to animate these bodies it comes off of two hands and the head that you get from the vibe right we can add other things like we have some HCC tracker pucks that we can put on feet and back and stuff like that to get more data and do a little bit better job below the waist but for the most part we with machine learning and AI K and all sorts of terrifying that we can we can animate a believable human out of just three three count them three can I do it three points of data three so can we talk about that for a second yeah we are using the knuckles controllers so yeah in foo what was it like to actually incorporate this hardware into your game well valve set some controllers to us and then I went to FedEx and sent one of the controllers that he sent to us to my lead dev in New Mexico and then like two days later it was working so it takes two days no it was um so Andre Infante who's our lead dev spent a couple of hours working on it and got it up relatively quickly ago we have to change like you'll notice with my avatar especially there's some stuff that changes and norm norm is using the old ones as you can see Jeremy and I are both using knuckles and basically because there's a definite position that you hold the knuckles you know where it's where it's always in the same orientation on your hand then then we have to change the way the math works that defines what the angle on the pans is but other than that it's all more or less the same and this is a more consistent angle than you were getting from the original ones yeah with the world if you look how norms holding the ones or you don't put your hands on your back also for whatever you do map mask rise when you put your hands behind your back but yeah you could hold the ones in different ways you could hold them like a baseball bat you can hold them like a golf club you can hold them I eat a Harry Potter style magic wand and this there's only really one correct way to hold the controller's and that matters because you're doing what's called inverse kinematics in order to determine where all the body parts are yeah it's on the only three tracking point so we well I mean we've inverse kinematics we use some machine learning we use a bunch of other stuff we did a GDC talked about it there BRDC talked earlier this year that explains a little bit more about how it works but the upshot is that yeah the the the relationship between the position of your hand the orientation of your wrist the Orient have your elbow and shoulder is much easier to predict if you know how people are holding the controller interesting so with these knuckles controllers I can actually pull one finger out two fingers out I can make a big gesture again I already did that Wow I love you guys I don't really mean this so because the what you're getting is capacitive sensors I can repoint them on CapSense your middle finger your pinky a ring finger and your pinky how are you getting analog them yeah that's insane okay so the back of the controller is capacitive so that means that it can detect when your finger first hits it at the lowest knuckle to your palm as well as when the tip of your finger wraps all the way around and that means that you can infer the position of the hand you're not measuring directly that you're you know you that's why when you first put the controllers on or when you change your your when you wiggle them around a little bit you have to kind of grip in and out a few times yeah and you just keep doing that until the fingers look right I have to say the amount of granularity that you get about where fingers are is better than I expected it would be right like look you can nut without calibration on your end why I've open my hands at least three times and that's all it takes right yeah it constantly is calibrating and refining and inferring and a lot of that stuff happens on the driver level well what we've built here is our quickest dirtiest implementation you know it's literally Andre working on it for a few hours over a couple of days and there's definitely room for refinement there's more information we can infer from what we're getting from the controllers and frankly these models my model wasn't really built to be rigged I mean we put bones in the fingers so that we can animate them individually but but there are things that we would change if we were doing it from scratch right now Jeremy when you release the knuckle controller so you're not actively gripping what is your natural pose they put on your side is it like that okay and you're not holding on right now you know gripping at all nope you're just it looks pretty natural yeah pretty much I mean that's pretty much how I stand I would guess especially if I was in an Old West oh yeah yeah yeah so of the our characters are all bowlegged we do that so that we avoid can collision of the legs when they cross over in front of each other it's just it was it was an artistic choice kind of directed by necessities of the engine so what you've built into your tool will including like the models the your I K but also with the way we're filming this is a camera system yeah so that's the other part of this if you want to make 2d content you need a way to get to the content back out and the easiest way to do that is to give people a camera right so the version of the camera that Gunther is controlling right now is actually a drone camera at models a drone camera and he's flying it around within a month one controller and it works difficult off the Pam to control so it's left stick is up and down a pan left and right and then right stick is forward backwards left right straight right and then you can press a to switch from up and down on the left stick to tilt mmm I would help to do roll which is a discomforting Emily a zoom on this because you know it turns out there's no optics it's all virtual so we just changed that will be on the camera dynamically when you zoom in and out so obviously that gives you camera control tip a gamepad no tracking require but you also have the ability to track using a track to tour yeah so we also built a rig that lets us put like a vibe control or HCC tracker on just a shoulder-mounted rig and we can mount that so that a camera person can get in this space and and shoot directly up in the face you get handheld handheld tracking and all the stuff that comes with that we can motion smooth that out we could build a one of the people we've been talking to what's a dolly so that you can dolly across oh right there's something else we laid on tracks that's cool um I mean we wouldn't actually put the real tracks there we would place them in the world and plan out those shots and event better than real world photography exactly we've done things like put a green screen in so we did the carl thing the aquitine it's a pigskin puck nuptse prognostications from the pigskin wizard wy z ze ard if you want to check that out and you can see we basically built a green screen for him so they can host him on anything so like it would be a virtual avatar yeah instead of this cartoonish living room environment it's flat frame walls that's exactly right Gunther if you press G on your keyboard that might just pop up I'm not sure no did it it made the camera green made the camera I feel that's the other thing that's worth matching we could put as many cameras in as you want so we can do like a traditional we can shoot like a traditional 4 camera sitcom-style where we have four cameras you know main conversations secondary conversation wide two-thirds all that kind of stuff is possible but then you can actually do the post later right you can choose what angles you want later you don't have to do all that you can do that as well you can record an engine just like we do the are you ply the cameras around you can record off of that and post if you want you can also do like a live mix so one of the things that we've had is that you like one of the keys to making this work for people is fitting it into their existing content pipeline so so for example we let you hook up to say a TriCaster you hook up your capture computers whether it's one that's a 1080p machine or two that are 1080 machines or one that's a 4k machine that splits into four to any Pequod's leave a virtual camera in each of the quads and then they pipe into a normal video mixer they can handle it just like they would you know an episode of The Tonight Show or you're live 30 rocks or that's year you can whatever virtual performance capture pretty much with these versatile cars and now with the precision of individual finger movement well and deposit into any type of video and the benefit of doing the virtual performers in this context instead of say putting them in a mocap suit with the ping-pong balls and all that stuff is that here the performers can actually see the other performers as they're going to be you know if we scale somebody up or down they'll be the right size if they're picking up a prop they're interacting with something then they can you know use it in the space of a storm button a lot which button is if it's this one so you know we and we can do things like that are impossible in real world like right in space you can't grab it here I'm sorry um but you know you can you know rung up I just out board it again donation has left the building yeah I'm over here yeah I'm the man in the money this is my favorite thing to do when you put new people in VR by the way just tell them to draw a box yeah and watch them draw a 2d projection of a box right drop the chalk II I did so yeah that's pretty much it that's what we've done knuckles is another part of what we're doing I mean it obviously gives us a lot more fidelity in control over the animation lets us do things i mean we can programmatically pick this up I think this chalk looks okay you can grab it for my hands if you want Jeremy to squeeze the it's not not to him how do I draw you draw by squeeze pressing the left but the it's your right hand the left button there you go cool if you find that using knuckles it's more comfortable and you you're using it for longer periods and then you would be grabbing a holding you know one of these wand controller um it's uh I haven't noticed that the battery life on the knuckles prototypes is a little bit lower than the ones um but I mean it's nice to not have to worry about dropping the controller and not have to always happens in your hand and this way you can leave your hands-free and like once you get used to having your hand three you can make more natural gestures there's a fall somewhere in here I don't know where it went how do I remove my robot pick up the chalk again and then press the circle pad peacock has left the building I'll go get the truck oh okay I knocked the camera down oh I just rode up again I did I'm going to clip through you guys for a sec I got it early it wasn't real I mean the the stock that it looks seriously is like actually looking at the camera the tessellation sort of blends into a real photographic image and I feel like I'm watching television norm over there right I mean that's the thing is you get you dwell in that cameras far away so we're dealing with resolution limits but like but the human motion is there which is enough to create a sense of realism exactly right and what we're seeing in the headset which you guys can't see is a box that looks like a preview of it's a basically a preview what Gunther sees on his monitor so you know we're able to look at that and we can tell when the animations getting edgy when you do something nightmarish like put your hands behind you and like weird melty things happen or if you're clipping through your body or whatever you can correct that the performance can crack down on the fly which we very quickly learned was an important thing one of the things that you first demoed with foo was incorporating a fire watch game aesthetic into your engine is that something you're still doing is that a imagine glorious process yes I mean obviously we're definitely still doing that we're still we still have episodes of the Kickstarter to fulfill we're super late and I'm sorry it turns out we we didn't really anticipate how long it was going to take us to do things like track down horrible bugs and and things like that but we've recorded we have two more episodes of the foo show in the can that are ready to go one is features total war war or two and we go inside we get human scale with Warhammer dragons and wizard men and characters it's really awesome the other one is even more incredible we go and talk to a biochemist biophysical chemist who builds nano machines out of DNA named Sean Douglas and we go full powers of 10 we start out in his lab and we get smaller and get smaller and gets smaller and gets smaller until we're standing someplace that people have never been before so I think you'll be really excited by that and you get to a little bit better understanding of how and why you might want to build nano machines out of DNA there's an example where I would love to see eye tracking you know yeah it's not so much for Vogue elated rendering or cope we know words for my purposes it's for your purposes that I can actually feel like I'm with you even more well so we eye track now you see I'm watching you nor might be more interesting than you than me too but it's all simulated well it turns out simulated is good enough when you can't really look around inside the headset that much so we do we do some fairly complex math and we add we make different things in the environment different levels of interesting so that people their eyes behave naturally normal eyes are actually misbehaving right now because he's using an old version of the model I thought I was getting his eyelids flip up and so he blinks up and turns down which is very disconcerting oh I thought that was cool because it looked like he was expressing himself well so when he but he if he gets loud his eyebrows will go up I think yeah my exact yuge I like that you've cheated the system by identifying things in your world you know people are going to naturally gravitate toward of other people obviously but thinking of taking things like gestures and also volume well there's a lot of things that are interested that you have to mimic in order to make people believable and make people not squeezed out by them and we're much better off but better with it now when we're using say the 2d avatar that Jeremy and I are wearing then you're older tessellated one but things like saccade saccade are those little tiny infinitesimal movements of your eyes we're kind of always glancing around even if we're looking at somebody more eye contact and you have to mimic that or else people get squeezed out it's uncanny valley territory well they've all rubbery they're if you're what's mold watcher but but things like the the movements of the fingers you need to be able to move the fingers a little bit even if they're even if we're not tracking them with the controllers we do just a little bit of idle animation on the fingers so that you don't have straight paddle hands because that's something that again freaks people out yet if you talk and your eyebrows don't move that's really weird so we had a little bit of eyebrow movement that bounces with the cadence of your voice just basic stuff like that Oh tracking issues okay is that a problem is it tracking an issue we get the tracking issues are the problem we're having here the your lighthouses and the tested office are a little bit lower than I would normally go to put three people in the same space if you want to get three people in the same States you really want to shoot down on the on the play area just so that you don't have occlusion issues because if say your right hand which is between Norman the lighthouse then it'll it'll get blocked when it doesn't see either lighthouse it flies off to the origin but one of you better than ever clean my controller in the lighthouse exactly Oh Jeremy how you feeling after using the knuckle controller for a bit oh it's cool it's cool yeah I wish that I could see my hands as but that's just a matter of the software so the scene your hands is something we didn't do with the wands because we could get it to line up one two one and then normal I found when we did the the vikon motion tracking thing like three four years ago now yeah it was that was really discomforting when you can't when you're when you see your hands in VR and they don't line up with the real hand I think with the knuckles we're probably going to go back and retry it because it seems like it's a little bit of a better another opportunity to show real hands I am not about to drop them that's one thing yeah I think even not a time we've been in here I'm aware I'm holding something I'm never forgetting the fact that I'm holding these controllers and you are you more relaxed so your hand position more or less because you know that actively grow you know I'm so I'm so used to having Digger controllers that half the time I probably have been gripping in yeah it takes a while to break yourself of that have exact route but uh you know it's nice to be able to let go and I can imagine playing a game or if I'm going to grab something and physically actually grabbing it rather than just pushing a trigger would have a real big effect on the immersion factor Wow well we're just scratching the surface here both in how you as developer or experimenting with these dumb controllers and how your platform is allowing us to create these these virtual videos of virtual actors winces inside virtual environments super cool we're making cartoons that people have never been able to make before without VR without the game engine technology that we're using if you want to find out more you want to make some cartoons with us give us a shout you can email me at info food to VR comm or just go to Poovey are calm and there's information there typical will hey guys thanks for excellent guy and we will see you next time whoa wait a minute we're in a household world now something's wrong with my modelyou hey everybody its norm from test is Jeremy from tested today we have a special treat for you guys we have a whole sub here with the vive headset but we're not gonna be talking about the headset or these controller or even the pro audio strap we're your probably shop which is pretty good we have a special guest that's gonna bring a new piece of hardboard let's bring him right now it's foo BR Will Smith alright that's it I will confer well here because one of the reasons we want to learn more about food what you've been doing the past year and also oh my god is it that this is so 2016 as well it's this old technology Jeremy and put these down we don't talk about those any actually those are still really good we use them all the time barely but the function valve were kind enough to send us some duxelles prototypes to use for an upcoming project which i terribly can't talk about right now but we'll be talking about in the two and a half three weeks maybe around September 6th perhaps if you want to follow me on twitter at twitter.com/zeromd ZAR the new valve knuckles controller yes that they unveiled last year actually at a talk and then they started to show up in developers hands yeah what do you think I'm really impressed like they solved the problems that I had with the one like I loved oculus touch controls because a definite way to hold them they have the grip button and the trigger button separately the grip buttons actually usable on the touch controllers which is important and and like it's just it's just a better controller design than the lungs for what we do like we need a definite hand position like grip ergonomics it's one thing one of the reasons like you said I think the ocular touch controls are more ergonomic than the the wide controllers maybe less versatile and how you can hold them is because your resting position of your hand is basically that grip it's almost like the it's a neutral prepare a neutral poses right all right but the knuckle controller which isn't this is not HTC this is still valve experimentation yes let's you have a controller in your hand without gripping it isn't at all that's kind of the magic so yeah there's a neoprene there's a knee opening of sleeve that you kind of put your hand between that and the controller and then the other magic part about this is the controller the back of the surface of the controller is capacitive so that means it can detect when your fingers are on it with a great deal of sensitivity the same way that if you put your palm on your iPad screen your phone screen it'll be able to tell there's a bunch of fingers then so as far as the strap goes I know there's a tensioning that's just a tighten it until it's comfortable and you don't ever have to do that again I mean if you put on something with tiny hands maybe it's elastic though so yeah I usually go in and off at once and have it adjusted it that's interesting it's interesting because whenever now I'm not gripping at all it really is attention I feel most of that tension on the bottom part by my by my pinkie so you might need a little too tight you want it to be kind of even across the top of your knuckles if you want to ride the at least what I found is the best is to ride the strap across the top of your knuckles and you said they're capacitive sensors along my grip here along the inside edge here yeah which they in the software divided believe in the three segments for each through your fingers and and so it calibrates that each time you put it on so when you put those controllers in your hand you grip them and you release your grip any release in the gripping at least three or four times until the fingers start tracking correctly or close to it you first put them on you'll see gnarled fingers representations and they're all over the place and then it dials it in as you as you did that tap and release and it figures out which fingers are which so with the touch controller with the active session feel of these two fingers would never be tracted no no way shape or form right but they are analog because it is a button that you depress well this one would be a button right I'm right about it there's a little thinking you're right you're right the reason fingers not analog on oculus touch controllers so there's nothing it's just a rental finger is just you know that long yeah but post capacitive when you touch it and then this analog going forward oh it's with these three it's it's almost made a lot of the entire travel see I thought was just capacitive when you touch and then it moves in when you pull the trigger but oh no you can have them grab yeah so um but then also the top is a little different like there is button parity between this and your standard vibe controller you know there is a touchpad here it's a little bit different shape so the way the way the buttons work is the round one is the steam button the one that pops up the steam menu the one on the inside edge you know on your right hand it to your left left button on your left your right button is the like grip button I believe and then the one on the outside edge is the is the button that's above the pad it's like the one that I'm reviews right users weapon select whatever make it in but then the circle pad is totally redesigned so it it's the same more or less the same surface area but much deeper it's a much deeper curve than the it's a bowl rather than like a disc is what I would say on the traditional one mhm and of course these are prototypes the the final design may change they're not committing to anything yet they're just putting these out there I think one of the things that they learn from the initial long design is that how they think people are going to use them and and this is also new that how people use this stuff in the real world versus how you think it's you can't design for how people are going to use it because nobody knows how they're going to use it yet you have to try stuff I can say I like that this is now a unique button the actual home the steamvr happen because I would always forget is at the top of the bottom one in game and always always flip-flop on yeah there's much better button place when you get one additional button also of course there are plenty of the sensors around the controller so you're not going to get occlusion I like that it's rather something I mean you never really hit the back of your hand in VR it's mostly the poem and then and then there's of course the trigger which which is organ AMA canned as I do today is your stage trigger and it's capacitive as well and the touching on the front there buttons I believe are as well not sure about the face surface but I would expect that to be in the final speaking of someone who literally has a gash in my drywall about that big from a friend who was playing something and yeah out with the vibe controller across the room this is the killer it's order for this room like I knocked my watch off a while ago and I was shadowboxing yeah I know yeah I'm not the man fell off it's not great I'm like throwing rope not coming up I love how crazy you are with my one-of-a-kind prototype years old but you've been using and you can cemented this in a version of food so what we're going to do is want to put on some headsets and jump into a food environment you've created talk about what you've been working on for the past two years I guess ready to jump in yeah all right one two three whoa wait a minute we're a virtual world now something's wrong with my model oh wait your hair Jenner I look a little funny dude you look awesome okay you look so cool so will is on the far right I'm only wearing using wills model is in the middle and I'm here on the right with a pixelated tessellated version of me a low poly version this is this is food all right yeah this is well this is one of the things we've built so for the last year but we started out doing this VR recorded shows right shows that happened in be odds you can watch in VR turns out there's not a lot of headsets on the market we like making money is nice so we've started doing 2d animation we did a the first thing with adults waiver earlier this year in January whose some really exciting stuff to come up coming up in the next couple weeks so I can't really talk about now this is so weird or maybe not you make live cartoons that's the takeaway we're using the flu animation to make guys that look like me and look like Jeremy and looks like Carl from Aqua Teen Hunger Force and we can animate them live so put a performer in instead of having to do a bunch of 2d animation I think you lost an arm there Jeremy then we can we can go in and do things they're possible in traditional animation like live call-in shows or you know putting characters in places that you wouldn't expect to see them and all sorts of other interesting things like that so some of interest saying people should know because we're going to switch between this virtual camera on the live camera is that our relative space it like you are actually right there in the real world and Jeremy is actually right there on TV I yeah there you go I just touch Jeremy is that like part of the foo formula is is that what you're going for it could could it be done remotely well so you can do it remotely we've recorded some pilots where one of the performers was in LA and one of them was on the East Coast we if you want to do something like live on stage or in a live context like we're doing right now then then doing it where we're all in the same physical space and virtual space makes a lot of sense but there are complications because for example I get over close to Jeremy without his lips will start flapping or maybe not because you're hitting my microphone hitting his mic alrightlet's flapping is my other action going on it Jeremy's isn't suicide might be Laurie sure checking one two I don't see you moving right now but your eyes are tracking nicely but we do all the animation procedurally so that this is the interesting thing about what we built is all of the data that we're getting to animate these bodies it comes off of two hands and the head that you get from the vibe right we can add other things like we have some HCC tracker pucks that we can put on feet and back and stuff like that to get more data and do a little bit better job below the waist but for the most part we with machine learning and AI K and all sorts of terrifying that we can we can animate a believable human out of just three three count them three can I do it three points of data three so can we talk about that for a second yeah we are using the knuckles controllers so yeah in foo what was it like to actually incorporate this hardware into your game well valve set some controllers to us and then I went to FedEx and sent one of the controllers that he sent to us to my lead dev in New Mexico and then like two days later it was working so it takes two days no it was um so Andre Infante who's our lead dev spent a couple of hours working on it and got it up relatively quickly ago we have to change like you'll notice with my avatar especially there's some stuff that changes and norm norm is using the old ones as you can see Jeremy and I are both using knuckles and basically because there's a definite position that you hold the knuckles you know where it's where it's always in the same orientation on your hand then then we have to change the way the math works that defines what the angle on the pans is but other than that it's all more or less the same and this is a more consistent angle than you were getting from the original ones yeah with the world if you look how norms holding the ones or you don't put your hands on your back also for whatever you do map mask rise when you put your hands behind your back but yeah you could hold the ones in different ways you could hold them like a baseball bat you can hold them like a golf club you can hold them I eat a Harry Potter style magic wand and this there's only really one correct way to hold the controller's and that matters because you're doing what's called inverse kinematics in order to determine where all the body parts are yeah it's on the only three tracking point so we well I mean we've inverse kinematics we use some machine learning we use a bunch of other stuff we did a GDC talked about it there BRDC talked earlier this year that explains a little bit more about how it works but the upshot is that yeah the the the relationship between the position of your hand the orientation of your wrist the Orient have your elbow and shoulder is much easier to predict if you know how people are holding the controller interesting so with these knuckles controllers I can actually pull one finger out two fingers out I can make a big gesture again I already did that Wow I love you guys I don't really mean this so because the what you're getting is capacitive sensors I can repoint them on CapSense your middle finger your pinky a ring finger and your pinky how are you getting analog them yeah that's insane okay so the back of the controller is capacitive so that means that it can detect when your finger first hits it at the lowest knuckle to your palm as well as when the tip of your finger wraps all the way around and that means that you can infer the position of the hand you're not measuring directly that you're you know you that's why when you first put the controllers on or when you change your your when you wiggle them around a little bit you have to kind of grip in and out a few times yeah and you just keep doing that until the fingers look right I have to say the amount of granularity that you get about where fingers are is better than I expected it would be right like look you can nut without calibration on your end why I've open my hands at least three times and that's all it takes right yeah it constantly is calibrating and refining and inferring and a lot of that stuff happens on the driver level well what we've built here is our quickest dirtiest implementation you know it's literally Andre working on it for a few hours over a couple of days and there's definitely room for refinement there's more information we can infer from what we're getting from the controllers and frankly these models my model wasn't really built to be rigged I mean we put bones in the fingers so that we can animate them individually but but there are things that we would change if we were doing it from scratch right now Jeremy when you release the knuckle controller so you're not actively gripping what is your natural pose they put on your side is it like that okay and you're not holding on right now you know gripping at all nope you're just it looks pretty natural yeah pretty much I mean that's pretty much how I stand I would guess especially if I was in an Old West oh yeah yeah yeah so of the our characters are all bowlegged we do that so that we avoid can collision of the legs when they cross over in front of each other it's just it was it was an artistic choice kind of directed by necessities of the engine so what you've built into your tool will including like the models the your I K but also with the way we're filming this is a camera system yeah so that's the other part of this if you want to make 2d content you need a way to get to the content back out and the easiest way to do that is to give people a camera right so the version of the camera that Gunther is controlling right now is actually a drone camera at models a drone camera and he's flying it around within a month one controller and it works difficult off the Pam to control so it's left stick is up and down a pan left and right and then right stick is forward backwards left right straight right and then you can press a to switch from up and down on the left stick to tilt mmm I would help to do roll which is a discomforting Emily a zoom on this because you know it turns out there's no optics it's all virtual so we just changed that will be on the camera dynamically when you zoom in and out so obviously that gives you camera control tip a gamepad no tracking require but you also have the ability to track using a track to tour yeah so we also built a rig that lets us put like a vibe control or HCC tracker on just a shoulder-mounted rig and we can mount that so that a camera person can get in this space and and shoot directly up in the face you get handheld handheld tracking and all the stuff that comes with that we can motion smooth that out we could build a one of the people we've been talking to what's a dolly so that you can dolly across oh right there's something else we laid on tracks that's cool um I mean we wouldn't actually put the real tracks there we would place them in the world and plan out those shots and event better than real world photography exactly we've done things like put a green screen in so we did the carl thing the aquitine it's a pigskin puck nuptse prognostications from the pigskin wizard wy z ze ard if you want to check that out and you can see we basically built a green screen for him so they can host him on anything so like it would be a virtual avatar yeah instead of this cartoonish living room environment it's flat frame walls that's exactly right Gunther if you press G on your keyboard that might just pop up I'm not sure no did it it made the camera green made the camera I feel that's the other thing that's worth matching we could put as many cameras in as you want so we can do like a traditional we can shoot like a traditional 4 camera sitcom-style where we have four cameras you know main conversations secondary conversation wide two-thirds all that kind of stuff is possible but then you can actually do the post later right you can choose what angles you want later you don't have to do all that you can do that as well you can record an engine just like we do the are you ply the cameras around you can record off of that and post if you want you can also do like a live mix so one of the things that we've had is that you like one of the keys to making this work for people is fitting it into their existing content pipeline so so for example we let you hook up to say a TriCaster you hook up your capture computers whether it's one that's a 1080p machine or two that are 1080 machines or one that's a 4k machine that splits into four to any Pequod's leave a virtual camera in each of the quads and then they pipe into a normal video mixer they can handle it just like they would you know an episode of The Tonight Show or you're live 30 rocks or that's year you can whatever virtual performance capture pretty much with these versatile cars and now with the precision of individual finger movement well and deposit into any type of video and the benefit of doing the virtual performers in this context instead of say putting them in a mocap suit with the ping-pong balls and all that stuff is that here the performers can actually see the other performers as they're going to be you know if we scale somebody up or down they'll be the right size if they're picking up a prop they're interacting with something then they can you know use it in the space of a storm button a lot which button is if it's this one so you know we and we can do things like that are impossible in real world like right in space you can't grab it here I'm sorry um but you know you can you know rung up I just out board it again donation has left the building yeah I'm over here yeah I'm the man in the money this is my favorite thing to do when you put new people in VR by the way just tell them to draw a box yeah and watch them draw a 2d projection of a box right drop the chalk II I did so yeah that's pretty much it that's what we've done knuckles is another part of what we're doing I mean it obviously gives us a lot more fidelity in control over the animation lets us do things i mean we can programmatically pick this up I think this chalk looks okay you can grab it for my hands if you want Jeremy to squeeze the it's not not to him how do I draw you draw by squeeze pressing the left but the it's your right hand the left button there you go cool if you find that using knuckles it's more comfortable and you you're using it for longer periods and then you would be grabbing a holding you know one of these wand controller um it's uh I haven't noticed that the battery life on the knuckles prototypes is a little bit lower than the ones um but I mean it's nice to not have to worry about dropping the controller and not have to always happens in your hand and this way you can leave your hands-free and like once you get used to having your hand three you can make more natural gestures there's a fall somewhere in here I don't know where it went how do I remove my robot pick up the chalk again and then press the circle pad peacock has left the building I'll go get the truck oh okay I knocked the camera down oh I just rode up again I did I'm going to clip through you guys for a sec I got it early it wasn't real I mean the the stock that it looks seriously is like actually looking at the camera the tessellation sort of blends into a real photographic image and I feel like I'm watching television norm over there right I mean that's the thing is you get you dwell in that cameras far away so we're dealing with resolution limits but like but the human motion is there which is enough to create a sense of realism exactly right and what we're seeing in the headset which you guys can't see is a box that looks like a preview of it's a basically a preview what Gunther sees on his monitor so you know we're able to look at that and we can tell when the animations getting edgy when you do something nightmarish like put your hands behind you and like weird melty things happen or if you're clipping through your body or whatever you can correct that the performance can crack down on the fly which we very quickly learned was an important thing one of the things that you first demoed with foo was incorporating a fire watch game aesthetic into your engine is that something you're still doing is that a imagine glorious process yes I mean obviously we're definitely still doing that we're still we still have episodes of the Kickstarter to fulfill we're super late and I'm sorry it turns out we we didn't really anticipate how long it was going to take us to do things like track down horrible bugs and and things like that but we've recorded we have two more episodes of the foo show in the can that are ready to go one is features total war war or two and we go inside we get human scale with Warhammer dragons and wizard men and characters it's really awesome the other one is even more incredible we go and talk to a biochemist biophysical chemist who builds nano machines out of DNA named Sean Douglas and we go full powers of 10 we start out in his lab and we get smaller and get smaller and gets smaller and gets smaller until we're standing someplace that people have never been before so I think you'll be really excited by that and you get to a little bit better understanding of how and why you might want to build nano machines out of DNA there's an example where I would love to see eye tracking you know yeah it's not so much for Vogue elated rendering or cope we know words for my purposes it's for your purposes that I can actually feel like I'm with you even more well so we eye track now you see I'm watching you nor might be more interesting than you than me too but it's all simulated well it turns out simulated is good enough when you can't really look around inside the headset that much so we do we do some fairly complex math and we add we make different things in the environment different levels of interesting so that people their eyes behave naturally normal eyes are actually misbehaving right now because he's using an old version of the model I thought I was getting his eyelids flip up and so he blinks up and turns down which is very disconcerting oh I thought that was cool because it looked like he was expressing himself well so when he but he if he gets loud his eyebrows will go up I think yeah my exact yuge I like that you've cheated the system by identifying things in your world you know people are going to naturally gravitate toward of other people obviously but thinking of taking things like gestures and also volume well there's a lot of things that are interested that you have to mimic in order to make people believable and make people not squeezed out by them and we're much better off but better with it now when we're using say the 2d avatar that Jeremy and I are wearing then you're older tessellated one but things like saccade saccade are those little tiny infinitesimal movements of your eyes we're kind of always glancing around even if we're looking at somebody more eye contact and you have to mimic that or else people get squeezed out it's uncanny valley territory well they've all rubbery they're if you're what's mold watcher but but things like the the movements of the fingers you need to be able to move the fingers a little bit even if they're even if we're not tracking them with the controllers we do just a little bit of idle animation on the fingers so that you don't have straight paddle hands because that's something that again freaks people out yet if you talk and your eyebrows don't move that's really weird so we had a little bit of eyebrow movement that bounces with the cadence of your voice just basic stuff like that Oh tracking issues okay is that a problem is it tracking an issue we get the tracking issues are the problem we're having here the your lighthouses and the tested office are a little bit lower than I would normally go to put three people in the same space if you want to get three people in the same States you really want to shoot down on the on the play area just so that you don't have occlusion issues because if say your right hand which is between Norman the lighthouse then it'll it'll get blocked when it doesn't see either lighthouse it flies off to the origin but one of you better than ever clean my controller in the lighthouse exactly Oh Jeremy how you feeling after using the knuckle controller for a bit oh it's cool it's cool yeah I wish that I could see my hands as but that's just a matter of the software so the scene your hands is something we didn't do with the wands because we could get it to line up one two one and then normal I found when we did the the vikon motion tracking thing like three four years ago now yeah it was that was really discomforting when you can't when you're when you see your hands in VR and they don't line up with the real hand I think with the knuckles we're probably going to go back and retry it because it seems like it's a little bit of a better another opportunity to show real hands I am not about to drop them that's one thing yeah I think even not a time we've been in here I'm aware I'm holding something I'm never forgetting the fact that I'm holding these controllers and you are you more relaxed so your hand position more or less because you know that actively grow you know I'm so I'm so used to having Digger controllers that half the time I probably have been gripping in yeah it takes a while to break yourself of that have exact route but uh you know it's nice to be able to let go and I can imagine playing a game or if I'm going to grab something and physically actually grabbing it rather than just pushing a trigger would have a real big effect on the immersion factor Wow well we're just scratching the surface here both in how you as developer or experimenting with these dumb controllers and how your platform is allowing us to create these these virtual videos of virtual actors winces inside virtual environments super cool we're making cartoons that people have never been able to make before without VR without the game engine technology that we're using if you want to find out more you want to make some cartoons with us give us a shout you can email me at info food to VR comm or just go to Poovey are calm and there's information there typical will hey guys thanks for excellent guy and we will see you next time whoa wait a minute we're in a household world now something's wrong with my model\n"