Video Game & Complex Bokeh Blurs - Computerphile

"WEBVTTKind: captionsLanguage: enAll right, so we talked about separableFilters and the nice thing about separable filters is that they're incredibly quickAnd this means that for example in computer games you could actually do some of this on the graphics cardAt 60 frames a second, right? These are the kind of things that people who are writing computer gamesNot me people who know what they're doingwell, I think if you the games are thinking about right, how can we do post processing like motion blur andBloom and other kinds of filters that look good in a game, but don't drive the whole thing to a haltSeparable filters are a great way of doing thisThe problem is that Gaussian blur which is the thingWe kind of focused onDoesn't look that good as a blur right if you say well I want to do some depth of fieldI want to do some optical depth of fieldAll rightSo I want the foreground person in my game to be nice and sharp and I want the background to be lovely blur, right?It will look a bit odd. If you use a Gaussian because that's not what a lens doesYou need to use a proper bokeh blur. I don't know how you pronounce that where bouquet is right bouquet right bokeh bouquetOh, yeah, I lost interest. It's because of the way that the optics of a camera work, right? So aGaussian your point is spread over a sort of Gaussian function. So it decreases like this, right?That's not what happens in an optical blur my lens blurSo if we remember back a while ago, we did a light-filled videoYou've got a point light source here, which is something in your scene a person a light. They've tree not important, right?It's gonna be quite a few of these if they if your image has more than one pixel in itYou have a lens right which looks like thisit might have other glass optical elements and you have an imaging plane here like this andWhat happens is point light don't just travel directly into your camera. Everything just emits light in lots of directions depending on its propertiesso it comes in like this and when it's in focusIt will come and it will hit a point on here and that pixel will obtain that colorthe color will be the sum of all of these light rays and then it sort of simplified to be usually between nought and255 if you're out of focusWhat will happen is these light rays will come pass the camera like this and focus on a point behind itAnd actually you'll have this light will now be spread over this whole areaFrom the front if this is the front of our square sensor every point will now be a circle like thisand each other point will also be a circle and another circle in another circle and this gets you that kind ofCharacteristic Lens Blur that you see when something like light sources christmas tree lights or something are out of focusCut to a nice shot of this so people spend a lot of money on lensesPrimarily, I think because of the quality of this kind of blur. They're not perfect circles all the timeSometimes there's slightly odd all their hexagons because you've got an aperture and things like thisThey give this blurry sort of unique characteristic even down to a lens which you can't really get on a computerBut we can have a go because we've got to do it at 60 frames a secondYou haven't got the option of thinking about Zeiss and like although this is most obvious when you're looking at point light actually this appliesTo any blur right? If you do a Gaussian blow of a scene, that doesn't have very bright bitsIt will still look a bit different than if you have a normal bouquet blurRight, and and actually you look much better. If you do it properly. Can we recreate ourSeparable filters, but now perform a lens blur or at least an approximation to a lens blurAnd the answer is not it's not so easy, but you can kind of have a goYou have to start looking at imaginary numbers a little bitBut if we just sort of not worry about it too much you essentially replace your separableConvolutions with two complex separable convolutions and we can start to start to get something goingSo one particular game engine that does this is EA's Frostbite engine which drives a lot of its sports gamesSo things like Madden will have a nice blur on certain shots that's created by an artificial lens blurNot a Gaussian blur other games might kind of fake it a bit of a GaussianI don't know Primavera folks. It's not technically correct. There's a kind of short paper on thisThere's a few blogs online that talk about stepper lens blur. There's some implementations out thereI've got my own implementation that I wrote which I'll release at the end of this videoThere was a nice presentation at the Game Developers Conference in 2017 showing office technology, right? So, this is real-timeBouquet blur not Gaussian blur right? So how does it work?wellthe idea is that what we want to do is instead of having our kernel which is sort of like this and aGaussian we need to have it be a circle right in an ideal world that rightWhich from the top looks like a nice cut clear-cut circle now you can do thatBut you can't separate ityou can have akernelWhich is jazz a load of zeros and then a circle of ones in the middle and that will produce you a very nice filterBut it will take ages because you have to do all this point wise and it's not separableSo what we need to do is find a function that we can split into a vertical pass and a horizontal passAll right, let's look at example, right one of the nice things about Gaussian is but it's separable right? It's a naturally separable function. SoHow can we convert a Gaussian to something that kind of approximates this circular shape?And the answer is that we can sort of add in an imaginary component and create sort of complex GaussianNow obviously we're not gonna go into huge amount of detail on complex numbersBut they have an imaginary and a real component. The imaginary component is multiplied by I which is a square root of minus 1the good news for us is that most of the time when we actually need to use it the eyes get multiplied out and they'da sort disappear or become a minus numberAll rightThat's all buddy time gonna go into actually it's a bit like when we talked about our discrete cosineTransform and when maybe some time we'll talk about Fourier transformsWe have waves that add together to create something the shape we want in this caseThe wave is going to be a Gaussian, right? But it's going to go up and down like thisSo instead of our Gaussian going like this, which is not quite centered what we're going to roll on itWe have one that goes up and down right in the complex the complexComponent of this Gaussian is the bit that makes it do thatSo we have a Gaussian that sort of goes down like this and sort of centers around here and it comes down like thisAll right. It's symmetricalAnd then we have another one but perhaps come sort of like thisand when you add them together youget the thing we want so these kind of cancel out and you sort of get some of looks a bit like this with aBit of wobbling around and then it comes up about here and then kind of goes like that and then down like thatRight and that is our circle and that is also separable because it's essentially got a Gaussian as it's sort of primary functionSo I've coded this up and I've been using it to mess around with different blursMost of the code is to do with creating the kernelsI do some normalization and things like this then I pass the image from each component and the very endI combine them Mayon. I'll draw a little diagram to show how that worksthe more components you use a bit like inJPEG or any kind of sort of adding of waves the more ways you use the better approximation. You're going to get right?So if you're not watching it for a game engine, you can use maybe two each time you do thisYou've got quite a few convolutions you can have to do these are complex multiplication operations in additionIt takes time, even on a graphics card, right and bear in mind. There's other game things to do, right? The game isn't staticwith just this nice blurSo you've got other stuff to be worrying about fact that there's an imaginary number involved in some sense to meAnyway, personally, I like to look at it. It's basically not making any difference at allThe only difference is but the addition and multiplication that you would normally do in a convolution is now a complex addition and a complexMultiplication which are sort of well defined structures that you can just apply and this is a mathematical term not just as saying it's differentIt's absolutely a mathematical term right and it's actually not very complicated. You have your image, right?We have our first component C naught here and we have another component so you want we could have lots of errsthis has aComplex kernel so that we'll have a real and imaginary right but they're just going to look a bit like gaussians like I showed youThis is going to be the same over here. These are different onesAnd the idea is that when we add these two together, we get a very nice diskso what we're going to do is we're going toconverge with these and we're going to converse in which with these and we're going to add them together at the end because I seeA lot of addition goes on. All rightso we take our image we go through here and we get another image out, which is ourReal and another image out which is our imaginary we go through here we get an image out. That's real and image outthat's an imaginary andThen we add these two togetherin a weighted sum and then we take our fine or weighted images for each component and we add them together in ourFinal output like that. The reason we have to do these separately is so that we can make it separable, right?This is this can be done separatelyThis can be done in the same way very very quickly, right if you're doing it for just two componentsThere's a few convolutions you have to do add it all together and you have your result the SEPA bility means that instead of doingall the kernel for every locationwe do a vertical pass of a strip and we take that output and put it through a horizontal pass andThen we combine the outputs at the end and that's that's it mathematically equivalent to doing the 2d version but much much fasterSo let's have a look at some outputs right nowI've written some code here very straightforward that essentially produces these complex kernels in one dimension and kind of higher than 10 imageYou know in the correct order add them up at the end so we can blur smooches, you knowI'll release the code so you can have a look the original post that led me to this workOutlined a few parameters for these kernels that butt looks good, right they heat optimized so I put them in as wellSo let's have a look a couple of these kernels and see what they are. So if I just look at the zeroThe one component pass, right?So this is if you want to try and create a disc with just one of these complex kernels, like what would it look like?Well, let's have a look. These are the real and imaginary parts of this particular componentAnd this is going to look roughly like a disc when we add these two together. Let's see. So if I take the two togetherWe can run it and it's kind of a doughnut he disc right?And the reason is because you can't get a perfect disc with just a real and imaginary part added togetherIt's not going to work. So what we can do is we can take another kernel add that to itand now we've got kind of two wheels and two imaginaries in some loose sense add these all together and maybe we can start toApproximate this disc so we'll get a little bit better if I change thisAnd run that togetherAll right. So now it's got a bit of a ripple to itBut this is starting to look quite a lot better. Like this looks to me quite a lot like the discI'm hoping for and this is still separable so I can take a strip from the middle with a strip here andI can run it in two passes and save a huge amount of timebecause this is a sort of a this is I think a64 or 65 by 65 pixel kernel, this will be a big deal if you wanted to run this over an imageEspecially, you know a sort of resolution of modern games run out so we can get even better we can add more and more componentsSo if I've ramped up the number of components like we get somebody looks really really quite good. There we goso this now looks I think quite a lot like a disc so essentiallyThis is equivalent to running a circle of ones in a convolution over an imageIt's just that now we can do it in these separable components and save ourselves a huge amount of time. It's not perfect, right?The size is difficult to manage. The edge is not as sharp as I'd like this should be a sharp edgeBut you can't get an absolutely sharp edge, but for a sort of game engine something in the background, it's gonna look greatAnd it's going to run really really quick in the EA and in the Frostbite engineThey'll only use two components so it won't be a perfectColonel but it will look pretty good stars are a classic example of some where we can use this blurBecause of course their point lights have a nice spread out nicelySo this is what happens if we Gaussian blur this image, it looks dull and light just like not high-resolution and doesn't look greatright, if we if we look at the lens version, I meanI think it's aesthetically better, right?When it looks like this, right each of these points has spread out into a nice discI've had to do some exposure here because it's that's difficult to do but I think it looks pretty goodSo that's the Gaussian and that is the lens blur. I think that looks a lot betterso this is our fake lens blur by using ourOur circular disc kernel with lots of components because I wasn't worried about speed right?Even if you did it with two components, we still look goodSo if you were running this in a gameWhat you would do is you would start to separate out the foreground and the background of a scene you'd blurAnything you wanted to not be in focus and keep anythingYou did want to be in focus sharp, and then you'd start to bring them all together at the end by you knowAlpha blending or overlayingso you sent me a picture ofSome Lego so we've got a lego man here on a table with some lights in a jar behind, right?so this is quite a good example of something we could try this out on so you alsoThankfully gave me some foreground and backgroundMasks for this so I blurred the background and I blended in the foreground as you might do in a gameBut love and encode it. I kind of hacked it together in paint, but I think it looks really goodSo so this is our Gaussian attempt. So here I've essentially taken the man awayI've blurred at this using a Gaussian and I put the man back in basically and he looks okayBut if we flip to the lens blur, I think it looks a lot better, right?It looks a lot more like a true sort of bouquet blur. You might expect from a lensand so this is exactly what you'll see when you play FIFA and when you playPGA Tour and when you play Madden and any of these sports titles when it cuts to a kind of nice depth of field shotLet let's say means instant replay. This will be at work behind the scenes. There's loads of different tricksIt's just one of them. I think this is really interesting because apart from the fact that I think this is quite mathematically elegant, right?Not everyone will agree. IJust think it's cooler in these games and in you knowThey're doing all these little tips and tricks to make it run at 60 framesbut still look really good and you don't notice like you know what no one really knows until you read up on it thatMadden has a different depth of field approach to some other games, but maybe you just liked that a little bit betterYou didn't know and that's one of the things you liked about that gameAnd I just think that's really cool that you knowThese things are going on behind the scenes so we don't know they exist but they just make it a little bit betterWhat you think right? So in this case, it's going through glass. The glass is moving things around thatIs going to be lost a little bit in our in our blurAnother thing is about our disk is not absolutely sharp. Right and we can't approximate hexagons with itSo you can't do that kind of blurBut I think the main problem is that the exposure is very hard to get\n"