The Nature of Ray Tracing: A Discussion with Wendell and Brian

The challenges of ray tracing in game development are multifaceted, and one of the most significant issues is the way it interacts with the camera frustrum. The camera frustrum refers to the space that the camera sees, including all objects within its field of view. However, when using ray tracing, the algorithm is looking for something that may not be there, causing it to bounce off screen or hit an object behind another object, resulting in unnecessary visual information.

This problem is a result of the way rasterization works in general. Rasterization involves modifying textures and objects to fit the camera's perspective, essentially creating a second scene with distorted textures and objects that are being viewed from different angles. If you were to set another camera in the scene that kept all of these distortions intact, it would look incredibly chaotic and wouldn't make sense.

To mitigate this issue, developers often resort to techniques such as rendering at lower resolutions or using secondary cameras to create a separate scene with simplified textures and objects. This process is not always easy to explain, and once you start looking behind the scenes, the mystery of game development dissolves, revealing the complexity and intricacy of the work that goes into creating visually stunning games.

The Valve Corporation's recent foray into ray tracing highlights the ongoing challenges of this technology in game development. The use of new hardware and software tools is allowing developers to push the boundaries of what is possible with ray tracing, but it also presents a number of challenges. As Wendell noted during our conversation, having technical expertise on hand can be incredibly valuable in understanding the inner workings of game development.

Wendell's background in pixel art and his experience working with game development tools made him an ideal guest for this discussion. He was open about the self-interest that drives many developers to learn about ray tracing and other advanced technologies. For Wendell, understanding how these technologies work is essential for making informed decisions when it comes to developing games.

One of the key takeaways from our conversation was the importance of having a basic understanding of the technical aspects of game development. While many developers may not have the time or interest in delving deep into the details, having a general understanding can make a significant difference. Wendell's experience working with different tools and technologies has given him a unique perspective on the challenges of game development.

As we discussed the complexities of ray tracing, it became clear that this technology is not just about rendering beautiful images; it's also about making informed decisions when it comes to performance optimization. By understanding how these technologies work, developers can make targeted optimizations that result in faster render times and improved visual quality.

The conversation with Wendell was a fascinating exploration of the challenges and complexities of ray tracing in game development. From the technical aspects of rasterization to the creative choices involved in rendering secondary scenes, this discussion revealed the intricacies of game development and the importance of having technical expertise on hand.

Wendell's work on pixel art collabs is an area that holds great promise for future collaborations. His experience with different tools and technologies has given him a unique perspective on the challenges of game development, and his passion for creating beautiful images is evident in all he does.

Finally, as we concluded our conversation, Wendell mentioned his interest in exploring new projects and technologies in the coming year. With the release of new hardware and software tools, there are always new opportunities to push the boundaries of what is possible with ray tracing and other advanced technologies. We look forward to seeing what Wendell and other developers will achieve in the months ahead.

In conclusion, our discussion with Wendell about the challenges of ray tracing in game development highlighted the complexities of this technology and the importance of having technical expertise on hand. From rasterization to secondary scenes, this conversation revealed the intricacies of game development and provided a unique insight into the world of game creation.

"WEBVTTKind: captionsLanguage: enso you want to introduce yourself i'm wendell i do stuff it's i'm a computer janitor i'm sort of glorified we're doing this cross posting thing it's really weird people can check out your channel but i'm also posting it on my channel so it's like in places but there's not a lot of overlap between our audiences right now probably so no and your audience is much larger than mine so this well i mean unless you're talking about art but it's like my i've had some reach but not not with podcasts or whatnot so i thought i'd give it a stab well for everybody in our audience this is brian you might remember pixel art lisa sue or uh pixel art me being chased by the boulder when threadripper launched because threadripper 3000 was an historic occasion very exciting that comes to us courtesy of bryant yeah i also did the one for you because how i started contact with you because i've listened to your level 1 tech for years but i did like a couple of series of just images for youtubers that i ended up using a lot of time like my day would be filled up by their content so i'd always watch you guys and and stuff on there so i'm like i'm gonna do an image for level one i just i feel like you can't see it but i appreciate it i pointed the camera at the poster it's hanging up in my office so i moved the camera so you can kind of see it it's sort of fun but uh yeah it's pretty glorious really yeah that one was funny i think ryan's lying on the floor yeah passed out yeah yeah so yeah i did another one for like john lennon i did one more for uh alex mccalla but it's like it's i'm like how often do you get it from like an actual professional artist you know what i mean it's like it's one thing i wish we had krista here because like artists that are also like super technical kind of unicorns and so uh you're definitely a unicorn because we've had some very super technical conversations and the main thing on the agenda today is to talk about you know stream deck and uh what kind of steam deck yeah steam deck yeah not the elgato thing steam deck i was so paranoid i was gonna do that ah yeah steamboat okay i got your back you can stream with the steam deck from your steam library well we're gonna get into that because that's part of the proton layer i believe so well this is uh is this the streaming part is or i thought the proton layer was for running natively well it is but you said that you could also use proton for streaming i remember you telling me that yeah so you can kind of cheat a little bit and uh it turns out that the graphics abstraction layers you can you can do some fun things we know that from working on like the looking glass side of things but that's conversation for another day or in a conversation for a little bit we're getting ahead of ourselves yeah and for anyone who who doesn't know wendell i i think it's funny because over the last like three years i've noticed you also kind of became tech support for youtubers on their obscure projects like you popped onto gamer's nexus and linus just when they're doing some weird server thing or something in linux that's out of their normal range and then you always show up on their channel um one of them you were the first person that i wanted to talk to when the steam deck was announced um i guess i should probably introduce myself now just in case but i'm i am brian hemskirk i'm the lead artist at massive damage i do art direction and stuff there as well i put a lower third in it'll be fine yeah yeah i uh i've been um sorry i've worked in the game industry for 10 years um shipped products on lots of different platforms consoles pc uh phones things like that so um anyways when it comes to the steam deck i'm gonna hope to bring some of that kind of perspective because and wendell was one of the first people i wanted to talk to because of this because i really wanted more insight on the linux perspective i did ask a bunch of developers what it's going to be like so for those who are listening that might not be aware um it looks like the steam deck runs on a version of steemo x which is based on a linux distribution that you would know the name of wendell debian i think so it's based on this specific distribution they're going to be doing a bunch of updates before so we can't it wouldn't be amazingly helpful to look into what performance is like right now because they're going to be updating proton and probably the base linux just drove before release so should be a lot of changes in that regard um but i have i have a lot of questions myself so uh one of the things that i was interested in is like with steam deck what what are developers like it doesn't look like they're releasing we've reached out at my work we haven't heard back from valve yet so i don't i don't know the details on this we have pretty good relationships with steam otherwise um but one of the big things that we're curious about is is there any way to target this directly because it sounds like to us from uh valve's perspective that um the dev kits are more or less just early steam deck so you can at least do your qa on it and that's that's good obviously knowing how it performs might encourage people to make a different performance mode if there are specific effects that you see tanking it when you're profiling you might might reduce or have a low version of your game that or medium version of the game that emits some of those effects or half-rezzes them or something like that um but yeah i'm curious what your thoughts are and what we would have to do as devs to target this like to me it looks like we would likely need to port our games to linux and i have thoughts on that we can get into that as well as uh target falcon as the api and we don't want to target oh opengl we do not want to gel so yeah opengl was one of those things where the the artists and the programmers i don't they didn't get together they didn't go to lunch they should have gone to lunch together a lot it should have um it's hard it's hard to predict but there's some really interesting things that have happened historically that suggests to me that valve sort of knows what they're doing with how to handle this and so the short the shortest answer is i don't think you're going to have to do anything super special to target the platform i think most of your work is probably going to be around the controls but even that i don't think it's going to be a lot of work and so what i if we look at what's happened historically one of the one one really interesting thing i think it was directx 9 there's this real-time vulcan translation thing like the dxvk like the directx to vulcan translation layer and there's a couple of things that have spun out of that some funded by valve and some from crazy people that can see in nine dimensions because they're super crazy programmers and some of the things that have fallen out of that is like the ability to run certain older versions of directx faster than native on modern hardware and it's because the the modern the older versions of directx and modern hardware no one is actually bothering to maintain anymore and having the real-time translation layer if you fix all the bugs because there's not it's not exactly a one-to-one mapping um you get better performance kind of automatically and this is probably intercepting all calls for specific hardware features and it's translating them on the fly to vulkan so there is in theory especially on a really unoptimized um really unoptimized api opportunities there for actually performance improvements versus yeah but there's also a lot of opportunities for regression both are possible yeah well sometimes you get a hardware assist so like the hardware of the day you know couldn't do culling the way that modern hardware does culling and so like you add a lot of overhead trying to deal with that programmatically computationally because now logic that you just loaded into a buffer and let the hardware deal with you have to do a little bit of logic on but if you get back some of the performance because all of that happens in hardware then mostly it's a win the specific example i'm thinking of is like gta 5 because gta 5 i mean it's a popular game and theoretically you would think rockstars actually put a lot of optimization into it of course i can't 100 say that with a straight face because half the loading time is improperly parsing json i see rockstar i see what you do but uh you know theoretically a lot of optimization has gone into that and some of you can't run it online because you've you've basically modified somebody else's game but um you can run it a little faster on more modern graphics apis than it would run on the older graphics apis although gta 5 is not the best example for another reason which is they implement some of those though they added the support for that over the years so i have a question based on this if you mind if i intercept for a second so obviously rdna 2 has like a set a set uh feature set that's unlikely to get targeted on pc right unless something helps either the apis take advantage of it or um we target it more directly so like i think is sampler feedback uh calling a new modern calling method you have any idea what that specifically is as an rdna2 feature no i have there's there's i it's relatively opaque because um if you look at how vega evolved into our dna uh into rdna one they did a lot of really clever things with the architecture um to carry forward some of the functionality from vega i'm thinking of like what it's going to take to get the most out of this because if we just talk about the base back so it's like it looks like what probably what 512 shaders uh so it's double the shaders of a switch right and it's double the clocks so at just a stock uh flops comparison it's basically a four times performance increase over a nintendo switch and then when you add the architecture shift going from what is it uh it's basically equivalent to the 900 series which was maxwell on nvidia so going from maxwell to rd and a2 i'd guess would be maybe like a 20 to 30 percent performance increase as well so we're looking at something roughly five to six times more powerful than a switch and uh the question is going from a closed ecosystem where they can really target that you know the 256 shaders of the switch and and so on versus an open ecosystem where we're more likely to lose performance from api overhead you know just the way that these the games are built not being able to target it as directly just what you typically lose so like i don't know what are your thoughts on some of that um there's a lot of things that are happening in parallel like samsung and the whole rdna thing and mobile and the whole so i think i think you're i think your numbers are a little optimistic overall because they're probably going to target much better power efficiency and if we look at the last few desktop generations it's like power efficiency we're way on the other side of that bathtub curve so um i i don't know i your numbers are a little are a little optimistic i think but no i'm talking about theoretical performance not in game performance right so i'm expecting it to be like this is like if you talk about what the hardware is in theory if you had both of them in like an environment where you could just write one game for each of them separately as accurately to the to the point as you can you get probably somewhere between four to six times the performance on the steam deck versus the switch but that's not the case because with with uh nvidia's direct api for the switch the nintendo api you can target the hardware much more directly and optimize more directly versus the steam deck where i'm not sure we're going to be getting that so i'm i'm guessing myself to see like a two to three times performance increase that's like my lowest expectation that's my low so that's probably more it's probably more accurate because i'm trying to think about like on the linux side there's the binary blob we still have to deal with the binary blob even though amd gpu is pretty open i don't think i guess i guess i would say that in terms of like you know steam deck as a switch thing will somebody target that specific set of hardware i don't think i don't think so i don't think valve will even want you to do that i'm not talking about hardware i'm talking about features okay so let me give you a really specific example this might help it out so uh um sorry vega and vega 56 and 64 introduced half floats right in your workloads and we saw them used in literally no games on pc even though they're used on console games fairly regularly on the ps4 pro as a means to push the performance higher but and you see it even on switch because the switch can do half floats which is odd because nvidia disabled it with the 3000 series by the looks of things you get same half float performances full flow 32 bit versus 16. that's interesting they probably they probably did that to avoid competing with their workstation cards a whole other dimension of crazy well so the funny thing is that nvidia added half floats to the 2000 series and we actually saw a small pocket of games targeted i think wolfenstein 2 did they targeted half loads so that we have mixed precision workloads in there um and then what happened is you would see all of a sudden if you look at those benchmarks for the game vega 64 and 56 would jump up a couple slots in their gpus when this feature that was never used it was never targeted for radeon was finally available on a set of nvidia cards too then they saw some games start to utilize it there's a bunch of things that we can start targeting in engines but they have to be in engines available for developers to implement easily for us to see performance and there has to be incentive on pc and this is something i mentioned on moore's law is dead is like game developers are hoping that like the minimum spec on pcs shifts to like a 30-60 or something so we can at least have our full feature set to target so i think the answer to that one one of the answers to that is why that is has not been the case on pc is because i think probably the lag time between the drivers and the hardware and then like the game engines because on a console platform that time is basically zero if you look at some of the experiments that we've done with unreal engine um some of the ray tracing stuff in unreal engine the functionality is there in amd cards but like the development environment for developers is a little sketchy and unpredictable and weird ways well you see you're talking about the six months like time pulling into pc versus consoles i mean console time the feature can exist right away right and then amd would probably work with the vendors but then getting engine integration and patches within engines and the other thing that a lot of people don't know about games is a lot of times game developers if they start developing a game and i know we do this in unity and i'm sure unreal developers do it too if you're working on a stable build of your game and you're in a stable build of that software you can be still using that build two years out you could be on unity 2016 or unreal 2016 for your game because you're more scared of introducing new bugs than you are of um of getting these new features and this is what like usually what happens when you start a new game then you're like oh let's jump to whatever the latest build is right now that we get even you google advanced which is the newest build that's stable right and then you start building your new game in that and then you get a chance to play with those features but oftentimes from the vast majority i mean there's some very technical developers that are willing to do the hurdles to get more features in all the time like um obviously the developers of like control and metro and all of that are crazy when it comes to stuff most of them make their own tools and stuff but the for the average developer you know they want to make sure their game is stable so they're a little bit scared to implement these new features even even outside the game engine you you have the exact same thing just like like looking at the amd gpu driver it's like oh variable refresh rate that's okay the hardware is there the primitives are there in amd gpu but does the compositor handle that is this a wayland compositor thing or is this something we're going to import to you know something for like pre-weyland it's kind of interesting when like a lot of people don't i mean the last few games that that i've worked on targeted pc first and expected otherwise but aaa developers often do the inverse they'll start on console and then they'll port to pc i remember when i was at gdc 2018 i had a sit down at a restaurant with someone that just did at that point in time one of the most successful pc ports of a game and i'm not going to mention it because just in case but they so they they did and people were commenting on how well optimized and what not this specific pc port was and that developer told me that they wished that they never had to port to pc and i thought that was a really funny perspective right from this developer wishing that pc didn't exist as an ecosystem after doing one of the most successful optimized pc ports the big problem with pc from a developer's standpoint is there's it's very hard to target it directly right like they they're not sure what to look at so like if the steam deck became like the lower end standard of it and they had something to shoot for that they can guarantee that and it really depends on adoption because if it sells even if it sold i'd say you know four or five million units in the first year or two probably wouldn't be enough to move a needle but if it sells 20 then all of a sudden people are going to actually start looking at it or 30 million you know what i mean so and this is kind of why i wanted to get into the specifics of it like what happens if it just took a giant portion of market share what would we have to change in our our development structure would we end up you know would that just be targeting vulcan and according to linux now i did talk to a message just a bunch of developers i knew had done linux ports in the past and the the perception was kind of like i'm not not going to say anyone specifically but they they basically most of them said that they were one of the hardest people to please because they would spend all of this time porting and then they would release it and they would get more negative reviews from linux gamers even though they tried really hard to release a linux distribution like a linux version of the game and then if they just didn't release the linux version they would have less headache and they would just play it and on windows somehow or either through a virtual machine or whatever they do to play it on windows and they wouldn't end up complaining about the linux port and that was the that was that perception i like i like i like what is happening with proton and i'm really excited by it and it is a really great amazing project but there's a part of me that thinks like you know all of the cruft and insanity of windows we're bringing all of that along trying to you know deal with that layer but at the same time the environment has got to be better than what it is on windows and i think that a lot of developers don't have have that experience because like the lower levels like the kernel and the command line stuff and that kind of stuff for linux is it is it it's a it's it's the pinnacle of civilization you know modern man it's like the greatest thing ever but then you know the graphical environment built on top of that really isn't that amazing this is kind of linux's biggest problem in a nutshell that it doesn't have people fine-tuning the interface to make it more manageable for them and those people would be luring the artists into the ecosystem right yeah and there's not there's not often not enough uh investment in that because you know open source love open source you know it's all really great stuff but it takes spending a lot of money and i think valve has done it right you know valve has greased an ungodly number of palms probably untold dollars but they don't want to talk about it and they don't want their developers to talk about it they just find people and they're like you're really good at doing what you do this is the general direction we need you to go in this these are the technical requirements we will fund you if you want to do this just you know don't talk about it don't make a big deal about it just go and uh you know the cat kind of came out of the bag surreptitiously with like dx vk and some of the other stuff it was like you know people are working on this i don't know what the details are behind that but it seems like a little cloak and dagger a little bit intrigued maybe the story's more interesting than or the imagine stories more interesting than reality but it is really exciting that those things and we will get there eventually like supporting legacy and supporting old stuff and supporting this that and the other we will get there eventually i think here's here's something maybe that you haven't thought about but it's very related to what we're doing so with the switch the nintendo switch and also you know valves steam deck got that right yes uh the importance of that thing doing computation is getting to be less and less and less important which dramatically increases the device lifetime potentially the the kind of the beauty of the steam deck is running these things at all at 15 watts right like yeah i mean you could you could play it locally it could be a thing it depends on depends on the game i guess yeah and the network connection and all the other stuff but the thing that changed my mind i was i thought along the same lines but what changed my mind was uh i built a pi boy dmg which is a raspberry pi vaguely reminiscent but for the purposes of copyright entirely distinct from the nintendo the old-school nintendo game boy and it's got a decent but not high resolution at all it's like 640 by something lcd screen and it has a reasonably long lithium polymer battery life and you've got all your classic games on it and your emulators and you know it can play some stuff locally but it is also a heck of a machine for streaming from steam and i've done the wi-fi hotspot thing off of my phone and streamed to that remotely with a vpn and it's not been a completely awful experience i mean latency is always an issue when it comes to streaming these kind of things right and the not not necessarily that the moment-to-moment latency is bad is that crucial moments of latency can hit you in a way that maybe wouldn't hit you locally and i like just like weird spikes or you know outages or things like that i mean that would hurt you if you're playing an online game anyways but it's weirder when it's your entire video feed like you feel like you're cut off from doing anything at that point in time um so it's like i can see somewhat that being the future and streaming your your desktop onto the switch but onto the steam deck i mean i'm i'm curious to see what this device can do right i'm wondering if you know because the big thing we're talking about earlier is like we're talking theoretically four to six times switch performance in game maybe less maybe three would having a game ported directly to linux and vulcan what percentage of that do you think we could scrape out of this device i think if the platform is optimized to the nth degree i think that will be surprising because i always go back to like the the doom eternal port with vulcan and the performance they were able to squeeze out of that when that game launched it was jaw dropping it's like is this what an optimized game with vulcan looks like because it was insane you and i we ran we never published it but you and i ran some tests on variable rate shading and variable rate shading and mesh shading when you run those demos on 3d marks it's like you look at variable right shading and it seems to have a slight the tier 2 seems to have a slight performance advantage on radeon compared to nvidia from a base it's close but it's it seems like there's a slight advantage to radeon with variable rate shading and with mesh shading it's kind of weird because the nvidia demo starts out at a much higher frame rate than the if you run the same test on a radeon on like a 6900 xt versus a 3090 but the performance gain percentage is much higher on a radeon with mesh shading on so part of me is wondering like we're not technically fully utilizing the steam deck until all of these features are turned on yeah from from my perspective we're not seeing what the hardware can theoretically do until we're using its full feature set and getting that never comes to pass though you gotta we have historically so this is the question that i'm saying is is this device going to make a big enough of a splash for that reality to come true i was thinking about you know as you said that importing and the available features have you seen some of the insane projects like people have ported mario brothers or a mario brothers-like game that is similar to but for the purposes of copyright holy distinct from uh the merit on an atari 2600 it's like we can play super mario brothers on an atari 2600 yes but not really because those development tools did not exist at that time and it also looks way different but it plays similarly steam deck targeting hardware right this is kind of what all of this comes down to is this something we can target the way that we want to target you know it's like i see all of these youtubers specifically it's like my theme's being filled with steam deck things with all these people being excited and it's like i think immediately it will give you everything up to ps4 portable right like you'll be able to play all of your games up to ps4 basically portable and some of the higher end games might not run as well as you like and some of them might run way better like do maternal or i would bet even the decimal engine games would probably run pretty well but you know like death stranding or horizon zero dawn because they but it's there's specific things that i think would target the hardware well and work well on it and then there might be some surprise cases where it's just like this doesn't run that great yeah yeah i think that's i i mean that i think that's likely but i think the thing that is going to surprise everybody about this the the the hardware is that it's going to have longevity that the switch probably doesn't one of the great things about the ds and like the original game boy like the clamshell gameboy is that you can just pick one of those up and you can play the old games and you know some people do that and it's basically works from now until the end of time i hope it's like that with this where if you've got your whole library on it you know you get the one terabyte model you can just pick it up and go steam kind of wants to be online it kind of wants a whole bunch of other stuff but it would be amazing if you know 10 or 15 years from now like some of nintendo's products you're able to do that with a pc without sort of needing good old games to have you know containerized in some way so that it runs in a in a modern way is not gonna be like proton 5000 that just translates everything to vulcan no matter what i mean maybe maybe that could happen but i think that because that device will have such a useful a very long useful lifetime i think that the kind of porting that you're talking about will is kind of inevitable if it gets the numbers because it just builds cumulatively you know if they do like a second version or whatever it's like well we can count on h.265 streaming you know for both of these and so even if it doesn't run natively you could still stream it and it's a little bit of a second class experience but at the end of its useful lifetime it's only just doing streaming but um that's kind of i mean that's fine that kind of makes sense i think that'll work well if this thing is as popular as i think it'll be it will probably have a cult following from now until the end of time right yeah yeah very much like that like the playstation vita has more it had such a short window of active developer support but the thing just lives on despite that hackability and openness it goes back to the thing that you said at the very beginning where it's like it's closed now it's open and then look it lives on yeah it lives on well most devices are hacked initially just to put linux on it right so can it run doom i was gonna say so it's like i bet your forums are already on this right it's like what's this this is like the livelihood of your forums like oh we're gonna get like a community that is gonna be maybe even a bit bigger that when we write stuff for they're going to be looking for our software immediately to use and it's like it's like the the window of people accessing and modifying their software is going to go up dramatically for a period of time right yeah this is this is kind of a no-brainer though because in you know look at like the gpd when and other people like that that's like i'm going to build a device that is exactly what i want and on the forum to your point there are a lot of people that are taking devices like that and they're like i'm going to take it apart and modify it i'm going to do a replacement motherboard for this thing that i think that they got everything particularly right i can keep using the input peripherals no problem but you know so it's like i can do a hardware replacement on this you know horribly ancient thing with something that's a little less ancient and even though it's proprietary having a custom circuit board made these days yeah i can probably get some other people to do it with me and kickstarter it and and we'll be good to go and so like gpd did that and now gpd has like a whole line of stuff that's that's basically kind of a predecessor and it's been insanely popular do you think this is going to have a significant shift in linux adoption for average consumers obviously linux is crazy in servers and several other places but i mean for the average user i think it will be the biggest ticket i think it'll have a knock-on effect but i think the fact that linux is in play is completely transparent to the end user like the fact that android you know has a linux kernel underneath hasn't really mattered for linux adoption and so in this same context i think that that is still going to matter now will the work that valve is doing be lift and shiftable over to the linux environment yes and ultimately i think it'll be up to game developers and members of the community to do that i don't think it's going to fall out of having this with no effort so you'll have a positive knock-on effect but i don't think that you're going to have a market where people are like yes i must have the linux um okay um however the only thing that i could see maybe changing that is if like linux developers like or just individual people that write things for linux write such useful programs that people are drawn to the platform from like oh i have a steam deck and i was able to pull this cool mod in or i was able to do this right and modify the game or the ecosystem in this way that's really interesting and then they go to windows and they're locked from being able to do that those kind of things can be more persuasive right for the average user they don't want to go nitty-gritty with code they don't want to go like trying to figure these things but if there's a feature that they find easy to implement and use in a specific ecosystem that they can talk about and communicate easily right that that has potential to catch on i was gonna say microsoft is ahead of you there and this is already played out for web developers so like web developing um depending on what you're doing and depending on the languages and a whole bunch of other things doing certain kinds of development on mac like brew kind of sucks and the old the old bash kind of sucks and like mac os is kind of rotting from the inside out like it's a pretty veneer but there's a lot of bugs and if you're a developer it's a special kind of suffering linux has a different kind of suffering and windows is just completely unprepared for doing you know that kind of development on the web but the windows subsystem for linux oh my gosh it is it is transformational because you can have your little sandbox you could run docker or coupe micro kubernetes or whatever you can have your little environment on your windows box that's just like your server box in the cloud so windows finally now with the windows terminal and everything else it is an incredible first class experience for those kinds of things so if somebody does what to your you know to your point if somebody does write a really amazing widget that makes life easy and does some stuff on you know linux blah blah it's immediately available on windows the reverse is not true on linux if you want to be able to run windows stuff your only way to do it now is like seamlessly run a virtual machine and we're as far as i know the the people that are always writing the how to's and doing the stuff on that so that you can run your windows apps seamlessly on linux but you're basically running a full fat virtual machine you can do a full fat virtual machine with gpu acceleration but that's the only thing that has guaranteed compatibility and if it's not for anti-cheats that works really well for gaming too and we've also got some technologies that'll let you you know uh run an accelerated game transparently alongside windows and how to's and stuff that go with that and now we have some questionably okay with the manufacturer scripts like vgpu unlock which will turn your geforce card into a card that can actually share a workload between two different heterogeneous uh operating systems now is this where i met my i o mmu groups come into play yeah i don't even know groups come into play when you've got multiple gpus and you need to separate one gpu and one group for one virtual machine versus another gpu and another group for another another virtual machine which i think i heard robert hallick in a talk from amd it was like a technical talk literally say i am i am mmu groups and said wenda will get a kick out of this and i'm like did robert halleck just nod to wendell i literally was the guy five years ago that showed up and was like we need to get on top of this because look if you're gonna have like all the old operating system baggage i think that like for computers computing in general and computing hygiene and security we cannot spend the time there's not the human existence is is is too amazing and too awesome to force people to work on this when we make a mistake we should be able to abandon the past but instead what happens is we have to maintain that mistake until the end of time and so whatever weird quirk the windows 95 api had for calculating the date or whatever you had there's thousands of man years of programmer time to be able to support that all the weird quirks and crazy stuff that's in proton for dealing with windows games is recreating something that already exists and is miserable on the windows platform look if we can just figure out this virtual machine thing we can package it all up in a container and when we need to go into the past we can go into the past in the container but we are not encumbered by it it's kind of funny that you mentioned that because this makes me think of microsoft strategy for xbox specifically and maybe i'm misunderstanding something because i'm not a genius on any of these fronts but it looks to me like microsoft shifted its game development uh ethos from hardware specific to api like xbox like an api and if you look at what they do with like retro it's like panzer dragon order which is like a crazy old sega it's an original xbox uh rail shooter made by sega and i love that game and it just runs natively on 4k on if you played on an xbox one x or series x and it's crazy to see this game from i don't know what 2004. in 4k all of a sudden like no jaggies like perfectly clear you're like this is this is really interesting but they couldn't do that if they weren't treating their entire ecosystem like an api yeah and then so it's like it's a completely different way to target game development it's very different than what you see from and it's it's closer in a lot of ways to what uh steam is doing so it's kind of interesting because i listened to someone i'm trying to remember who it was to give them credit where it's due but they were talking about how steam they did they said it like a month before the steam deck came out and they were saying that steam is the only person that could pull over handled like this because um when you look at gpd win or the ineo or things like that they can't hit economy of scale yeah it's it's impossible and they can't recoup hardware costs in software sales and that's the only pc player that has the chat well there's only two pc players that have the chance to win in that space and that's microsoft and steam those are the only two people in that ecosystem that could actually pull this off yeah a lot of a lot of this this this archetype this paradigm this the thing that i'm spouting is not anything new even in terms of like the computer science architecture of it there's a thing in windows called windows side by side where different conflicting versions of things are maintained and that's one of the ways that they're able to support 47 billion versions of dotnet and the applications that are built in them while they can update the version of.net without without retroactively breaking any applications and so you know people inside microsoft know the pain and suffering and misery of trying to support old stuff and it's like well the more we can stuff that into into a box to where we don't have to deal with it then the better off we're going to be this is the talk like we can go back to where engines are going because there's two things i want to say from this because obviously there's a push in the game industry to have because your pixels are set right and what people don't understand with your game development and with your lods and stuff like this is your scene is variable and you make lods that you said at specific distances or whatnot right but sometimes you have like a character model really close on the screen right and it has like seven polygons or maybe that's exaggeration nowadays but like it's a hundred polygons 2012 tomb raider yeah yeah it's like 150 polygons filling up like 20 of your pixels right and in the far background you have like a mesh with 60 000 polygons in four pixels right so it's like and this is like this is what nanite is trying to solve in unreal and there are other solutions that i've heard i've tried for this because i mean the the raster the render workload for rendering the pixels should be the same but it's not but it's it's so data-heavy to do that so you have to manage your data to be essentially rebuilding the scene on a frame by frame base on a frame by frame basis to use only the pixels that are necessary and this is like so this is the one thing i wanted to say from this and there was another thing that i wanted to say from this i was trying to remember what it is um so yeah hardware vendors are attempting to do this right to target specific and you know mesh shaders are a means to this to some extent or closer to it right just that your your meshes are doing that automatically and you don't have to make all of the lods if the computer science hadn't been invented yet but we're getting to the point where we do have the computer science and you do see things like this in the engines that you have and so that those kinds of things will make how much horsepower you need more a function of um gpu horsepower than future upgrade technology like it's kind of a curve like the it's like okay we're going to add a ton more computational horsepower but it didn't really matter the reason the algorithms the reverse of this is really easy to understand you look at the 3090 performance look at the 3090 performance at 1080p it's basically the same as 1440p it's it's it's almost the same as 4k we know that the 3090 can do the computation why is it not four times and the answer is the software well it's partially the software it's like it is much harder on the uh like okay so it's harder on output but not everything in your scene it changes right it's like you're still you're still uh mapping the same textures onto your meshes you're still mapping some of these things uh scale with little performance loss and some of these things scale with massive it's just like alpha transparencies and blending on pixels that that murders it right so it's like if you have like four explosions in the foreground and they're all like semi-transparent and they're blending with pixels behind it the computation on that is insane right versus some things like there's almost no difference on your hardware in terms of how many pixels it's targeting and then other things there's a lot of difference which is why it doesn't scale completely linearly from resolution to resolution as it some things just use different hardware and some things uh don't actually push they don't they don't need the scale um i i did think again of what i was going to say before oh some of what we were just talking about is the reason so like when i was talking with moore's law's dad about the draw call issue and if you work in a game studio you hear the word especially if you're targeting pc you hear draw calls spoken more than almost anything else especially when you're in the optimizations phase of games and the the whole issue with why it's an issue because i think it was oxide games who did ashes of singularity and whatnot and before that they had another game but they talked about how they thought it was funny because when you worked on consoles you'd have 20 times more draw calls than doing pc and directx 11. and that was because at the time right it's like directx was just built on top of all the previous versions going back to like directx 1 and this is before we had fixed function hardware we didn't have pixel shaders we didn't have unified shaders we didn't have like any of these things in a gpu right it was like wasn't even called the gpu it was called a vga card or whatever yeah yeah so it's like and people were using it either their agp or their pcie slot or pci slot not even pcie you know and it's like i remember those days i had a tnt2 and a geforce 256 and all that stuff going back you know but it's like the the software was trying so hard to keep compatibility that it ended up creating massive bottlenecks in the pipeline so it's it's like how often do we reinvent the wheel is kind of the question yeah some of it you know it's just the evolution of the methodologies that we use for rendering and four this that any other and you know um ray tracing throws a big wrinkle in all of that because nobody thought that we would be doing real-time ray tracing now and so now that we are doing real-time ray tracing that's opening up a whole other set of computer science that needs to be done in order to get it back to the well we're filling up a finite amount of pixels how much horsepower you need to do that should be a function of number of pixels not necessarily what's going on on them or at least the horsepower should be dominated by number of pixels and detail i guess until ray tracing the way that it was is if a tree falls in the woods it literally does not make a sound not only that the tree doesn't exist so the problem is imagine that you're looking in a room that's filled with marbles and lights and all that all kinds of stuff if you're looking at stuff and you're doing ray tracing and the stuff that you're not looking at doesn't exist then things get weird with the ray tracing because the light is bouncing around the room and all of the stuff that you were previously calling and all of this magical computer science we've been doing up until ray tracing which what we're talking about is magical stuff to do a whole other layer of culling with like simplifying meshes and reducing the complexity of models that are far away because you can't even tell anyway think of like a more ghetto fabulous version of this is like the texture pop in like a wolfenstein because it's like hey we're not even going to bother rendering this until we can get it just right and it's like oh look it's suddenly more high fidelity but ray tracing just takes all of that and just cause the reflections depend on stuff you can't see necessarily well this is a you can summarize it pretty easily so like one of the most common reflection methods used today is called spring screen space reflections but when you see in games literally the second you get in front of it so if your head is blocking it you'll see like weird lines going up and down off of your character's head blocking the reflection where it doesn't have that so what we're talking about here involves i think it's called the camera frustrum which is everything the camera sees right exists and then anything outside of that space doesn't and the problem is is when you bounce array it's looking for something that may not be there so it can bounce up and down and behind an object and catch like the back side of an object right or it can bounce off screen and hit something and suddenly you need visual information that you didn't need before right and this is this is the nature of why ray tracing makes some things more difficult right and you know sometimes they just i don't i don't know what ratchet and clank solution is it looks like they just do it at a lower resolution so that it's basically doing another version of the scene with not necessarily identical assets sometimes they're lower quality assets to try to speed up the process of that being a secondary scene i mean gamecube did weird versions of this with like wave race where i had a second camera that it was rendering everything to and then it was combining them again but it's cheating well i mean games are cheating if you ever work in games it's like the mystery around games dissolves immediately when you're like janking behind one of these scenes to create visual effects it's like you're making things just warp weird it's like the second you look behind the camera and you look at a scene from the side you're like what is going on i mean rasterization is that in general it's literally as far as i'm aware it's it's changing the textures and all the objects it's modifying them to the camera's perspective so if you could set another camera in the scene that just kept all of the textures being distorted and stuff off of the initial camera you would see the crazy looking scene it wouldn't even make sense right and that's why rasterization can happen so quickly but we're kind of going all over the place but it's that's fine but all of this has to do with the uh the new thing from valve and game hardware like the developer side of things this has been a fun chat well i hope you've enjoyed this uh level one ramble with uh with me and brian you know be sure to check out his his new channel and all the fun stuff that he's been working on and uh there's probably a longer unedited version of this or less edited version of this it's available on his channel so you can check that out well i don't even know i might just wait for the version from you no no 2023 is like okay there we go yeah yeah but maybe i'll maybe i'll put the unknotted version live or i'll do a quick edit i miss myself but yeah thank you so much for having me thanks for taking the time to talk um no thank you for talking with me because you for what you do you are a really incredibly talented artist and having the technical perspective also just saves me a lot of time from having to like try to figure out how it relates to the stuff that i'm already interested in so it works out a lot it's it's there's definitely some self-interest in uh being able to talk to you and and understand your perspective on industry stuff because you get what's happening with the shader you get what's happening with the render pipeline and you get what's happening with the hardware people are going for because really a lot of this stuff is educated guesses they made five years ago that's just now making it into hardware because that's the life cycle that we're dealing with and so i have like i have like a bare minimum understanding like i see some profiling i see you know the typical hey what's this why is this taking up so much time in the render pipeline right this like stuff that is it's part of all of this but it's not like i'm i want to find out even more right i'm looking to have like part of the reason i want to do this is just talk to developers who've gone even more in depth right like the ones that have scraped that little bit of processing out of the corner right or found some some kind of work around that saves a bunch of time but thank you so much for your time wendell thank you it's no it's it's always great to to do this if you ever want to do any pixel art collabs in the future you just feel free to reach out and i heard there's some new thread rippers coming out i don't know sometime maybe i don't know yeah anyways so yeah catch this on pixels refreshed and on level one tech in 2023 cool we'll see you there\n"