The Complexity of Compile Testing: A Developer's Perspective

When it comes to compile testing, developers often find themselves grappling with the nuances of how different tools and systems interact with each other. As arts people, they typically don't have programming experience, but their work requires them to understand the complexities of compiling code. In many cases, developers will compile bits and pieces of their code or work on individual components of a larger project. However, when working in teams with version control, such as Git, the compilation process can become significantly more complicated.

The first compile of the day is often the most time-consuming, as it involves recompiling not just the developer's own code but also the changes made by their colleagues. This can be a lengthy process, especially if multiple developers are working on different parts of the project simultaneously. In contrast, subsequent compiles may be faster, as only the changes made since the last compile need to be recompiled.

In web development scenarios, such as with Node.js, background processes can be kicked off to perform automated testing. This approach allows developers to focus on their work while still ensuring that the code is thoroughly tested. Test-driven development (TDD) is another strategy used by some developers, where every time a bug is reported or a trouble ticket is submitted, a test is created to verify that the issue persists.

The process of compile testing can be further complicated by the need for CPU performance benchmarks. There are two main mindsets: one focused on ensuring that the code produces reliable results with consistent performance across different processors, and another concerned with fairness in benchmarking.

One challenge in compile testing is the difficulty in creating unbiased benchmarks. Developers must navigate complex systems to ensure that their tests accurately reflect real-world scenarios. In some cases, this can be a daunting task, especially when dealing with large projects or multiple contributors. To address this issue, it's essential to have a deep understanding of how different tools and systems interact.

The author of the text recalls a situation where multiple benchmarks were conducted for GCC and Mingw, but the results raised concerns about the accuracy of the tests. After retesting and reevaluating the data, the results were confirmed, and the findings were published with caveats to ensure that readers could understand the complexities involved.

In conclusion, compile testing is a complex process that requires a deep understanding of how different tools and systems interact. While there are challenges in creating unbiased benchmarks, developers can use various strategies such as test-driven development and automated testing to improve the accuracy and reliability of their results. By acknowledging the nuances of compile testing and using the right approach, developers can ensure that their code is thoroughly tested and meets the required standards.

A Note on Wendell's Content

The author of the text mentions that they will provide a video link in the comments section for viewers interested in learning more about compile testing and related topics. The video is produced by Wendell, an expert in his field who has created numerous educational content pieces. Wendell's work is highly recommended by developers looking to improve their skills and knowledge in this area.

The Author's Experience

In a surprising twist, the author mentions that they had to cut one of their initial videos from publication due to concerns about its accuracy. After retesting and reevaluating the data, the results were confirmed, and the video was revised to ensure that readers could understand the complexities involved. This experience highlights the importance of rigorous testing and peer review in the development process.

The Future of Compile Testing

As developers continue to push the boundaries of what is possible with code, the need for accurate and reliable compile testing will only grow. By continuing to explore new strategies and approaches, developers can improve their understanding of how different tools and systems interact, leading to faster and more efficient development processes. As Wendell's content continues to educate and inspire, we look forward to seeing the next steps in this ongoing conversation about compile testing.

"WEBVTTKind: captionsLanguage: enhey Ron we're still at LTX 2019 and now I'm joined by Wendell from level one tax LTX is crazy yes it is so weird we've been on the show floor and Linus brought us out here so well thank him for making this possible thank you and we have a pretty fun topic so I mentioned to you at the am the event in June that we had a GCC compiler for CPUs and we saw performance we didn't understand before that this video is brought to you by coursers hydro x water cooling series of course our strength is bringing water cooling to the masses and it has built out cooling solutions with industry leaders to help newcomers get into open-loop cooling of course hair has fittings adaptors GP water blocked CPU water blocks pump has combos and radiators all available in the hydraulics line as you can see in our footage these kits can be used to build the beautiful open loop systems learn more at the link of the description below so for people who've missed that basically the performance we saw was Intel was always a hundred percent of time no matter what CPU is way at the bottom even a ninety nine hundred K was losing to like 2600 or something and then AMD as we've tested rise in three thousand we've confirmed what you told me then which was we've invented or we've accidentally stumbled upon a cache benchmark turns out compiling code it's the branches thing ever and your l1 and your l2 cache it's not really gonna help a lot but the giant l3 turns out mean game cache okay well I wonder what they'll call it on Rome yeah that's nice yeah can't be game server cache yeah so so that's what we were talking about was basically the results were more cash more better for those benchmarks can you explain to me why well it depends it depends a lot on the compiler well not a lot okay well in general terms it actually doesn't really depend on the compiler or a whole bunch of other things but with a very large code base we're talking ultimately about processing text files and there's different stages of the compile process like some of it is lexical and you're taking the textual code and converting it into symbols and lookups and stuff like that and some of that stuff maybe is gonna be super clock speed dependent but there's also like linking type process where you take your compiled code and then you link it to a whole bunch of other libraries you have to do a lot of i/o for that there's a lot of memory operations but the compiler one is not super optimized because compilers are hard enough and optimizing a compiler is really hard but the compiler does know about optimization in another way in terms of you can tell it to the binary that it's producing should be optimized for memory space or runtime space or something else like that and the compiler will have different strategies for dealing with that in the end code but the problem is that what it's doing is so complicated and so branchy that the way that modern processors are designed you're not really going to get a lot of benefit short of being able to read stuff in from somewhere else faster okay that makes sense yes so because that's what we're talking about is a 3900 X and anything else well I guess a good example 3630 700 X 3800 X all the same compile time in that benchmark yeah so nothing else mattered yeah yeah when I when you email me it was like I don't know I'm not really 100% sure what's gonna happen in like scenario a or C so I took a system and set up I think it was 32 gigs of RAM and I set up a 24 gig ram disk and then 8 gigs of like actual ram and just had it run from the RAM disk because it seemed like it was mostly gonna be down to iOS but I am larger projects and like with the fir Onix test suite like compiling the Linux kernel I mean almost all of that is down to IO so like they'll run that from a ram disk or whatever and your kernel compile time and I mean modern system is like 30 seconds now ok yeah and it's just you know the kernel is maybe so that point are you are you if you use an SSD would you just be bottlenecking the processors on the SSD yeah well I mean to a point yeah but it depends on how much you're having to recompile to because like let's say I'm a I'm a device driver developer for the kernel so I'm gonna go in and I'm gonna work on my device driver which is relatively tiny I don't have to recompile the entire kernel every time even if it's like even if I want to build the module and I'm not even thinking about kernel modules at this point I want to actually build it into the kernel but even if I make a change it's not gonna have to recompile literally everything in the kernel it just has to recompile the part that I did and then just relink everything to get so all of that is gonna be cash hits all the way down okay and let's talk about victim cash for a moment - oh yeah so Andy uses I think notably in this point victim cash yeah so what that means is that stuff falls out of the l2 cache and then goes into the l3 cache so meaning that the anything that is a victim of eviction from the l2 cache ends up being in l3 so l3 really just sits between main memory and like whatever the processors doing so you end up in a situation where maybe if there was a more intelligent cache strategy you could get a better benefit from having the monster game cache or whatever but really it's just a list of everything that you've you know needed in in the recent past but when you're compiling you might have a library that you revisit or some part of the program that you revisit over and over again that is not spatially or temporally located anywhere by having the giant cache you might need something five or ten minutes later not really in five or ten minutes but a long time later in the process and most or all of it is still living in in the victim cache okay as opposed to a more intelligent caching strategy where it might look at what it's actually doing and say okay I'm gonna return to this lookup table a whole bunch for converting like this dictionary of symbols in this code into actual binary code and then like the caching strategy thing might say okay we're gonna use this a whole bunch let's just keep this in the l3 cache for now there's nothing that's that intelligent there so why does why does does the victim cache end up working to the benefit in this particular in let me ask the two versions of this question does victim cache work to the benefit of the CPUs in our compiles has the GCC won and does it work to the benefit of them in general and compiling I think in general it works pretty well in the GCC compiler test just because of the way that GCC is architected but like the Microsoft Visual C++ compiler does some more clever things with how it compiles and so I think the victim the victim cache ends up not working quite as well in those scenarios okay but on the AMD side you've got so much of it that it just it ends up not really matter and then you could make the argument that like the cache that you're looking at unlucky na minha RK because it tries to be a little bit more intelligent and because compiling is such an unusual workflow compared to like what you would normally be wanting to do in terms of executing a program that that cleverness on an Intel CPU is perhaps working against it okay so is there is there a thing I know the answer to this question but is there a perfect benchmark no give me something give me what you want that is right we can compile chromium let's see how fast it takes to compile chromium but my fear of that is that there were ultimately just reducing it to an IO like how many a ops can we do of how fast and you know read it in and eventually that becomes a ram benchmark yes but you run it a bunch of times and then dot the entire project lives in RAM at that point all right so does that even does that even really make sense but also like the whole optimization like the o2o3 things like I'm gonna optimize for memory footprint or I'm going to optimize your execution depending on what's going on inside the compiler that can also have vastly different performance characteristics so you get into some really interesting situations with the compiler in terms of like you tell the compiler to unroll loops and so in your source code you've got a loop where you're looping over a structure it's like the compiler can look at that and say well this structure is never going to be more than 64 autumn's big it will literally take your code remove it and just have the loop actually be just a linear list of instructions to go through all 64 things because that's faster for the CPU to execute just a linear list of instructions than to execute two or three or 100 or 1000 instructions and then jump back to so that is that behavior dictated by the compiler yes itself okay so you get into you gotta take compiler is you highlight benchmark yeah yeah okay okay I see the rabbit hole now yeah and I'm afraid of venturing yeah it's like oh look we compiled you know there's a microsoft's got a lot of open source stuff you can use the visual c++ compiler and and do that and intel you know for a long time intel was giving away their compiler and their compiler had a lot of really cool optimizations in it for their processors right that and perhaps less good optimizations for not their processor which may or may not have been intentional yeah or on top so you could end up with like this compiled took a very long time but this program actually runs better under this compiler and so some people misunderstand that when they're it's like I want to you know build a system for it's like it will the compiler will produce a deterministic result but you've got so many knobs and switches that you can tweak to control the output from the compiler that will have an effect on performance right so but at the end of the day if you are actually a developer working on a project like chromium most of the time you aren't even recompiling the entire project so it ends up just being how fast can we load this the stuff that we previously compiled from disk into memory link it and then kick the new thing out the door so is there do you have recommendations for this is where we started trying to do a compile benchmark we had a lot of people in the audience saying I work with code for a living I really want to know I don't have any benchmarks go do I won't know what CP is better so ignoring the part where we need to still figure out which how to test which CPU is better what is your recommendation for those people what do you what do you look up do you do you figure out what type of code you work on and then you look up a benchmark specifically for that we're yeah something it's it's really like more holistically workflow so maybe we could maybe we could put some projects together that are common types of projects kids and I wouldn't know this but I from your your email and the question in June I sort of started experimenting with this because let's say a web developer is using like nodejs and they're wanting to transpile their JavaScript into other JavaScript that like looked doing the performance counters without on Linux like I showed you the tabular alcoholic how it was doing all this crazy stuff the performance characteristic of that was wildly different than like compiling chromium and that was wildly different than like compiling GCC or even come halima Linux kernel Linux kernel actually this is fairly straightforward and then you go into the rabbit hole of like the whole Microsoft compiler so I think that because the I think we're in a situation park generally Andy's going to win because cash is so much larger in just about every scenario so and but more cores is not necessarily always better because some of those things are not multi-threaded processes not really but I think it's going to come down to finding a project that's like your project that somebody has actually benchmarked to look at it to say okay this is actually gonna this is actually gonna similar to my work yeah it's kind of like premiere because like you look at Adobe Premiere and it's like we want a benchmark premiere and it's like okay well that sounds reasonable but then it's like oh boy wait there's all these plugins and you know it's like are we benchmarking de luck thing or are we benchmarking encoder too and yeah yeah we got the red codec is like oh we you benchmark the red codec I need you to bench ProRes yeah it's like well it's the compiler thing I mean even though it's compiler things very similar in a lot of ways so it's it's it's it's there's not it there's not a silver bullet yeah there's something people are bringing up linking to me as well yeah I guess is that typically single threaded yes well I need to double check that but I think that in general it probably is a single thread of thing because you're usually bound to i/o you could go well I mean on an nvme you could go multi thread I'm not aware of a compiler that will are not aware of a linker that'll kick off and link a whole bunch of things but if you're again if you're in like developer mode and you're making a little bit of change to your tiny part of a project and recompiling the entire thing over and over again the linkers not gonna have to do much work in that scenario okay yeah so yeah I guess it sounds like then one of the approaches is kind of the shotgun approach that we take with some other types of benchmarks where you you pick a few different things and say here's the compiled benchmark suite here's like four different applications hopefully one of them fits your needs yeah and I think there's some so many user it sounds like what you're saying it sounds like there's a user responsibility to define your workload understand what you're doing right what problems are you trying to solve there a and then and then hopefully someone has a benchmark for you to solve that problem it might be that what you need is raid zero nvme that also sounds realistic based on your saying yeah yeah so I mean yeah you think about it it's like it really is true I mean with like the from the rise in 3600 the 3900 X might be a little less but that the just kiss of the cash differences but yeah I mean those those those processors are just and even like thread Ripper is kind of a monster for those kind of workloads background processes I think are also important to developers so that might be a thing that makes how responsive is the system while it's doing stuff in the background right well I have another question for you on this too so in your experience with code compiles are people typically compiling I'm not a programmer obviously arts people typically compiling bits and pieces of their code or do they do they do the whole projects all once I just talking about kind of like during the day it's usually like the first compile of the day will be like the one that takes the longest because it's recompiling a lot of things other than your code it depends on how you work so like if you work in a in a group where you've got like git or version control you might sit down at your machine and pull all of you all the team's work that they committed at the end of the day yesterday and then your first compile for the first thing that you change is going to compile all their stuff plus your stuff I see and so that one might take a while but then as you're banging on whatever it is that you've changed you're not really recompiling the whole project and in like a web development scenario who are you working on something like nodejs if you've got like a pretty good-sized set of changes you as a developer you might actually kick off some other background processes to do automated testing because like I automated testing is like a big thing so yeah like ways that take you seen account yeah yeah it'll run through and it'll do it's like we had this bug we're depending on what you're doing you know this whole other concept of like test-driven development where every time somebody creates a trouble ticket for your software you create a test to test for that problem so every time somebody says like okay there's a bug in the help screen of the program when you enter foo it crashes right so then you write a test that will interfere in the help screen and determine if the program crashes or not so if you're working in the help area you might while you're while you're still working you can say okay you know I've made a lot of changes let's test these and see if they actually work and you can kick those off those kid run on a build server so those could be like somewhere in your organization but increasingly they're running on your workstation okay because you can accidentally bring down the test infrastructure for the company if your codes really bad right yeah I've seen a few times I think I think the the biggest challenge is there's sort of two different mindsets looking at a looking for a benchmark of CPU performance for compiling which is our mindset is how do I produce something that's that's reliable that has the same results that test only one things not the compiler not the code quality just the CPU and then also how do we make it fair so that it represents both processors equally and it sounds like as is the case of premiere and some other instances it's not possible to do all those things yeah I don't think so so the only approach is I guess multiple benchmarks that are carefully planned yeah and and it is very difficult to come up with something that is not misleading if you don't understand how deep the rabbit-hole goes yes right on a lot of this stuff that's why when we first presented that data it was before we did any reviews for the GCC and we I almost I think I almost cut it from the initial publication because we had talked about it internally like this looks wrong it does not make sense and we retested it all and when Patrick retested it it produced the same results here we tested with mingw instead of cygwin and emesis too I think and I produced roughly the spoon is all how many people in the audience are actually recompiling mingw every day yeah well exactly and so we're not publishing it with a giant disclaimer saying this is the result for this specific thing and if you do this specific thing then hopefully this helps you and if not sorry give us your suggestions sometimes I think we've had it a couple of people in our audience and they're just like you have a video on blah blah and I talked to them and at the end of it really all they wanted was a video that they could send their boss to be like look I don't I think you giving me in a dual core i3 I need something better than there and I need a video that says that a dual core i3 is not going to cut it and it's like if that's what you need we'll glad to makes you that video right yeah that's easy well thank you for the education I really really hit it no no problem glad to be here this is a lot fun yeah absolutely and link below for level one tanks that is where he is from if you want to see more of Wendell's content highly recommended you'll learn a lot and hopefully we can maybe continue this discussion on compile testing in the future but in the four neckbeards in the comments yeah thank you for watching that we'll see you all next time the linker has been multi-threaded since before humanshey Ron we're still at LTX 2019 and now I'm joined by Wendell from level one tax LTX is crazy yes it is so weird we've been on the show floor and Linus brought us out here so well thank him for making this possible thank you and we have a pretty fun topic so I mentioned to you at the am the event in June that we had a GCC compiler for CPUs and we saw performance we didn't understand before that this video is brought to you by coursers hydro x water cooling series of course our strength is bringing water cooling to the masses and it has built out cooling solutions with industry leaders to help newcomers get into open-loop cooling of course hair has fittings adaptors GP water blocked CPU water blocks pump has combos and radiators all available in the hydraulics line as you can see in our footage these kits can be used to build the beautiful open loop systems learn more at the link of the description below so for people who've missed that basically the performance we saw was Intel was always a hundred percent of time no matter what CPU is way at the bottom even a ninety nine hundred K was losing to like 2600 or something and then AMD as we've tested rise in three thousand we've confirmed what you told me then which was we've invented or we've accidentally stumbled upon a cache benchmark turns out compiling code it's the branches thing ever and your l1 and your l2 cache it's not really gonna help a lot but the giant l3 turns out mean game cache okay well I wonder what they'll call it on Rome yeah that's nice yeah can't be game server cache yeah so so that's what we were talking about was basically the results were more cash more better for those benchmarks can you explain to me why well it depends it depends a lot on the compiler well not a lot okay well in general terms it actually doesn't really depend on the compiler or a whole bunch of other things but with a very large code base we're talking ultimately about processing text files and there's different stages of the compile process like some of it is lexical and you're taking the textual code and converting it into symbols and lookups and stuff like that and some of that stuff maybe is gonna be super clock speed dependent but there's also like linking type process where you take your compiled code and then you link it to a whole bunch of other libraries you have to do a lot of i/o for that there's a lot of memory operations but the compiler one is not super optimized because compilers are hard enough and optimizing a compiler is really hard but the compiler does know about optimization in another way in terms of you can tell it to the binary that it's producing should be optimized for memory space or runtime space or something else like that and the compiler will have different strategies for dealing with that in the end code but the problem is that what it's doing is so complicated and so branchy that the way that modern processors are designed you're not really going to get a lot of benefit short of being able to read stuff in from somewhere else faster okay that makes sense yes so because that's what we're talking about is a 3900 X and anything else well I guess a good example 3630 700 X 3800 X all the same compile time in that benchmark yeah so nothing else mattered yeah yeah when I when you email me it was like I don't know I'm not really 100% sure what's gonna happen in like scenario a or C so I took a system and set up I think it was 32 gigs of RAM and I set up a 24 gig ram disk and then 8 gigs of like actual ram and just had it run from the RAM disk because it seemed like it was mostly gonna be down to iOS but I am larger projects and like with the fir Onix test suite like compiling the Linux kernel I mean almost all of that is down to IO so like they'll run that from a ram disk or whatever and your kernel compile time and I mean modern system is like 30 seconds now ok yeah and it's just you know the kernel is maybe so that point are you are you if you use an SSD would you just be bottlenecking the processors on the SSD yeah well I mean to a point yeah but it depends on how much you're having to recompile to because like let's say I'm a I'm a device driver developer for the kernel so I'm gonna go in and I'm gonna work on my device driver which is relatively tiny I don't have to recompile the entire kernel every time even if it's like even if I want to build the module and I'm not even thinking about kernel modules at this point I want to actually build it into the kernel but even if I make a change it's not gonna have to recompile literally everything in the kernel it just has to recompile the part that I did and then just relink everything to get so all of that is gonna be cash hits all the way down okay and let's talk about victim cash for a moment - oh yeah so Andy uses I think notably in this point victim cash yeah so what that means is that stuff falls out of the l2 cache and then goes into the l3 cache so meaning that the anything that is a victim of eviction from the l2 cache ends up being in l3 so l3 really just sits between main memory and like whatever the processors doing so you end up in a situation where maybe if there was a more intelligent cache strategy you could get a better benefit from having the monster game cache or whatever but really it's just a list of everything that you've you know needed in in the recent past but when you're compiling you might have a library that you revisit or some part of the program that you revisit over and over again that is not spatially or temporally located anywhere by having the giant cache you might need something five or ten minutes later not really in five or ten minutes but a long time later in the process and most or all of it is still living in in the victim cache okay as opposed to a more intelligent caching strategy where it might look at what it's actually doing and say okay I'm gonna return to this lookup table a whole bunch for converting like this dictionary of symbols in this code into actual binary code and then like the caching strategy thing might say okay we're gonna use this a whole bunch let's just keep this in the l3 cache for now there's nothing that's that intelligent there so why does why does does the victim cache end up working to the benefit in this particular in let me ask the two versions of this question does victim cache work to the benefit of the CPUs in our compiles has the GCC won and does it work to the benefit of them in general and compiling I think in general it works pretty well in the GCC compiler test just because of the way that GCC is architected but like the Microsoft Visual C++ compiler does some more clever things with how it compiles and so I think the victim the victim cache ends up not working quite as well in those scenarios okay but on the AMD side you've got so much of it that it just it ends up not really matter and then you could make the argument that like the cache that you're looking at unlucky na minha RK because it tries to be a little bit more intelligent and because compiling is such an unusual workflow compared to like what you would normally be wanting to do in terms of executing a program that that cleverness on an Intel CPU is perhaps working against it okay so is there is there a thing I know the answer to this question but is there a perfect benchmark no give me something give me what you want that is right we can compile chromium let's see how fast it takes to compile chromium but my fear of that is that there were ultimately just reducing it to an IO like how many a ops can we do of how fast and you know read it in and eventually that becomes a ram benchmark yes but you run it a bunch of times and then dot the entire project lives in RAM at that point all right so does that even does that even really make sense but also like the whole optimization like the o2o3 things like I'm gonna optimize for memory footprint or I'm going to optimize your execution depending on what's going on inside the compiler that can also have vastly different performance characteristics so you get into some really interesting situations with the compiler in terms of like you tell the compiler to unroll loops and so in your source code you've got a loop where you're looping over a structure it's like the compiler can look at that and say well this structure is never going to be more than 64 autumn's big it will literally take your code remove it and just have the loop actually be just a linear list of instructions to go through all 64 things because that's faster for the CPU to execute just a linear list of instructions than to execute two or three or 100 or 1000 instructions and then jump back to so that is that behavior dictated by the compiler yes itself okay so you get into you gotta take compiler is you highlight benchmark yeah yeah okay okay I see the rabbit hole now yeah and I'm afraid of venturing yeah it's like oh look we compiled you know there's a microsoft's got a lot of open source stuff you can use the visual c++ compiler and and do that and intel you know for a long time intel was giving away their compiler and their compiler had a lot of really cool optimizations in it for their processors right that and perhaps less good optimizations for not their processor which may or may not have been intentional yeah or on top so you could end up with like this compiled took a very long time but this program actually runs better under this compiler and so some people misunderstand that when they're it's like I want to you know build a system for it's like it will the compiler will produce a deterministic result but you've got so many knobs and switches that you can tweak to control the output from the compiler that will have an effect on performance right so but at the end of the day if you are actually a developer working on a project like chromium most of the time you aren't even recompiling the entire project so it ends up just being how fast can we load this the stuff that we previously compiled from disk into memory link it and then kick the new thing out the door so is there do you have recommendations for this is where we started trying to do a compile benchmark we had a lot of people in the audience saying I work with code for a living I really want to know I don't have any benchmarks go do I won't know what CP is better so ignoring the part where we need to still figure out which how to test which CPU is better what is your recommendation for those people what do you what do you look up do you do you figure out what type of code you work on and then you look up a benchmark specifically for that we're yeah something it's it's really like more holistically workflow so maybe we could maybe we could put some projects together that are common types of projects kids and I wouldn't know this but I from your your email and the question in June I sort of started experimenting with this because let's say a web developer is using like nodejs and they're wanting to transpile their JavaScript into other JavaScript that like looked doing the performance counters without on Linux like I showed you the tabular alcoholic how it was doing all this crazy stuff the performance characteristic of that was wildly different than like compiling chromium and that was wildly different than like compiling GCC or even come halima Linux kernel Linux kernel actually this is fairly straightforward and then you go into the rabbit hole of like the whole Microsoft compiler so I think that because the I think we're in a situation park generally Andy's going to win because cash is so much larger in just about every scenario so and but more cores is not necessarily always better because some of those things are not multi-threaded processes not really but I think it's going to come down to finding a project that's like your project that somebody has actually benchmarked to look at it to say okay this is actually gonna this is actually gonna similar to my work yeah it's kind of like premiere because like you look at Adobe Premiere and it's like we want a benchmark premiere and it's like okay well that sounds reasonable but then it's like oh boy wait there's all these plugins and you know it's like are we benchmarking de luck thing or are we benchmarking encoder too and yeah yeah we got the red codec is like oh we you benchmark the red codec I need you to bench ProRes yeah it's like well it's the compiler thing I mean even though it's compiler things very similar in a lot of ways so it's it's it's it's there's not it there's not a silver bullet yeah there's something people are bringing up linking to me as well yeah I guess is that typically single threaded yes well I need to double check that but I think that in general it probably is a single thread of thing because you're usually bound to i/o you could go well I mean on an nvme you could go multi thread I'm not aware of a compiler that will are not aware of a linker that'll kick off and link a whole bunch of things but if you're again if you're in like developer mode and you're making a little bit of change to your tiny part of a project and recompiling the entire thing over and over again the linkers not gonna have to do much work in that scenario okay yeah so yeah I guess it sounds like then one of the approaches is kind of the shotgun approach that we take with some other types of benchmarks where you you pick a few different things and say here's the compiled benchmark suite here's like four different applications hopefully one of them fits your needs yeah and I think there's some so many user it sounds like what you're saying it sounds like there's a user responsibility to define your workload understand what you're doing right what problems are you trying to solve there a and then and then hopefully someone has a benchmark for you to solve that problem it might be that what you need is raid zero nvme that also sounds realistic based on your saying yeah yeah so I mean yeah you think about it it's like it really is true I mean with like the from the rise in 3600 the 3900 X might be a little less but that the just kiss of the cash differences but yeah I mean those those those processors are just and even like thread Ripper is kind of a monster for those kind of workloads background processes I think are also important to developers so that might be a thing that makes how responsive is the system while it's doing stuff in the background right well I have another question for you on this too so in your experience with code compiles are people typically compiling I'm not a programmer obviously arts people typically compiling bits and pieces of their code or do they do they do the whole projects all once I just talking about kind of like during the day it's usually like the first compile of the day will be like the one that takes the longest because it's recompiling a lot of things other than your code it depends on how you work so like if you work in a in a group where you've got like git or version control you might sit down at your machine and pull all of you all the team's work that they committed at the end of the day yesterday and then your first compile for the first thing that you change is going to compile all their stuff plus your stuff I see and so that one might take a while but then as you're banging on whatever it is that you've changed you're not really recompiling the whole project and in like a web development scenario who are you working on something like nodejs if you've got like a pretty good-sized set of changes you as a developer you might actually kick off some other background processes to do automated testing because like I automated testing is like a big thing so yeah like ways that take you seen account yeah yeah it'll run through and it'll do it's like we had this bug we're depending on what you're doing you know this whole other concept of like test-driven development where every time somebody creates a trouble ticket for your software you create a test to test for that problem so every time somebody says like okay there's a bug in the help screen of the program when you enter foo it crashes right so then you write a test that will interfere in the help screen and determine if the program crashes or not so if you're working in the help area you might while you're while you're still working you can say okay you know I've made a lot of changes let's test these and see if they actually work and you can kick those off those kid run on a build server so those could be like somewhere in your organization but increasingly they're running on your workstation okay because you can accidentally bring down the test infrastructure for the company if your codes really bad right yeah I've seen a few times I think I think the the biggest challenge is there's sort of two different mindsets looking at a looking for a benchmark of CPU performance for compiling which is our mindset is how do I produce something that's that's reliable that has the same results that test only one things not the compiler not the code quality just the CPU and then also how do we make it fair so that it represents both processors equally and it sounds like as is the case of premiere and some other instances it's not possible to do all those things yeah I don't think so so the only approach is I guess multiple benchmarks that are carefully planned yeah and and it is very difficult to come up with something that is not misleading if you don't understand how deep the rabbit-hole goes yes right on a lot of this stuff that's why when we first presented that data it was before we did any reviews for the GCC and we I almost I think I almost cut it from the initial publication because we had talked about it internally like this looks wrong it does not make sense and we retested it all and when Patrick retested it it produced the same results here we tested with mingw instead of cygwin and emesis too I think and I produced roughly the spoon is all how many people in the audience are actually recompiling mingw every day yeah well exactly and so we're not publishing it with a giant disclaimer saying this is the result for this specific thing and if you do this specific thing then hopefully this helps you and if not sorry give us your suggestions sometimes I think we've had it a couple of people in our audience and they're just like you have a video on blah blah and I talked to them and at the end of it really all they wanted was a video that they could send their boss to be like look I don't I think you giving me in a dual core i3 I need something better than there and I need a video that says that a dual core i3 is not going to cut it and it's like if that's what you need we'll glad to makes you that video right yeah that's easy well thank you for the education I really really hit it no no problem glad to be here this is a lot fun yeah absolutely and link below for level one tanks that is where he is from if you want to see more of Wendell's content highly recommended you'll learn a lot and hopefully we can maybe continue this discussion on compile testing in the future but in the four neckbeards in the comments yeah thank you for watching that we'll see you all next time the linker has been multi-threaded since before humans\n"