**The Complexity of Computer Security: Expert Opinions and System Analysis**

One of the significant challenges in computer security is understanding how secure a system is, particularly when dealing with complex systems that involve multiple stages and various types of attacks. To address this challenge, experts from different fields have been consulted to provide their opinions on the security of such systems.

In our research, we found that the internal and external experts as a whole give more or less the same answers. This was a concern for GCHQ, which wanted to ensure that they were selecting the right experts, but it turned out that there was a mixture of expertise among the selected individuals. We also explored whether different groups of experts would provide different answers, and found that some groups are narrowly focused on specific technical aspects, resulting in consistent opinions, while other groups have broader perspectives, offering a wider range of opinions.

For example, if we focus on firewalls, a group of experts may be highly knowledgeable about this topic and will give the same answer. However, when we consider broader topics or involve different groups with diverse expertise, we get a more varied range of answers. This suggests that it's not just the individual expert who matters, but also their context and the dynamics of the group they are part of.

**The Dynamics of Expert Groups**

One crucial aspect to consider is the dynamics of the expert group itself. Research has shown that groups with strong leaders may produce more homogeneous opinions, as everyone follows the lead of the dominant figure. In contrast, groups with a more free-thinking culture may yield a broader range of answers. Therefore, it's essential to take into account the social dynamics within an expert group when evaluating their opinions.

**Computer Science and System Analysis**

Computer science is crucial in addressing the complexity of computer security. While human experts can provide valuable insights, they have limitations in terms of time and information processing capacity. This is where computer science comes into play – by analyzing data from various sources, including system documentation, network diagrams, and expert opinions.

In our research, we have used various methods to gather data about the system's security, including asking experts for their opinions and analyzing the system itself. By combining these approaches, we can gain a more comprehensive understanding of the system's vulnerabilities and strengths.

**Measuring Security**

One of the significant challenges in measuring security is that it is a dynamic concept – what makes a system secure today may change tomorrow with new patches or updates. This means that our measures must be flexible and adaptable to account for these changes.

We have found that one method used by experts is to rate the most difficult stage (or "hub") of the system, with a score out of 10. By focusing on this most critical aspect, we can get a more accurate estimate of the overall security of the system. However, it's essential to note that this approach may not be sufficient for all systems, and other methods may need to be employed.

**Limitations and Opportunities**

While computer science provides a powerful toolset for analyzing and understanding system security, there are limitations to consider. For example, image processing techniques like JPEG handling may not effectively capture the complexity of certain types of data.

In contrast, statistics alone is often insufficient to address this challenge. By combining expertise with computational power and sophisticated algorithms, we can develop more robust methods for evaluating system security. This highlights the importance of interdisciplinary collaboration between experts from computer science and other fields.

**The Flower Example**

Interestingly, some images – like the flower example provided – have sharp changes in intensity that are difficult to capture using traditional image processing techniques like JPEG. In these cases, statistics alone may not be enough, and more advanced computational methods are needed to effectively analyze the data.

This illustrates how computer science can provide an advantage over traditional approaches when dealing with complex data sets. By leveraging statistical analysis combined with expertise from various fields, we can develop more robust models for understanding system security.

**Conclusion**

In conclusion, evaluating system security is a complex task that requires expertise in multiple areas, including computer science and human experts' opinions. By combining these approaches with advanced computational methods, we can gain a deeper understanding of system vulnerabilities and strengths. While there are limitations to consider, the potential benefits of this approach make it an attractive solution for addressing the challenges of system security.

Moreover, our research highlights the importance of considering various aspects when evaluating system security, including the dynamics of expert groups and the role of statistics in data analysis. By embracing interdisciplinary collaboration and leveraging computational power, we can develop more robust models for understanding system security, ultimately leading to improved cybersecurity measures.

"WEBVTTKind: captionsLanguage: ensome of the work that we do is related to security uh to computer security in particular um there is a government agency called gchq in the uk which stands for government communications headquarters so there's sort of super authority in the uk that looks after all things to do with security including computer security and they've come to us a little while ago with a really interesting question basically they said we're looking at a lot of computer systems and is there any way we can measure how secure they really are and how do we understand what our experts are saying data mining is all about crunching large volumes of data but then you come to these points where suddenly there's all sorts of data there but it relies on human expertise and it's not so clear anymore what exactly you're looking for and here it's a classic example so you get your experts to look at a computer system and this is actually some real data this is the unclassified version i don't have to shoot you when you have to go you're okay this is an example where actually lots of experts from gchq have been asked to assess the security of a system the details i can't reveal but there was a particular system that was set up there was different ways of attacking the system and the experts were basically asked okay tell me this particular type of attack on a scale of 1 to 10 do you think it's easy or difficult and this is the answers we got and each dot is a particular expert the colors just relates to what part of cheese head shoe they're working in there might be in slightly different subgroups and you can see some of the experts they said oh yeah this is a really easy attack to do they gave it rank number one but some of the experts actually said oh it's kind of in the middle and some experts said no no it's really difficult this is number 10. and this is typical for when you're asking experts i mean these are highly qualified people and the same is true in medicine and this is kind of where our work started with all this and why gchq came to us we worked a lot with doctors doctors assessed patients and why do you have to ask a second in a third opinion it's because they don't always give you the same answer do they so the same here you ask your different experts and they can give you very very different answers it doesn't become a data mining problem anymore now because well what are you going to do with this because take the average but the average is probably completely wrong because what's really happening here is one of two things one either this expert who is sort of a bit of an outlier he's untrained and he actually just needs a bit of training or maybe he just become aware of a zero-day attack somebody nobody else knows yet and he really knows his stuff whereas these people here that might work on this day today and they're probably right in a general sense but maybe they've missed this latest attack so if you go for the average you probably end up if the completely wrong answer because either it's trivial or there's something that's just happened recently and we need to find out but you can't just take the average when we work with gchq that's kind of what it looks like it's a big doughnut actually so they're basically saying look lots of government agencies are coming to us and they say we want we want our system to be assessed by your experts how secure is it should we spend more on security and well okay so how many experts should we ask is there any does it matter whether we ask internal or external experts does it matter whether we ask experts of different teams and what do we do when we get five opinions do we just take the average so through our studies and through our collections of data we were able to come up with a number of findings which were quite interesting one of the things that we looked at for example was whether experts which are internal to the organization and experts which are external to your organizations give different answers maybe they had different training maybe they have different years of experience and one of the things we found was that actually the internal and the external experts as a whole they give more or less the same answers so that was good because there was some concern on gchq's part that they need to be very careful about who they select but turns out no it's a mixture it's good another thing that we looked at was different groups of experts do they give different answers it turns out that some groups they're obviously very narrowly technically focused and they will always give you more or less the same answer whereas other groups that seem to be broader and they give you a wider variety of answers so while it's maybe good idea to ask five experts it's not a good idea to ask five experts from some group because it doesn't matter it's just wasting your time you're getting five times the same answer so you might wanna ask other five people from these other groups or five people from different groups so what's an example of that is it just that perhaps they're specifically studying networks yeah so somebody might some of these groups they might be really really focused on firewalls so they know everything about firewalls so when when you ask them about firewalls all of them going to say the same thing but as other groups well they've studied more broader things also it can just be to do with the dynamics of the group maybe there's a very strong leader in some groups so everybody will give you the opinion of the leader whereas in other groups it's a bit more free thinking what we're really after is computer security and understanding how secure is the system ideally what we want is a measure of security and to obtain such a measure this is a computer computer science issue we get data we get data by asking experts there's also data by not asking experts we have data about the system i mean look for example uh this is kind of what a system looks like here's a network this is how many hops there are there's a firewall here there's an encryption machine there there's a content checker here you've got all this information as well so you've got all this information and you've got the expert opinions now how do you put it all together to give you a number or something that tells you okay actually this is how secure we are or we should spend more money on this or we don't need to is it possible to do that i have a bicycle lock and uh it's rated you know as to how secure it is i'm guessing that's based on how thick it is how heavy duty the metal is but things change in the computer the problem is with this the system might be this secure today and then tomorrow something happens and the patch a new patch comes out or some update comes out and things change so what you're actually going to get is something more dynamic here so you have to ask a question again and again but at least at this moment in time when you ask your five experts you're going to get a consistent answer or at least you know how to get a consistent answer out of their opinions one of the things that we found about this which is very interesting and here's a publication where you can read much more about this because these computer systems are very complicated things many hops many stages many different ways of attacking them this is where humans we have a problem we can't have all the information in our head all the time so one of the things that we're interested in is how do experts actually form their opinion of this and it turns out that what experts really do is they look at the whole system and they identify which one of the many steps is the most difficult step they call it the most difficult hop and once they've identified the most difficult hop they then think very hard okay how difficult is this most difficult hop for example the encryption machine might be the most difficult of breaking the encryption how difficult is that and they give that a rating between one and ten and their final rating for the whole overall system is 95 of the rating they gave this most difficult hub which shows you how we as humans you know do our data mining really simple so that's that's basically so that you we may as well we may as well use that same technique that's right there's basically one way you can do it i mean if you if you're happy with a 95 percent answer which you might be happy on most systems then actually it's sufficient to ask your experts which one is the most difficult part and how difficult is this part one thing that this image has that our last image of the flower didn't have is sharp changes in intensity so this c has a sharp step down into the background and that is not something that jpeg handles very well at all but statistics on it on its own it's not enough and that's where computer programming is really good because it's more flexible than statisticssome of the work that we do is related to security uh to computer security in particular um there is a government agency called gchq in the uk which stands for government communications headquarters so there's sort of super authority in the uk that looks after all things to do with security including computer security and they've come to us a little while ago with a really interesting question basically they said we're looking at a lot of computer systems and is there any way we can measure how secure they really are and how do we understand what our experts are saying data mining is all about crunching large volumes of data but then you come to these points where suddenly there's all sorts of data there but it relies on human expertise and it's not so clear anymore what exactly you're looking for and here it's a classic example so you get your experts to look at a computer system and this is actually some real data this is the unclassified version i don't have to shoot you when you have to go you're okay this is an example where actually lots of experts from gchq have been asked to assess the security of a system the details i can't reveal but there was a particular system that was set up there was different ways of attacking the system and the experts were basically asked okay tell me this particular type of attack on a scale of 1 to 10 do you think it's easy or difficult and this is the answers we got and each dot is a particular expert the colors just relates to what part of cheese head shoe they're working in there might be in slightly different subgroups and you can see some of the experts they said oh yeah this is a really easy attack to do they gave it rank number one but some of the experts actually said oh it's kind of in the middle and some experts said no no it's really difficult this is number 10. and this is typical for when you're asking experts i mean these are highly qualified people and the same is true in medicine and this is kind of where our work started with all this and why gchq came to us we worked a lot with doctors doctors assessed patients and why do you have to ask a second in a third opinion it's because they don't always give you the same answer do they so the same here you ask your different experts and they can give you very very different answers it doesn't become a data mining problem anymore now because well what are you going to do with this because take the average but the average is probably completely wrong because what's really happening here is one of two things one either this expert who is sort of a bit of an outlier he's untrained and he actually just needs a bit of training or maybe he just become aware of a zero-day attack somebody nobody else knows yet and he really knows his stuff whereas these people here that might work on this day today and they're probably right in a general sense but maybe they've missed this latest attack so if you go for the average you probably end up if the completely wrong answer because either it's trivial or there's something that's just happened recently and we need to find out but you can't just take the average when we work with gchq that's kind of what it looks like it's a big doughnut actually so they're basically saying look lots of government agencies are coming to us and they say we want we want our system to be assessed by your experts how secure is it should we spend more on security and well okay so how many experts should we ask is there any does it matter whether we ask internal or external experts does it matter whether we ask experts of different teams and what do we do when we get five opinions do we just take the average so through our studies and through our collections of data we were able to come up with a number of findings which were quite interesting one of the things that we looked at for example was whether experts which are internal to the organization and experts which are external to your organizations give different answers maybe they had different training maybe they have different years of experience and one of the things we found was that actually the internal and the external experts as a whole they give more or less the same answers so that was good because there was some concern on gchq's part that they need to be very careful about who they select but turns out no it's a mixture it's good another thing that we looked at was different groups of experts do they give different answers it turns out that some groups they're obviously very narrowly technically focused and they will always give you more or less the same answer whereas other groups that seem to be broader and they give you a wider variety of answers so while it's maybe good idea to ask five experts it's not a good idea to ask five experts from some group because it doesn't matter it's just wasting your time you're getting five times the same answer so you might wanna ask other five people from these other groups or five people from different groups so what's an example of that is it just that perhaps they're specifically studying networks yeah so somebody might some of these groups they might be really really focused on firewalls so they know everything about firewalls so when when you ask them about firewalls all of them going to say the same thing but as other groups well they've studied more broader things also it can just be to do with the dynamics of the group maybe there's a very strong leader in some groups so everybody will give you the opinion of the leader whereas in other groups it's a bit more free thinking what we're really after is computer security and understanding how secure is the system ideally what we want is a measure of security and to obtain such a measure this is a computer computer science issue we get data we get data by asking experts there's also data by not asking experts we have data about the system i mean look for example uh this is kind of what a system looks like here's a network this is how many hops there are there's a firewall here there's an encryption machine there there's a content checker here you've got all this information as well so you've got all this information and you've got the expert opinions now how do you put it all together to give you a number or something that tells you okay actually this is how secure we are or we should spend more money on this or we don't need to is it possible to do that i have a bicycle lock and uh it's rated you know as to how secure it is i'm guessing that's based on how thick it is how heavy duty the metal is but things change in the computer the problem is with this the system might be this secure today and then tomorrow something happens and the patch a new patch comes out or some update comes out and things change so what you're actually going to get is something more dynamic here so you have to ask a question again and again but at least at this moment in time when you ask your five experts you're going to get a consistent answer or at least you know how to get a consistent answer out of their opinions one of the things that we found about this which is very interesting and here's a publication where you can read much more about this because these computer systems are very complicated things many hops many stages many different ways of attacking them this is where humans we have a problem we can't have all the information in our head all the time so one of the things that we're interested in is how do experts actually form their opinion of this and it turns out that what experts really do is they look at the whole system and they identify which one of the many steps is the most difficult step they call it the most difficult hop and once they've identified the most difficult hop they then think very hard okay how difficult is this most difficult hop for example the encryption machine might be the most difficult of breaking the encryption how difficult is that and they give that a rating between one and ten and their final rating for the whole overall system is 95 of the rating they gave this most difficult hub which shows you how we as humans you know do our data mining really simple so that's that's basically so that you we may as well we may as well use that same technique that's right there's basically one way you can do it i mean if you if you're happy with a 95 percent answer which you might be happy on most systems then actually it's sufficient to ask your experts which one is the most difficult part and how difficult is this part one thing that this image has that our last image of the flower didn't have is sharp changes in intensity so this c has a sharp step down into the background and that is not something that jpeg handles very well at all but statistics on it on its own it's not enough and that's where computer programming is really good because it's more flexible than statistics\n"