The Importance of Facial Temperature Monitoring in Helicopter Pilots

In recent years, researchers have been exploring the use of facial temperature monitoring as a non-invasive method to assess the physical and emotional state of individuals, particularly pilots. The project involved studying the thermal patterns on the faces of helicopter pilots during various tasks, with the aim of understanding how their bodies respond to different levels of demand.

One of the key findings of the study was that the nose tip is the most evident point in terms of temperature changes. As the task demand increases, the nose tip tends to warm up, and as the demand decreases, it cools down. This phenomenon is not unique to pilots and can be observed in other individuals who experience stress or fatigue.

The study also showed that the side areas of the nose exhibit a similar response to the temperature changes, although not as pronounced as the nose tip. The perceived demand was measured using a color map, which allowed researchers to visualize the thermal patterns on the face. By analyzing these patterns, they were able to determine the level of stress or fatigue experienced by the pilot.

The study revealed three levels of task difficulty, with the demand peaking in the middle level. This is consistent with the concept of "threshold theory," which suggests that there are specific thresholds beyond which performance declines significantly. The temperature changes observed on the face were used to validate this theory and provide evidence for its application in real-world scenarios.

The researchers developed an algorithm that could process the thermal data from the camera feed and estimate the temperature distribution on the face. This algorithm was capable of processing the data in almost real-time, with a frame rate of 7.5 frames per second. The video output was not compressed to preserve the high level of detail required for accurate temperature estimation.

The study also highlighted the importance of calibration curves in converting the digital camera feed into actual temperatures. By using a color map, researchers were able to convert the thermal data into a visual representation that could be used to extract specific features, such as the center of the pupil or the position of landmarks on the face.

To verify the accuracy of the algorithm and ensure consistent results, researchers compared the temperature estimates with the original matrix of temperatures. This allowed them to validate the algorithm's performance and identify potential sources of error.

In addition to the technical aspects of facial temperature monitoring, the study also explored its practical applications in real-world scenarios. The research team demonstrated that the algorithm could be used to track the features of interest on the face, such as the pupil or landmarks, even in situations where the camera is moving or the lighting conditions are changing.

The study's findings have significant implications for the field of human performance and safety, particularly in high-stress environments like helicopter piloting. By developing a non-invasive method to assess pilot stress levels, researchers can identify individuals who may be at risk of fatigue or decreased performance, allowing them to take corrective action before it's too late.

The study also has broader implications for the development of wearable technology and sensor-based systems that can monitor human physiological responses in real-time. By understanding how the body responds to different levels of demand, researchers can design more effective and efficient monitoring systems that can be used in a variety of applications, from athletic performance to medical diagnosis.

In conclusion, the study demonstrates the potential of facial temperature monitoring as a tool for assessing pilot stress levels and detecting changes in physical and emotional state. By developing an algorithm that can accurately estimate temperatures on the face, researchers have taken a significant step towards improving our understanding of human performance and safety in high-stress environments.

"WEBVTTKind: captionsLanguage: enwe have been looking at physiological monitoring to try to infer how hard people are working on a given task what we've been trying to do is find the least intrusive means for collecting physiological data we were looking at the aviation domain but it can obviously be extended to a lot of other a lot of other areas so initially we had a lab study which looked at a very simple computer task where people were tracking balls and shooting them and we could quite easily varied demand by increasing and decreasing the number of balls in a given pattern and then we could see if the physiological measures reflected that that pattern and correlated well with it so um that was a lab study to see how feasible this is and then we went on to doing a helicopter simulator study one of the questions that we had was would highly trained individuals and helicopter pilots reacts in much the same way as an usual person does you know to increase level of demand we did a computer file on something that sounds similar where horia was using a brain scanner and turning the lights red when people were practicing things like air traffic control is this a similar kind of idea yes the idea behind it is basically the same the efner sensor what i was using is maybe more accurate in certain scenarios and reflects more of the brain activity we are aiming to use thermal imaging as a way to make it less intrusive so people don't have to wear any sort of equipment so this small flirt thermal camera the resolution is quite small it's 640x480 so it's not an amazing resolution there are high resolution cameras nowadays but still not up to the level of phone camera what it does it's picking up thermal radiation this should be within the range of 7.5 to i think 13 micrometers and it's converting it to temperature so obviously that's one step of the way the other the next step was extracting temperature from various areas of the face i'm hiding behind the camera here this is the scale here on the right blue is cold black is even colder and then as you go towards red yellow and white that symbolizes a higher temperature yeah when we extracted the data we used these sort of images to find facial features those facial features allowed us to get data from various areas we covered quite a large area of the face whereas previous study looked at smaller areas like the nose forehead and indeed actually it most of the changes occur there on on the nose and around the nose area people have reported it before we try to do a more structured study where we could control the demand in a more accurate way so what we've seen that usually in most people but not in all of them those temperatures tend to drop when they are engaged in a high demand task and actually they do tend to go back up as the task demand diminishes the nose tip is the most evident point but also the side areas of the nose show a similar response just not as large here in grey you can see the perceived demand so there are three levels of the task towards the middle of each of the levels the demand peaks so things get more difficult yeah things get more difficult and at the same time you can see here in green for example which is temperature in point p which was the tip of the nose you can see how that drops and then as the task becomes easier again it starts going back up this here in blue is point v which was right below the nose so here you can see also drop maybe it's not as high but still you can see sort of the same pattern we got a similar pattern in most people and in a lot of the helicopter pilots some other people do have naturally colder noses they're just cold by default so then there won't be any meaningful drop in temperature uh that's yeah one of the one of the other effects but yeah we're still exploring to see if for those cases there are other signs indicating to the increase in demand is this something that you can do in real time or how does it work we can actually do it in almost real time with the algorithm i used which i have to admit is not highly optimized but you could do it maybe one frame per second the camera itself does not output a very high frame rate it's around 7.5 frames per second it's actually limited uh you cannot buy it at higher frame rates and yeah then the videos are quite large because it's the data is not compressed as you would get in a normal visual video so it could end up being within the range of 30 gb for for for one of the videos and then yeah there's it's quite a lot of data to process initially the video is outputted in a digital level that the camera records which is then converted to actual temperatures by using a calibration curve and then we used that temperature basically matrix to convert it to a visual rgb using a color map which obviously you can change the format coming out of the camera is is digital so it's it's the temperatures and then you convert that to video do you ever go back from the video the visual video back to the original to get the temperatures uh yes yes i do that at the end so once you get an area of interest and you need to extract data from it then you go to the original matrix of temperatures and you get the temperature area from that in between those specific coordinates we're used to seeing a pixel as being three or four numbers rgb alpha whatever so instead of having that you've got a temperature in kelvin or something yeah you just have a temperature well i used to convert it in celsius but yeah you can convert it in any unit based on that color map we did the tracking of features we had various approaches to it the first approach from the paper that was published relied on cascade classifiers to detect the eye area so it was trained to to pick up eye patterns and to find the eyes and then we would rely on another property of the thermal image the pupils are colder than the surroundings so if you look at this rectangle which is zoomed in here this is in actual thermal data so whereas here you see the rgb picture the height of this represents temperature so you could see here how temperature drops across the pupil so this is basically the center of the pupil the low temperature there is most likely because there's no blood vessels passing through the pupil then it could quite accurately find the center of the pupil and then going from there we found the rest of the landmarks for the helicopter study using this smaller camera we used a different approach which actually didn't belong to us we retrained an algorithm that was previously published and it's more robust to poster changes and it works better and faster we retrained the algorithm to pick up these specific points on the on the face it does fail at points in extreme angles but then we filtered out those intervals as you can see as we go through the video the nose gets colder this pilot was going through a very very difficult scenario where he was performing an auto rotation which means that he had no more engine power and he was just going down and trying to cushion the landing just by using the propellers that were not spinning at the moment here as you can see they crashed and you can see the nose being really really cold he looked like he laughed then yeah he laughed yeah well luckily it was a simulator you can laugh after can you remind me what pethagoro's theorem let's is the triangles the lights are about five seconds behind what um max is going through this much it takes from the heart to pump up the blood uh 32-bit images that is um four channels per pixel is very common even when we're not usedwe have been looking at physiological monitoring to try to infer how hard people are working on a given task what we've been trying to do is find the least intrusive means for collecting physiological data we were looking at the aviation domain but it can obviously be extended to a lot of other a lot of other areas so initially we had a lab study which looked at a very simple computer task where people were tracking balls and shooting them and we could quite easily varied demand by increasing and decreasing the number of balls in a given pattern and then we could see if the physiological measures reflected that that pattern and correlated well with it so um that was a lab study to see how feasible this is and then we went on to doing a helicopter simulator study one of the questions that we had was would highly trained individuals and helicopter pilots reacts in much the same way as an usual person does you know to increase level of demand we did a computer file on something that sounds similar where horia was using a brain scanner and turning the lights red when people were practicing things like air traffic control is this a similar kind of idea yes the idea behind it is basically the same the efner sensor what i was using is maybe more accurate in certain scenarios and reflects more of the brain activity we are aiming to use thermal imaging as a way to make it less intrusive so people don't have to wear any sort of equipment so this small flirt thermal camera the resolution is quite small it's 640x480 so it's not an amazing resolution there are high resolution cameras nowadays but still not up to the level of phone camera what it does it's picking up thermal radiation this should be within the range of 7.5 to i think 13 micrometers and it's converting it to temperature so obviously that's one step of the way the other the next step was extracting temperature from various areas of the face i'm hiding behind the camera here this is the scale here on the right blue is cold black is even colder and then as you go towards red yellow and white that symbolizes a higher temperature yeah when we extracted the data we used these sort of images to find facial features those facial features allowed us to get data from various areas we covered quite a large area of the face whereas previous study looked at smaller areas like the nose forehead and indeed actually it most of the changes occur there on on the nose and around the nose area people have reported it before we try to do a more structured study where we could control the demand in a more accurate way so what we've seen that usually in most people but not in all of them those temperatures tend to drop when they are engaged in a high demand task and actually they do tend to go back up as the task demand diminishes the nose tip is the most evident point but also the side areas of the nose show a similar response just not as large here in grey you can see the perceived demand so there are three levels of the task towards the middle of each of the levels the demand peaks so things get more difficult yeah things get more difficult and at the same time you can see here in green for example which is temperature in point p which was the tip of the nose you can see how that drops and then as the task becomes easier again it starts going back up this here in blue is point v which was right below the nose so here you can see also drop maybe it's not as high but still you can see sort of the same pattern we got a similar pattern in most people and in a lot of the helicopter pilots some other people do have naturally colder noses they're just cold by default so then there won't be any meaningful drop in temperature uh that's yeah one of the one of the other effects but yeah we're still exploring to see if for those cases there are other signs indicating to the increase in demand is this something that you can do in real time or how does it work we can actually do it in almost real time with the algorithm i used which i have to admit is not highly optimized but you could do it maybe one frame per second the camera itself does not output a very high frame rate it's around 7.5 frames per second it's actually limited uh you cannot buy it at higher frame rates and yeah then the videos are quite large because it's the data is not compressed as you would get in a normal visual video so it could end up being within the range of 30 gb for for for one of the videos and then yeah there's it's quite a lot of data to process initially the video is outputted in a digital level that the camera records which is then converted to actual temperatures by using a calibration curve and then we used that temperature basically matrix to convert it to a visual rgb using a color map which obviously you can change the format coming out of the camera is is digital so it's it's the temperatures and then you convert that to video do you ever go back from the video the visual video back to the original to get the temperatures uh yes yes i do that at the end so once you get an area of interest and you need to extract data from it then you go to the original matrix of temperatures and you get the temperature area from that in between those specific coordinates we're used to seeing a pixel as being three or four numbers rgb alpha whatever so instead of having that you've got a temperature in kelvin or something yeah you just have a temperature well i used to convert it in celsius but yeah you can convert it in any unit based on that color map we did the tracking of features we had various approaches to it the first approach from the paper that was published relied on cascade classifiers to detect the eye area so it was trained to to pick up eye patterns and to find the eyes and then we would rely on another property of the thermal image the pupils are colder than the surroundings so if you look at this rectangle which is zoomed in here this is in actual thermal data so whereas here you see the rgb picture the height of this represents temperature so you could see here how temperature drops across the pupil so this is basically the center of the pupil the low temperature there is most likely because there's no blood vessels passing through the pupil then it could quite accurately find the center of the pupil and then going from there we found the rest of the landmarks for the helicopter study using this smaller camera we used a different approach which actually didn't belong to us we retrained an algorithm that was previously published and it's more robust to poster changes and it works better and faster we retrained the algorithm to pick up these specific points on the on the face it does fail at points in extreme angles but then we filtered out those intervals as you can see as we go through the video the nose gets colder this pilot was going through a very very difficult scenario where he was performing an auto rotation which means that he had no more engine power and he was just going down and trying to cushion the landing just by using the propellers that were not spinning at the moment here as you can see they crashed and you can see the nose being really really cold he looked like he laughed then yeah he laughed yeah well luckily it was a simulator you can laugh after can you remind me what pethagoro's theorem let's is the triangles the lights are about five seconds behind what um max is going through this much it takes from the heart to pump up the blood uh 32-bit images that is um four channels per pixel is very common even when we're not used\n"