"WEBVTTKind: captionsLanguage: enlast week i was honored to be invited by apple to be a guest at the iphone 14 launch i can't really describe how much this meant to me i remember watching steve jobs announce the very first iphone and back when i worked at the museum my co-workers and i would arrange our entire lunch hour once a year in september just to see what apple was coming up with next it was such an honor to be invited out for this event after the keynote presentation and press event apple arranged for me to get to go inside apple park to demo the new phone with worldwide iphone product line marketing manager francesca sweet and also vice president of camera software engineering john mccormack it's extremely rare to be invited to film inside the ring so to speak and so i found it only fitting to film this entire video on iphone it was such an honor to be able to demo this new camera system and sit down and talk to francesca and john about the technical aspects of iphone 14. so we have an all new photonic engine which is a reimagined image pipeline and what we were able to do is bring deep fusion earlier in the process and so in order to do that we had to rethink from the ground up and this is this is john's work and you know we've looked at uncompressed images and being able to process pixel by pixel do deep fusion earlier which means we're able to retain more detail all of the information so when we take it through the image pipeline we are able to increase our light performance by by 2x and even more on some cameras this obsession with light that is the core of the photonic engine because our inspiration for photonic is to be able to get the absolute most out of every single camera in any different lighting condition so let me give a couple of examples this year we really wanted to get into that mid and lower light photography and push that as far as we could and to do that we wanted to get the most information off the sensor we could and so typically or traditionally our pipeline into the fusion has been a 12-bit pipeline this year we've moved that to be a 16-bit full-color pipeline and so now we are pushing way more information way more data way more color it's a big leap into deep fusion it is i mean you can do the math and it's a big number the amount of kind of like the amount of more information is but this gives diffusion so much more to operate on and and this is especially important as the light gets lower and as because as light gets slower colors kind of like smush together and so this gives us more information to pull those colors apart so now deep fusion can faithfully recreate exactly what the iphone saw exactly what the human eye saw or moving to the 48 as we looked at building the 48 we realized that the traditional method of the step called demosaicing which is taking a bunch of raw data from the camera and turning it into something that starts to look like an image the traditional d mosaic methods just weren't going to work they weren't going to be good enough and so we built a whole new machine learning based block that is at the very first stage of the pipeline and we can do this because of the power of the a16 bionic we've got this supercomputer that takes this 48 megapixel image gobbles it up and puts out this extraordinary result and uh we've been super happy with how far we've been able to get and now with all of this extra color detail and with all this extra image fidelity our back end processing our somatic developing has taken a big leap forward this year as well so now our skin tone skin texture color rendering so much higher fidelity this year so it's a really big year for iphone photography it's a really big year for photography in general because back to your point the throughout the history of photography there have been these big leaps and i think computational photography is another one of those really big leaps that we don't get to sit on the sidelines of we're right in the middle of it i love that well i think that a lot of people don't think yeah and nobody would have a reason to think if you're not a photographer who thinks about images all the time like you know when you're talking about low light in francesca you were showing me some images yesterday and we were talking about how you get those low mid tones before they start going into shadows and you might have this beautiful orange lighter skin tones are a great example and it's not one color it's thousands of colors that your eye is processing and to get what's essentially a monochrome sensor to render that out and split it into color is a big challenge now you guys went with a quad bear filter on the on the 48 this year we did which is probably going to get into a very technical explanation but that has a lot to do with with the the enhanced quality on that i'm sure well what it gives us is it gives us so much more resolution but it also comes with a set of challenges because the the traditional bayer filter has been around for a long time and it's a very well understood thing now with the new sensor pattern on the 48 it really is like something that landed from another planet and so we went and tried to use the traditional demosaicing techniques and it gave us an okay level of quality but we really weren't able to go and pick out those fine details as you move from one shade of orange to a slightly different shade of orange which is the thing that gives you that beautiful kind of rendering that differentiation and fine color and this is where we went back and said let's look at solving this another way what resources do we have how can we go use our investment in the photonic engine to be able to just go and solve this problem and so then we had a bunch of really smart mathematicians go spend a long time and they came out with something fantastic that's amazing so i mean this is really the benefit of iphone right we have incredible sensors incredible lenses we've pushed the limits of what we can do we are introducing these really pro level features like 48 megapixel sensors doing this intelligent computation that allows it to take the full advantage of the light that's coming in and then we do all of this intelligent processing in the background and so what the user sees is just a gorgeous image at the end of the day that's what's important that's what's important well the other cool thing that i love is the way you guys handled the 48 megapixel sensor because there's been other companies that do it so for those who are not familiar i mean kind of the standard with a phone is a 12 megapixel resolution a lot of people that's that's a lot of resolution actually if you just go back a couple of years a dslr with 12 megapixels was considered tired uh and so introducing a 48 megapixel sensor you have two outputs with this the standard output you can go down to 12 megapixels which gives you the ability to crop that sensor well that's and you get a fourth lens yes and it is it's like it's like we took iphone and put a fourth lens on the back of it because what we do is we take that middle 12 megapixels which is out of the middle of the sensor yeah and that's a 2x optical so it's not digitally enhanced it's a 2x optical and so now it is like it's got a fourth lens but the other thing this lets us do is bring back that 2x portrait mode and so now folks can use that beautiful depth of field rendering on that storied 50 mil focal length so you actually get the depth of field but and but then in portrait mode as well we can do the computational work on top of it that's amazing yeah it's amazing so what are you most excited about with the with iphone 14 now i'm most excited about action mode and then you'll see at the top left there's uh ah the new action icon the running man the running man and then turn him on and then you're ready to go and so now then i'm going to do this crazy thing of i'm going to try and follow them and as you see i'm running around and there is no way that would be usable footage so the key though is is don't don't you just shoot you just shoot you don't worry about it all right so now you look at the footage it comes out and it's almost like i'm a real sports videographer it's just i don't understand how that happened and i'm one of the engineers who built this thing it just feels like a map so not just that but like you're not there's not a lot of action coming off no offense john but i can be action and do my crazy shots where i run around yeah hold it with don't even worry about it huh all right yeah we can and then we'll start the uh russian museum you can dunk one right franchise yeah look you got your tennis shoes are you ready living at my middle floor love it and just follow with traditional image stabilization it's all about kind of dealing with like the minor handshakes and so like which is the thing that you had kind of would really what that replaced was those kind of like small gimbals that we all used to carry around and now those are basically a thing in the past computational photography computational videography replace the gimbal and so our next challenge was if you look at sort of folks on the sidelines of sporting events they've got these giant back mounted rigs that they're kind of like hanging cameras off and so our question was like well can we get rid of things like that too because to your point about folks with little kids that is sports photography so like you know but you know mom and dad aren't going to go go and pick up some giant steady camera the best to chase the toggle around and so our question was well can they just do that with the iphone and so what we do is because we've got such a large sensor with 4k and so much processing power we can say well let's shoot this in 4k so we've got this giant overscan region that we can then stabilize and as we do that then you get to basically focus on living your life chasing your toddler through the room uh watching your friends play basketball but doing it from the middle of the basketball court and like sort of following them around and not having to think about like oh this is going to look like unstabilized garbage because we take care of that for you yeah it's pretty cool you have something that's useful in the end you guys thank you so much this is such an honor to be on the apple campus i've never been here before and here we are welcome an actual orchard yeah there we go and you can't see it in this shop but we're in the center of the the bowl here it's yeah this is a real treatlast week i was honored to be invited by apple to be a guest at the iphone 14 launch i can't really describe how much this meant to me i remember watching steve jobs announce the very first iphone and back when i worked at the museum my co-workers and i would arrange our entire lunch hour once a year in september just to see what apple was coming up with next it was such an honor to be invited out for this event after the keynote presentation and press event apple arranged for me to get to go inside apple park to demo the new phone with worldwide iphone product line marketing manager francesca sweet and also vice president of camera software engineering john mccormack it's extremely rare to be invited to film inside the ring so to speak and so i found it only fitting to film this entire video on iphone it was such an honor to be able to demo this new camera system and sit down and talk to francesca and john about the technical aspects of iphone 14. so we have an all new photonic engine which is a reimagined image pipeline and what we were able to do is bring deep fusion earlier in the process and so in order to do that we had to rethink from the ground up and this is this is john's work and you know we've looked at uncompressed images and being able to process pixel by pixel do deep fusion earlier which means we're able to retain more detail all of the information so when we take it through the image pipeline we are able to increase our light performance by by 2x and even more on some cameras this obsession with light that is the core of the photonic engine because our inspiration for photonic is to be able to get the absolute most out of every single camera in any different lighting condition so let me give a couple of examples this year we really wanted to get into that mid and lower light photography and push that as far as we could and to do that we wanted to get the most information off the sensor we could and so typically or traditionally our pipeline into the fusion has been a 12-bit pipeline this year we've moved that to be a 16-bit full-color pipeline and so now we are pushing way more information way more data way more color it's a big leap into deep fusion it is i mean you can do the math and it's a big number the amount of kind of like the amount of more information is but this gives diffusion so much more to operate on and and this is especially important as the light gets lower and as because as light gets slower colors kind of like smush together and so this gives us more information to pull those colors apart so now deep fusion can faithfully recreate exactly what the iphone saw exactly what the human eye saw or moving to the 48 as we looked at building the 48 we realized that the traditional method of the step called demosaicing which is taking a bunch of raw data from the camera and turning it into something that starts to look like an image the traditional d mosaic methods just weren't going to work they weren't going to be good enough and so we built a whole new machine learning based block that is at the very first stage of the pipeline and we can do this because of the power of the a16 bionic we've got this supercomputer that takes this 48 megapixel image gobbles it up and puts out this extraordinary result and uh we've been super happy with how far we've been able to get and now with all of this extra color detail and with all this extra image fidelity our back end processing our somatic developing has taken a big leap forward this year as well so now our skin tone skin texture color rendering so much higher fidelity this year so it's a really big year for iphone photography it's a really big year for photography in general because back to your point the throughout the history of photography there have been these big leaps and i think computational photography is another one of those really big leaps that we don't get to sit on the sidelines of we're right in the middle of it i love that well i think that a lot of people don't think yeah and nobody would have a reason to think if you're not a photographer who thinks about images all the time like you know when you're talking about low light in francesca you were showing me some images yesterday and we were talking about how you get those low mid tones before they start going into shadows and you might have this beautiful orange lighter skin tones are a great example and it's not one color it's thousands of colors that your eye is processing and to get what's essentially a monochrome sensor to render that out and split it into color is a big challenge now you guys went with a quad bear filter on the on the 48 this year we did which is probably going to get into a very technical explanation but that has a lot to do with with the the enhanced quality on that i'm sure well what it gives us is it gives us so much more resolution but it also comes with a set of challenges because the the traditional bayer filter has been around for a long time and it's a very well understood thing now with the new sensor pattern on the 48 it really is like something that landed from another planet and so we went and tried to use the traditional demosaicing techniques and it gave us an okay level of quality but we really weren't able to go and pick out those fine details as you move from one shade of orange to a slightly different shade of orange which is the thing that gives you that beautiful kind of rendering that differentiation and fine color and this is where we went back and said let's look at solving this another way what resources do we have how can we go use our investment in the photonic engine to be able to just go and solve this problem and so then we had a bunch of really smart mathematicians go spend a long time and they came out with something fantastic that's amazing so i mean this is really the benefit of iphone right we have incredible sensors incredible lenses we've pushed the limits of what we can do we are introducing these really pro level features like 48 megapixel sensors doing this intelligent computation that allows it to take the full advantage of the light that's coming in and then we do all of this intelligent processing in the background and so what the user sees is just a gorgeous image at the end of the day that's what's important that's what's important well the other cool thing that i love is the way you guys handled the 48 megapixel sensor because there's been other companies that do it so for those who are not familiar i mean kind of the standard with a phone is a 12 megapixel resolution a lot of people that's that's a lot of resolution actually if you just go back a couple of years a dslr with 12 megapixels was considered tired uh and so introducing a 48 megapixel sensor you have two outputs with this the standard output you can go down to 12 megapixels which gives you the ability to crop that sensor well that's and you get a fourth lens yes and it is it's like it's like we took iphone and put a fourth lens on the back of it because what we do is we take that middle 12 megapixels which is out of the middle of the sensor yeah and that's a 2x optical so it's not digitally enhanced it's a 2x optical and so now it is like it's got a fourth lens but the other thing this lets us do is bring back that 2x portrait mode and so now folks can use that beautiful depth of field rendering on that storied 50 mil focal length so you actually get the depth of field but and but then in portrait mode as well we can do the computational work on top of it that's amazing yeah it's amazing so what are you most excited about with the with iphone 14 now i'm most excited about action mode and then you'll see at the top left there's uh ah the new action icon the running man the running man and then turn him on and then you're ready to go and so now then i'm going to do this crazy thing of i'm going to try and follow them and as you see i'm running around and there is no way that would be usable footage so the key though is is don't don't you just shoot you just shoot you don't worry about it all right so now you look at the footage it comes out and it's almost like i'm a real sports videographer it's just i don't understand how that happened and i'm one of the engineers who built this thing it just feels like a map so not just that but like you're not there's not a lot of action coming off no offense john but i can be action and do my crazy shots where i run around yeah hold it with don't even worry about it huh all right yeah we can and then we'll start the uh russian museum you can dunk one right franchise yeah look you got your tennis shoes are you ready living at my middle floor love it and just follow with traditional image stabilization it's all about kind of dealing with like the minor handshakes and so like which is the thing that you had kind of would really what that replaced was those kind of like small gimbals that we all used to carry around and now those are basically a thing in the past computational photography computational videography replace the gimbal and so our next challenge was if you look at sort of folks on the sidelines of sporting events they've got these giant back mounted rigs that they're kind of like hanging cameras off and so our question was like well can we get rid of things like that too because to your point about folks with little kids that is sports photography so like you know but you know mom and dad aren't going to go go and pick up some giant steady camera the best to chase the toggle around and so our question was well can they just do that with the iphone and so what we do is because we've got such a large sensor with 4k and so much processing power we can say well let's shoot this in 4k so we've got this giant overscan region that we can then stabilize and as we do that then you get to basically focus on living your life chasing your toddler through the room uh watching your friends play basketball but doing it from the middle of the basketball court and like sort of following them around and not having to think about like oh this is going to look like unstabilized garbage because we take care of that for you yeah it's pretty cool you have something that's useful in the end you guys thank you so much this is such an honor to be on the apple campus i've never been here before and here we are welcome an actual orchard yeah there we go and you can't see it in this shop but we're in the center of the the bowl here it's yeah this is a real treat\n"