Our 36 Core Video Rendering Server – Finally Explained

The server version of Sorenson's Squeeze requires a Windows Server operating system to run on top of and costs $5,000 plus yearly maintenance fees. Furthermore, despite the assurance received from Sorenson's engineers that there shouldn't be any gamma or color shifts using QuickTime as a wrapper between Squeeze, DNx Export, and Premiere's import, it was found that there were indeed issues with this setup. As a result, the team had to go back to the drawing board and explore alternative options.

This led to a conversation with Blackmagic Design, who suggested Cineform as a great mezzanine codec. Initially dismissed due to its limited compatibility with most software, including Sorenson's Squeeze, the team was informed that Blackmagic Design could add compatibility with the next yearly release. In light of this, the team decided to quickly transcode their footage to Cineform. To their surprise, it turned out that even with only 30 CPU utilization (effectively 10.5) Adobe Media Encoder (2015) was able to kick Sorenson's Squeeze in terms of converting to Cineform versus DnxHD.

The benefits and drawbacks of using 4K footage were discussed in a separate video, but for this project, the team settled on up-sampling 1080p footage to achieve better green screen and punch-in performance. The decision was made to use Adobe Prelude 2015 to ingest their footage automatically, dumping the raw files off the camera to a local storage array as a precautionary measure. The raw files were then queued up for transcode jobs in Media Encoder 2015 to send to their network share.

The team used Media Encoder 2014, which is included with your Creative Cloud license, to monitor the watch folders that they exported their finished jobs into and turn those into H.264 files ready for publishing on websites like YouTube, Vimeo, Yooku, Billy, etc. Although hitting both instances of Media Encoder resulted in CPU usage as high as 90%, this didn't mean that a multi-thousand-dollar network render machine was required to utilize this workflow.

In fact, the team found that their setup could be scaled down with a low-power networked machine if desired, allowing for improved timeline performance and reduced waiting times on their main workstation. The use of Adobe Media Encoder allowed them to work in the background while something else worked on it.

TunnelBear is an easy-to-use privacy app for mobile, desktop, and browser that allows users to tunnel into 14 different countries, accessing geo-blocked websites as if they were in that country. The app is super easy to use, with a simple interface that allows users to pick their desired country and turn TunnelBear on. Their internet connection becomes fully encrypted, and users don't need to be technical to use or install the app.

TunnelBear offers 500 megabytes of free data per month and an extra gigabyte if users tweet about it. If needed, prices for unlimited plans start at $699 a month. The app has plain English privacy policy and over 5 million users trust them already. Users can try it out for free by visiting tunnelbear.com.

The final section of the article is dedicated to what viewers should do next. They are encouraged to click on the little button up in the top right corner to check out Luke's video, where he delves into the ins and outs of password protection, making sure that users' passwords remain secure from prying eyes.

"WEBVTTKind: captionsLanguage: enthis journey begins over six months ago when i reached out to intel about supporting us with some chips a low-power xeon to build the high-speed storage server for our new office that i first showed off here then a pair of their top of the line e5 2699 v3 18 core xeon processors to build a network video rendering server also for the new office well as it turns out they couldn't send us the low power chip for the storage server so we bought our own but whatever the reason was they were able to honor our request for the pair of 4 500 processors so it is with much thanks to intel along with supermicro who provided a dual socket motherboard kingston who provided 128 gigs of ddr4 ecc ram and norco noctua and fsp who provided our case cooling and redundant power that we are able to bring you these findings because you see the deal was this send us the chips and we'll make a video about how we're using them which sort of puts a lot of pressure on us to figure out not only if the concept of network rendering also known as a render farm works i mean that's been pretty standard stuff for years especially in animation but also to find a way to efficiently use those resources in our workflow so without further ado thanks to weeks of work by edsel and much patience from the rest of the team i am pleased to present our new editing workflow it's fast it has built-in redundancy for our files and to quote dimitri from hardware canucks who has already switched to it it brought the joy back to 4k video editing for me so here we go the logitech g303 features a lightweight design and advanced optical sensor with delta zero technology for precise tracking and rgb lighting to match your setup check out the link in the video description to learn more so the most obvious bottleneck in a video editor's daily life is waiting around for encoding tasks to complete outputting a finished ready to upload h.264 video file can take anywhere from 15 to 30 minutes for us with one pass vbr or even over an hour with two passes so that was the first thing we tried to tackle with the 36 core server machine for software telestream episode and sorenson squeeze desktop were the front runners initially telestream was intriguing thanks to its unique ability to split an encoding project into pieces processed them across many cores and then stitched them back together at the end regardless of the codec and sorenson due to its excellent handling of multiple concurrent projects also a time saver if you have many processing cores and its ability to utilize all cores for a single project with supported codecs so episode is a great concept but we abandoned it quickly due to stability issues sorensen on the other hand impressed the snot out of us the software worked their support staff was professional and even as a trial customer we were escalated to engineering whenever we encountered more complex issues outstanding so next we tested a variety of different output formats and found that thanks to optimizations within premiere pro our projects could be exported very quickly by our editing workstations in dnxhd to our server where sorenson would utilize all cpu cores to output an h.264 master copy that was suitable for upload to youtube and other video sharing sites in a fraction of the time that adobe media encoder could do it and all of this while leaving the video editors computers free to work on other things instead of just sitting there barely usable while they encoded video so mission accomplished then right well you know how the rabbit hole is the discoveries we made about how dramatically a program's optimizations around a given codec could affect performance raised more questions than they answered and while premiere pro's claim to fame is that unlike competitors like avid and final cut it allows any video file you want to simply be plunked onto the timeline and edit it in real time it made us consider the way that 4k footage off our panasonic gh4 camera just seemed to chug as you scrub through it on the timeline even on six cpu cores and a 10 gigabit network connection maybe there's some merit then too going back to the old way so we devised a workflow that would utilize our copious amounts of cpu horsepower to transcode footage from whatever format our various cameras captured in natively to an intermediary or mezzanine codec that was compatible with all the programs in our workflow so for a number of reasons avid's dnxhd was chosen and would you look at that comparing pre-fetch latencies with native gh4 footage the delay when moving the playhead in premiere was reduced by nearly 25 times at 4k depending which program exactly was used for the trans code so it was at that point that the goal actually changed obviously we could just have the individual video editors convert all the footage off the cameras to our mezzanine codec when they're working but then we'd be right back where we damn well left off with highly skilled video editors staring at their barely functional computers waiting for a big queue of videos to transcode so no we needed a way to avoid that by using our overpowered hardware and the answer of course is to do the transcode at the time of ingest or when the footage is initially removed from the camera and here's some bad and some good news while squeeze desktop sorenson's low end offering can perform a task like this across many cpu cores because we dump so many video clips off our sd cards at a time it just wasn't stable enough with our workload so we turned to their server offering which operated much more smoothly to automatically monitor our video file dumping folders and transcode everything we dropped in them so the benchmark was a folder of 41 video files totaling 16.7 gigs and by prioritizing multiple tasks this could be processed in about 14 minutes a small price to pay even on a video that needed to be edited immediately for the improved timeline performance but unfortunately time wasn't the only price the server version requires a windows server operating system to run on top of and costs 5 000 plus yearly maintenance fees and furthermore despite the assurance we received from sorenson's engineers that there shouldn't be any gamma or color shifts using quicktime as a wrapper between squeezes dnx export and premier's import it was there and very difficult to compensate for so it was back to the drawing board somewhat which led us to a conversation with blackmagic design where they said that cineform could also be a great mezzanine codec an option that had been dismissed early on due to its limited compatibility with most software including sorensen squeeze although they had said they could add compatibility with the next yearly release so could we quickly transcode our footage to cineform it turns out that yes even with only 30 cpu utilization effectively 10 and a half of our 36 course adobe media encoder yes back to that again managed to kick sorensen's ass converting to cineform versus sorenson converting to dnxhd and all of this without a significant loss in quality regardless of whether we're working with native 4k footage for better green screen and punch in performance or settling for up sampling 1080p footage for our finished project by the way please see this video for more details about the benefits and the drawbacks of 4k so that's all fine and good linus but does cineform deliver the answer again yes while file sizes are significantly larger especially at 4k than even the source files timeline performance is better than even dnxhd thanks to an extraordinarily poorly documented feature of cineform it's gpu accelerated so even though dnxhd also performs like a champ it can eat 50 to 60 of a 12 core xeon while scrubbing through footage while cineform is using the fancy titan x graphics cards that nvidia sent us for our workstations to keep cpu usage much lower so then here is the process that we finally settled on we're using adobe prelude 2015 to ingest our footage automatically dumping the raw files off of the camera to a local storage array on the machine in case of an emergency and then queuing up transcode jobs for each of those clips in media encoder 2015 to send to our network share we then use media encoder 2014 which is included with your creative cloud license by the way to monitor the watch folders that we export our finished jobs into and turn those into h.264 files ready for publishing on websites like youtube vessel yooku billy billy and facebook and while hitting both instances of media encoder we've seen cpu usage as high as 90 percent but that doesn't mean that you need a multi-thousand dollar network render machine to utilize this workflow all we've demonstrated here is that it's scalable to that kind of hardware for a small team you could easily take advantage of this on a smaller scale with a low power networked machine if you just wanted to improve your timeline performance and not sit around waiting for exports on your main station while something else works on that in the background speaking of things that run in the background tunnelbear is an easy to use privacy app for mobile desktop and browser so they got support for ios android mac pc and chrome it allows you to tunnel to 14 different countries allowing you to browse the internet as if you're in that country it works for accessing things like geo-blocked websites the apps are super easy to use so you just like pick your country and turn tunnel bear on and your internet connection gets fully encrypted and you don't have to be technical to use or install tunnelbear and if you get stuck you can contact their friendly support bears that are standing by 24 hours a day they've got a plain english privacy policy and they've got 5 million users that trust them already so you can try it out for free tunnelbear actually gives you 500 megs of data for free every month and an extra gig if you tweet at them but if you need more prices for unlimited plans start at 699 a month so head over to tunnelbear.com ltt linked in the video description to try it out today thanks for watching guys if this video sucked come on this was a lot of work but if it was awesome please hit that like button get subscribed or even consider supporting us directly by using our affiliate code to shop at amazon buying a cool t-shirt like this one or with a direct monthly contribution through our community forum which you should definitely join links up there and now that you're done doing all that stuff you're probably wondering what to watch next so click that little button up in the top right to check out luke's video where he goes through the ins and outs of password protection that is protecting your passwords making it so other people don't have them see you next timethis journey begins over six months ago when i reached out to intel about supporting us with some chips a low-power xeon to build the high-speed storage server for our new office that i first showed off here then a pair of their top of the line e5 2699 v3 18 core xeon processors to build a network video rendering server also for the new office well as it turns out they couldn't send us the low power chip for the storage server so we bought our own but whatever the reason was they were able to honor our request for the pair of 4 500 processors so it is with much thanks to intel along with supermicro who provided a dual socket motherboard kingston who provided 128 gigs of ddr4 ecc ram and norco noctua and fsp who provided our case cooling and redundant power that we are able to bring you these findings because you see the deal was this send us the chips and we'll make a video about how we're using them which sort of puts a lot of pressure on us to figure out not only if the concept of network rendering also known as a render farm works i mean that's been pretty standard stuff for years especially in animation but also to find a way to efficiently use those resources in our workflow so without further ado thanks to weeks of work by edsel and much patience from the rest of the team i am pleased to present our new editing workflow it's fast it has built-in redundancy for our files and to quote dimitri from hardware canucks who has already switched to it it brought the joy back to 4k video editing for me so here we go the logitech g303 features a lightweight design and advanced optical sensor with delta zero technology for precise tracking and rgb lighting to match your setup check out the link in the video description to learn more so the most obvious bottleneck in a video editor's daily life is waiting around for encoding tasks to complete outputting a finished ready to upload h.264 video file can take anywhere from 15 to 30 minutes for us with one pass vbr or even over an hour with two passes so that was the first thing we tried to tackle with the 36 core server machine for software telestream episode and sorenson squeeze desktop were the front runners initially telestream was intriguing thanks to its unique ability to split an encoding project into pieces processed them across many cores and then stitched them back together at the end regardless of the codec and sorenson due to its excellent handling of multiple concurrent projects also a time saver if you have many processing cores and its ability to utilize all cores for a single project with supported codecs so episode is a great concept but we abandoned it quickly due to stability issues sorensen on the other hand impressed the snot out of us the software worked their support staff was professional and even as a trial customer we were escalated to engineering whenever we encountered more complex issues outstanding so next we tested a variety of different output formats and found that thanks to optimizations within premiere pro our projects could be exported very quickly by our editing workstations in dnxhd to our server where sorenson would utilize all cpu cores to output an h.264 master copy that was suitable for upload to youtube and other video sharing sites in a fraction of the time that adobe media encoder could do it and all of this while leaving the video editors computers free to work on other things instead of just sitting there barely usable while they encoded video so mission accomplished then right well you know how the rabbit hole is the discoveries we made about how dramatically a program's optimizations around a given codec could affect performance raised more questions than they answered and while premiere pro's claim to fame is that unlike competitors like avid and final cut it allows any video file you want to simply be plunked onto the timeline and edit it in real time it made us consider the way that 4k footage off our panasonic gh4 camera just seemed to chug as you scrub through it on the timeline even on six cpu cores and a 10 gigabit network connection maybe there's some merit then too going back to the old way so we devised a workflow that would utilize our copious amounts of cpu horsepower to transcode footage from whatever format our various cameras captured in natively to an intermediary or mezzanine codec that was compatible with all the programs in our workflow so for a number of reasons avid's dnxhd was chosen and would you look at that comparing pre-fetch latencies with native gh4 footage the delay when moving the playhead in premiere was reduced by nearly 25 times at 4k depending which program exactly was used for the trans code so it was at that point that the goal actually changed obviously we could just have the individual video editors convert all the footage off the cameras to our mezzanine codec when they're working but then we'd be right back where we damn well left off with highly skilled video editors staring at their barely functional computers waiting for a big queue of videos to transcode so no we needed a way to avoid that by using our overpowered hardware and the answer of course is to do the transcode at the time of ingest or when the footage is initially removed from the camera and here's some bad and some good news while squeeze desktop sorenson's low end offering can perform a task like this across many cpu cores because we dump so many video clips off our sd cards at a time it just wasn't stable enough with our workload so we turned to their server offering which operated much more smoothly to automatically monitor our video file dumping folders and transcode everything we dropped in them so the benchmark was a folder of 41 video files totaling 16.7 gigs and by prioritizing multiple tasks this could be processed in about 14 minutes a small price to pay even on a video that needed to be edited immediately for the improved timeline performance but unfortunately time wasn't the only price the server version requires a windows server operating system to run on top of and costs 5 000 plus yearly maintenance fees and furthermore despite the assurance we received from sorenson's engineers that there shouldn't be any gamma or color shifts using quicktime as a wrapper between squeezes dnx export and premier's import it was there and very difficult to compensate for so it was back to the drawing board somewhat which led us to a conversation with blackmagic design where they said that cineform could also be a great mezzanine codec an option that had been dismissed early on due to its limited compatibility with most software including sorensen squeeze although they had said they could add compatibility with the next yearly release so could we quickly transcode our footage to cineform it turns out that yes even with only 30 cpu utilization effectively 10 and a half of our 36 course adobe media encoder yes back to that again managed to kick sorensen's ass converting to cineform versus sorenson converting to dnxhd and all of this without a significant loss in quality regardless of whether we're working with native 4k footage for better green screen and punch in performance or settling for up sampling 1080p footage for our finished project by the way please see this video for more details about the benefits and the drawbacks of 4k so that's all fine and good linus but does cineform deliver the answer again yes while file sizes are significantly larger especially at 4k than even the source files timeline performance is better than even dnxhd thanks to an extraordinarily poorly documented feature of cineform it's gpu accelerated so even though dnxhd also performs like a champ it can eat 50 to 60 of a 12 core xeon while scrubbing through footage while cineform is using the fancy titan x graphics cards that nvidia sent us for our workstations to keep cpu usage much lower so then here is the process that we finally settled on we're using adobe prelude 2015 to ingest our footage automatically dumping the raw files off of the camera to a local storage array on the machine in case of an emergency and then queuing up transcode jobs for each of those clips in media encoder 2015 to send to our network share we then use media encoder 2014 which is included with your creative cloud license by the way to monitor the watch folders that we export our finished jobs into and turn those into h.264 files ready for publishing on websites like youtube vessel yooku billy billy and facebook and while hitting both instances of media encoder we've seen cpu usage as high as 90 percent but that doesn't mean that you need a multi-thousand dollar network render machine to utilize this workflow all we've demonstrated here is that it's scalable to that kind of hardware for a small team you could easily take advantage of this on a smaller scale with a low power networked machine if you just wanted to improve your timeline performance and not sit around waiting for exports on your main station while something else works on that in the background speaking of things that run in the background tunnelbear is an easy to use privacy app for mobile desktop and browser so they got support for ios android mac pc and chrome it allows you to tunnel to 14 different countries allowing you to browse the internet as if you're in that country it works for accessing things like geo-blocked websites the apps are super easy to use so you just like pick your country and turn tunnel bear on and your internet connection gets fully encrypted and you don't have to be technical to use or install tunnelbear and if you get stuck you can contact their friendly support bears that are standing by 24 hours a day they've got a plain english privacy policy and they've got 5 million users that trust them already so you can try it out for free tunnelbear actually gives you 500 megs of data for free every month and an extra gig if you tweet at them but if you need more prices for unlimited plans start at 699 a month so head over to tunnelbear.com ltt linked in the video description to try it out today thanks for watching guys if this video sucked come on this was a lot of work but if it was awesome please hit that like button get subscribed or even consider supporting us directly by using our affiliate code to shop at amazon buying a cool t-shirt like this one or with a direct monthly contribution through our community forum which you should definitely join links up there and now that you're done doing all that stuff you're probably wondering what to watch next so click that little button up in the top right to check out luke's video where he goes through the ins and outs of password protection that is protecting your passwords making it so other people don't have them see you next time\n"