The Complexity of Recommendation Systems: Understanding the Challenges of Diversity and Personalization

Our recommendation systems are designed to suggest relevant content to users based on their past behavior and interests. However, as we've observed firsthand, these systems can become confused and lose sight of what's truly important - connecting creators with their fans and increasing viewership. In our own case, a friend sent us a video about a ski race from Southern California, which sparked an interest in my cousin's boat racing endeavors. While I watched the videos initially out of curiosity, I soon realized that I wasn't genuinely interested in the topic.

This experience highlights the challenges our systems face when it comes to diversity and personalization. Our algorithms are designed to make recommendations based on user behavior and preferences, but this approach can lead to a narrow focus on what's already worked before. We've come to realize that simply adding more random elements to our system isn't enough - we need to strike a balance between diversity and relevance.

The problem is that when we try to increase diversity, we often reduce the amount of viewership in the short term. This can be frustrating for both creators and users, who are looking for content that resonates with them. Our intuition tells us that increasing diversity will lead to more opportunities for viewership, but our experiments have shown us that this isn't always the case.

One approach we've considered is adding a random element to our system, with the goal of introducing new and unexpected content. However, this approach has proven to be ineffective, as it often leads to a decrease in viewership. Instead, we're focused on developing algorithms that can accurately identify diverse interests and connect creators with their target audience.

This is a complex problem that requires a nuanced understanding of human behavior and preferences. While it's true that our individual tastes may align with those of others, there are many factors at play when it comes to video selection. For example, children, parents, and people from different countries all have unique interests that diverge from one another.

Ultimately, the holy grail is to develop a system that can connect each person with their diverse interests, with the goal of increasing viewership for creators and providing users with relevant content they'll love. This requires a deep understanding of our users' behavior, preferences, and cultural context. By prioritizing diversity and personalization, we aim to create a recommendation system that truly understands what makes each user unique.

For us, the key is to understand how to connect the dots between creators and their fans. We're committed to helping creators succeed by providing them with accurate insights into their audience's interests and preferences. By doing so, we hope to foster a community of engaged users who will discover new content they love, and support creators in their pursuit of artistic excellence.

The challenge ahead of us is significant, but we're committed to solving it. We'll continue to develop our algorithms and refine our approach, always with the goal of connecting users with diverse interests to the content that resonates with them. By doing so, we aim to create a recommendation system that truly understands what makes each person unique, and delivers on their needs in a way that's both personal and relevant.

"WEBVTTKind: captionsLanguage: enWe we want to help people find the videos. They're interested in watching andwe start with the premise thatthere are hundreds of hours of video that each viewer would like to watch andWe're trying to figure out a way to connect those viewers with that 100 hours of videoMy name is Christos KudrowAnd I'm responsible for search and discovery on YouTube the easiest thing which actually is not always that easy isIf someone were to search for a video, right?So if you search for a video you're looking for something particularWe want to make sure that that shows up right at the top of the search results?But even that can sometimes be hard because searches are ambiguous and even when the search itself isn't ambiguousIt's not always clear which video you are looking forBecause there's a lot of videos about just about every possible imaginable thing on YouTube, but then there's another problemwhich is the fact that you may not know all the things that there are for you to watch on YouTube and soWhat if it doesn't occur to you to search for something I used to pole-vault?I wouldn't imagine that they are pole-vaulting videos on YouTubeBut how am I ever gonna find them if I don't bother to search for them well somehow?We've got to connect you with those videos as well. Why do you feel compelled to connect me with a video?I wasn't looking for is that a business thing for you because you want to keep people engaged with your productwe know that creators uploaded these videos andThere are viewers interested in these videosAnd we feel like it's our job to make that connection if we're not gonna help make that connectionWhy should the creator's bother to upload those videos?I mean that's one thing that I'd like to make clear is that our success depends on helping the creators find their viewers?The simplest thing is that if we see you watch one of a certain kind of video then?We know that other people who watch that video in the past?Went on to watch this other videoAnd so it's quite natural that if we see you've watched this first one that we might think that you watch thatYou'd want to watch that other one, too. That's mostly a matter of just accounting more than anything elseIt's just keeping track of which videos get watched together and so so that's the simplest wayWe do it if you take that to a more sophisticated level let's you watched a few videos about Lionel MessiAnd you watched a few others about Cristiano Ronaldo. I think maybe you might be interested in soccerSo we might recommend to a video that isn't just about lionel Messi or Cristiano Ronaldo, but some other soccer teamThat's playing because you've demonstrated some evidence in a in the broader subject another way that we might find to recommend things to you arevideos that are popular in yourGeographical area soas you can imagineBollywood is very popular in IndiaIt's also popular amongPeople outside of India but in order to recommend it to those people we would want to see some evidence that they areInterested in Bollywood some of this is quite obvious, but also some of it's quite crude. You know geographic areasYou're an engineer. You like bollywood how sophisticated are you guys getting with this now. I think the sophistication comes fromfiguring out what's important about the video that indicates your interest in it, soFor instance you might have watched some soccer videos, but it was because those were the most amazing goalsAnd you're not really that interested in soccer, but you're really interested inJust amazing sporting thingsor you might have watched those videos becausethey were from a team that you happen to follow because that's an area of the world that you're from and so thesophistication comes in separate separating those two kinds of associationsOr even in a more crude level in figuring out. What's important about that video that?That would make us want to suggest it to some other personDo you have to understand people or are you just like just number crunchers?The human understanding part is where we take our intuition for what Sorts of changes. We might try nextyou could imagine that there's a lot of things that we could try andWe can't do all of them at once and so we have to use someIntuition about which sort of things we expect might be more or less successful than than other things first of all we look atexperiments a lotSo we're running a b tests on YouTube itself we alsodoqualitative research, so weinvite people to come andsit with us andWatch them use YouTube and ask them the kinds of things they're interested in and ask them that they would expect to find that onYouTube if they've ever seen that on YouTubeyou know how would they go about finding it on YouTube andand we use those sorts of qualitative analyses to try to guide our intuition for what sort of experiments we run butit all comes back tomaking these broad changes and then experimenting with them andWhat's important about that is because each change that we make because we've done experiments. We can be sure that it's improved the situationNow it may not have improved itUniformly across the board but overall it's it's been an improvement. What is the biggest grunt and who does it come from?The biggest gripe is you have a billion videos and I can't find anything good to watchandThat's that's basically solving that problem for the viewers is our missionof course people people often say as well that they can't find the video that they're looking for andyou know there's just many many different queries and ways that people go about looking for things andYou know we we keep working on them one after another until we figure them outWell, it's easy for our systems to get confusedBy in my caseI'll give you an example a friend sends me a video about aSki race from in Southern California, and it's because my cousin has a boat entered in this ski raceI watch his video and it's actually quite long and then I watch a couple more videosBut I'm not really very interested in this sort of thingThe next day I come back, and there's more videos waiting for me about this ski race. I think okayWell that's fine, but two weeks later. I come back, and there's yet more videos about this case, and I think to myselfI'm sorry that I watched those videos about the skI raceyou got me I did but butWhat do I have to do to get you to realize that I'm not Gonna watch any more of these?Well, I mean it's easy for our systems to become confused about the fact that you happen to have watched it at one timeBut you're not really interested in it very muchAs opposed to well you you watched a little bit less of somethingBut that's something that you really like to see more of and soWe we don't haveThe information we have to go on is what you and other people have watched in the past and so using all of thatAnd by the way, it's only you and a billion other peopleusing all of that and trying to make sense of it is is what becomes hard do you build aRandom chance Element into it. I'm not saying like here's a completely random videoBut does it allow for a degree of this is another baseball video, but it's bit of a wild cardOr is then is that not because I guess I'm touching on making it a moduluswe do work on trying to increase diversity and the challenge there is thatWe have an intuitive belief that increasing Diversity will lead toThe opportunity to get even even more viewership in the future, but every time we try to increase diversityWe tend to reduce the amount of Viewership we have in the short termWell, I think it's because we haven't quite gotten it right yet, and so um you know until we can until we canrecommend, sort ofUnexpected things that are almost always, right?People will just go awayThey won't see as much of what they were looking for or what they expected and they go awayAnd so I guess it's a quite a high bar to add some diverse things thereAnd so even though we have this intuition and our experiments tend to make us more conservative with itThat's the way that we want to go about trying to addI would call it that I would say that areThe naive approach to trying to increase diversity would be to add a random elementand I can assure you that that's not going to work becauseYou give up too much viewership in the short term and so what we've learned. Is that random isn't good enough andWe need to make it so that we're almost always right when we add some diverse and unexpected elementmy holy Grail isTo come to YouTube and have itfind the hundreds of hours of videos that I can't stop watching becausethose are the ones that I'm interested in andI think the important point about this is thatit's not going to be the same asYours or someone else'sand you might thinkWell how different can it be I mean you know I and all my friends seem to like all the same videos that we seeOn YouTube and while that may be truethink aboutyou know think about some children that you know andThey're you know their favorite videos probably won't will be different than yours and then think about your parentstheir videos would probably be different than yours and now think about the people in another country andHow different what they're interested in watching is from yours like we need to solve this problem for everybodyacross the across the world andThat is really the holy grail is how do we connect?each one of those people with theirDiverse interests with the videos they're looking for and want to watch I think the most important thing is thatWe are very interested in helping the Creator'sconnect to their fans andHelping them increase their viewership because that's entirely in our interestsAnd so that's the thing that I want to make sure people understand is that when we make changes to thingsIt's because we absolutely believe that it improves the ability for creators to be successfulandeverybody was going to click on that one andWhen someone's writing code the video guy they need to put the logic in the code that says you know where you should stop\n"