The Perils of Social Identities: How They Can Mask True Optimal Convergence in Software Development

As a software engineer, it's easy to get caught up in the hype surrounding various technologies and frameworks. We often find ourselves identifying strongly with specific tools, such as rails or ruby, which can make it difficult to consider alternative options. However, this kind of social identity can be misleading, masking the severity of the problem of convergence - finding what is genuinely optimal.

In reality, our goal should be to find the true global maximum, rather than just exploring different approaches. But how do we achieve this when our identities are tied to specific technologies? The answer lies in jettisoning labels that we've come to associate with ourselves as software engineers. As Paul Graham once said, "keep your identity small." This means letting go of the need to identify ourselves solely with certain tools or frameworks, and instead focusing on solving problems in a way that's agnostic to specific technologies.

For example, if you consider yourself a rails developer, you may think that this defines who you are as a software engineer. However, what happens when the right tool for the job isn't rails? What happens when 10 years from now, rails is no longer the best choice for the project? By not identifying ourselves solely with specific tools, we can be more adaptable and open to changing circumstances.

Another important aspect of this approach is exploring new terrain. This means taking risks and trying things that are outside our comfort zones. It's easy to get caught up in paying lip service to different technologies or attending talks, but true exploration requires doing things that are genuinely challenging. We must be willing to walk down the hill, even if it makes us nervous - because this is where real progress happens.

The concept of fake risks is also essential to consider. These are risks we take without truly compromising our safety, comfort, or prestige. We may attend a conference out of obligation rather than genuine interest, or stay within our familiar bubble rather than taking the leap and trying something new. Real risk-taking requires us to be vulnerable and open to uncertainty.

Ultimately, exploration should be a fun and enjoyable process. When we let go of the need to constantly move up and instead focus on learning and growth, we can have more genuine experiences. It's okay to take risks that might make our work harder in the short term, but will lead to greater progress in the long run.

As a software engineer at AirBnB, I've learned that taking these kinds of risks is essential for growth. If you're a senior software engineer or data scientist looking for a new challenge, know that we're always hiring. We value individuals who are willing to take calculated risks and explore new territory - because it's in these moments of uncertainty that true progress happens.

By keeping our identities small, exploring new terrain, and taking real risks, we can move beyond the constraints of social identity and find what is genuinely optimal in software development.

"WEBVTTKind: captionsLanguage: entoday i'll be giving a talk on why software engineers disagree about everything okay uh my name is haseeb qureshi i'm a software engineer at airbnb i work on the risk team i'll be talking a lot more about that and a lot more about kind of my own history and the different uh different things that i've done in my past uh because that really informs a lot of what this talk is about uh i don't know that i'll totally answer the question of why software engineers disagree about everything but i think uh this talk is gonna be kind of an exploration of this question from a number of different angles and another things can happen in this talk is i'm going to disagree pretty strongly with dhh's keynote so if so that might be interesting but we'll hopefully i'll at the very least if not convince you of what i'm saying uh you know maybe get you thinking about some things so uh this talk in large part is going to be about philosophy so i know uh you know it's a railsconf usually people come up here and they talk about controllers or i guess someone else is talking about sorting and augmented reality uh i'm not gonna be offended if you get up right now and want to go to another talk that's totally cool but uh we are gonna go way into the weeds today because that's that's what i'm all about so uh specifically i want to talk about the field of philosophy known as epistemology and epistemology the easiest way to define epistemology is epistemology is the nature of knowledge of justification and of the rationality of belief uh so this actually this diagram here is uh of uh phrenology which is this old science of you know trying to figure out what parts of the brain responsible for what which is a very old and ancient version of epistemology which we've now thankfully displaced but essentially in epistemology are the two questions how does anyone know anything and how do they know that they know it uh i mean these two questions are kind of navel-gazy but that's that's fine um so let me tell you about me so i studied english in philosophy in school and i before i ever came into the tech world i used to be a professional poker player and i did that for about five years uh so you know i kind of took a very different path into programming than most people uh and then i worked as a programming instructor i taught programming at a coding bootcamp and for the last a little bit over a year i've been working as an engineer uh on the risk team at airbnb uh basically fighting fraud and so because of this i've kind of you know snaked my way through many different subcultures i've been something of a of a chameleon in that i've learned the different norms and the different beliefs and the different knowledge systems that many different worlds uh have and try to assert on you if you become a part of that world and one thing that i've noticed and you can't help but notice the more worlds you step in and out of is that knowledge is deeply cultural so in the world of programming you know the kind of things you might hear people tell you when you come into this world they might tell you that full stack javascript is the future they know this for sure uh they'll tell you that everyone should know c if you don't know c what are you doing go out there crack open a book and learn some c they'll tell you rust is the best language for systems programming they'll tell you that relational databases don't scale they'll tell you that tdd is a fantasy and nobody serious tries to do it anymore it's it's not it's not practical so now all these are interesting questions and they're all things that people disagree on obviously uh but they're people they're things that people hold with very very strong conviction that this is clearly and just irrefutably true now i'm not interested in convincing you of any of these claims each of these could probably be great talks themselves what i'm interested in is why we disagree about them why isn't that we all converge on an answer to these questions now it's funny because when i was in the poker world i remembered this this was very familiar to me when i was learning how to be a poker player because in the poker world people will tell you things like no limit is dying you have to learn mixed games or they would tell you everybody needs to use a hub or they would say only fish play loose passive styles they would say gto is a fantasy nobody actually plays like that and there's a wonderful analogy between these these two worlds not just in that people argue a lot and then people are maybe kind of whiny and excessive but that you know they're they're all these things that people fundamentally disagree on and each side of them holds tremendous conviction that their side is clearly and obviously correct um and so when most people hear stuff like this when they hear someone say hey everybody should know c and if you don't know c you're not a real programmer their natural reaction is oh god is that true if that's true what should i do about it uh but i for me having having gone through this song and dance so many times my reaction is why do they think they know that and why what is it that implanted so much confidence in them so much conviction that the thing they're saying is actually true and universally true and that's very interesting to me it's almost like i'm asking uh i tend to ask a more evolutionary question than a question about the world as it is right now you know the thing that's kind of weird about programming in particular is that nobody agrees you know there's so many things that people disagree on they disagree about functional programming object oriented programming tdd old you know robust proven frameworks shiny new ones that solve new problems in different ways serverless event driven blah blah blah right you guys you guys know all this uh nobody agrees why that's really weird now you might think it's obvious that people don't agree that's that's an understandable reaction you might say like well you know disagreement is a normal part of a society and like you know the way that we uh you know this this divergence is a source of discourse and that's how we end up having a healthy society so you might think that's obvious so of course people shouldn't agree um i think it's weird i think it's weird that people don't agree and i want to develop in you an intuition to also believe this is weird that people don't agree because in a way it makes more sense for people to agree than for them to disagree another way of putting what i'm saying is that systems in general tend to converge when you see a system you should assume that over time it's going to converge on what's optimal what's the sort of the optimal state for that system uh let me give you an example so i'm going to give you an example related to poker but you don't need to know anything about poker in order to understand this i can explain it in a couple sentences so in poker there's a strategy called set mining okay and set mining is very simple so in texas hold them you get two cards and if you want to set mine what you do is you wait for two cards that make a pair so let's say you get delta pair of threes then what you do is you wait to see if you make three of a kind if you make three of a kind you bet it really aggressively and if you don't you just fold that's set mining super super simple um so set mining was a strategy that was pretty simple and pretty stupid uh almost anybody could do it and it worked it worked really really unreasonably well given how simple the strategy was uh and so pretty soon what you would see in the world of online poker is that pretty soon once people started talking about this almost everybody started set mining at low to mid stakes full ring games the strategy just kind of took over like wildfire and what you could say is that the game converged on set mining everybody saw that set mining was the high ground and they all moved in that direction and uh so this is this is kind of one of the features that happened to poker uh after internet poker took off so internet poker took off really around 2004 2005 uh maybe a little bit before that as well and if you think about it before internet poker ever existed uh there was live poker right but playing poker in a brick and mortar casino and live poker was fundamentally really different than online poker in ways that people didn't really predict before it happened so you can imagine uh live poker you know let's say in the 80s and 90s you know uh there are some people playing poker out in phoenix maybe there are people playing poker in vegas that people play poker in dallas people playing poker in london um and these different groups of poker players weren't really communicating with each other you know ideas that were germinating in london weren't you know going over the pond and uh showing up in card rooms in dallas right people were kind of isolated to their little groups and they were just playing poker and figuring things out as they went but with the advent of online poker and the communication that it enabled for a lot of people uh sorry there was like a flyer running around with the advent of that communication what happened was it allowed the system to converge where suddenly somebody has an idea for a strategy somewhere in you know dallas or new hampshire or whatever and suddenly they can share it and everybody can learn about this strategy very very quickly uh and so what this meant was that the the state of poker strategy for a long time it was pretty static the way that people played in the 40s and 50s was not that different from the way people played in the 80s and 90s you know information just wasn't able to evolve that effectively but with the advent of online poker suddenly if you if you look at the curve of the complexity of poker strategy it just it just takes off right after 2003 when online poker comes to town uh and so right here uh is a graph uh that audio's not playing but that's fine he's saying look at this graph uh this is a joke uh that didn't play audio that's fine we can we're gonna roll with it um so this is actually a graph of uh this is actually just a graph that goes up into the right there's nothing to do with what i'm talking about but i thought it would give a sense legitimacy to the slides like oh this guy knows what he's talking about look at this graph um cool so the point is the point is online poker converged in a way that that live poker never did it was never able to um and this to me is it makes a lot of sense like of course online poker is going to converge right when you have these people able to communicate to see what each other is doing and move around in this terrain of course they're going to settle on what's optimal um and nature is full of convergent systems like this so let's say for example you're you're walking downtown and uh you you buy a loaf of bread maybe you're french i don't know uh and uh let's say you you eat half a loaf of bread and you toss the rest on the street well what happens is pretty soon a bunch of pigeons just it's not like they're all in one place standing over watching you throw the loaf of bread right but pretty soon one pigeon comes three pigeons come and then the a chorus of pigeons from presumably all over town just come in and start you know just just binging on the piece of bread that you've left uh and this this kind of makes sense right like the pigeons are able to very quickly come down to whatever's the optimal place for them to be they they eat as much food as they can and they disperse back to wherever was they came from maybe to the next best piece of food that they can find somewhere downtown um so the example of a convergent system so uh you know there's also uh this great ram scene album that actually i loved when i was a kid um it's not this is actually just cell membrane in german because that was the one i could find like 10 minutes ago uh but uh the the the uh cell membrane you guys might remember from high school chemistry you know there are water molecules on one side they permeate through this membrane and pretty soon uh you have the exact same pressure the exact same uh density of water molecules on each side of the membrane the system converges and you see the same sort of thing in stock markets a lot of natural phenomenon uh it seems like this sort of thing is everywhere so when i started working as a risk engineer at airbnb working in the fraud space naturally i started looking for convergence too because it just seems like one of those things that any any sufficiently uh complex system should eventually have some optimal state and it should find convergence so uh working in in the fraud industry is uh is its own little subculture its own little world uh and the subculture is not just the subculture of people working against fraud which is which is interesting but that wasn't really what fascinated me so much i mean it is very interesting but the really fascinating thing about fighting fraud is that you're actively fighting against a community of people you're actively fighting against a culture that's optimized to take you down and to basically exploit your defenses as effectively as they possibly can uh and so you know really there is actually a subculture that i can't see and don't have direct access to that's organizing and trying to attack all these you know major online companies uh and they're trying to make money and that's really interesting uh and so i wondered you know how do fraudsters figure out what to do i mean what they're what they're learning is pretty non-trivial right like they're learning how to how to script how to do you know sometimes cross-site requests forgery stuff xss whatever uh they come up with all these different ways to try to probe our defenses and attack us and ideally try to make some money how do they learn this and how do they communicate this to each other you know how does how does this knowledge spread in the world of fraud fundamentally the question i want to ask is is there convergence in fraud and this actually uh pretty soon after the new macbook pro replaced the escape cube we saw fraud go way up so that was uh one person after that that was all right there we go there we go a little just a little bit that's all i need in order to keep going uh cool uh awesome so i just want to run with you through a quick example of kind of what uh a standard fraud scheme looks like this is this is some site called chipmunk does anyone know what chipmunk is anyone familiar with this with this esteemed company great then we're gonna on them that's great uh so chipmunk uh let's say they cut some corners they kind of implement a feature they don't really know uh exactly uh you know they're not really thinking about fraud when they implement this right like some product manager is like hey you know we've got the story we've got to get up the end of the week uh we're gonna all look good once we fill this thing out great engineers who work on this just push it out don't really think too much about it so this feature is essentially uh some micro deposit uh where essentially they verify that you own this account because you they deposit two random amounts you report them back right pretty standard for some financial institutions to do this sort of thing um so let's say that they don't implement rate limiting easy thing to overlook uh well you know they go to sleep that night feeling great that they've launched this new feature and pretty soon someone gets alerted uh you know let's say at 2am that night that uh somehow we've lost a hundred thousand dollars which really should not have happened and the only way you could lose that amount of money obviously is if fraudsters are at scale repeatedly hitting this api repeatedly uh micro depositing money to their account and then just keeping it and so people scramble you know people like oh my god i can't believe this is going on they go and they try to patch this hole implement some rate limiting do some rules maybe reverse whatever transactions they can this is a common type of attack that you know a very easy vector that if you're a site influence something like this uh a kind of attack you would be vulnerable to if you didn't implement rate limiting so let's say they get that under control that's fine now what happens is that once you patch that vulnerability of course the fraudsters disperse they don't keep hitting it maybe they maybe they try to verify that in fact it is patched they try a few ways around it doesn't work cool you've patched the vulnerability and the fraudsters disperse but kind of like the pigeons they don't just go randomly to different places and just kind of you know leave your platform what ends up happening is that they then go to the next best place to defraud you so what they'll do is they'll find okay that endpoint wasn't that good but there's another thing that we can do like this may be longer more complex or more expensive fraud scheme uh and we're gonna go do that and basically what you see is that the fraudsters head down to the next highest peaks in this terrain of fraud which you know this this totally makes sense this is uh this is convergent behavior right this is exactly what you'd see if the terrain suddenly changes peak became a valley and they would go towards the next high speed so this seems to me like convergent behavior it seems like fraudsters converge on what's optimal but the more that i thought about this the more i realized that this didn't totally make sense and the reason that this totally didn't make sense because i asked the question why are they defrauding us at all why are any fraudsters attacking chipmunk this might seem like a weird question right like of course they're doing it so that they can make money and that's what fosters are motivated by but if you think about it the the terrain of all the companies that they can defraud is huge there are many many many companies that are vulnerable to fraud and pretty much all of these companies experience fraud to one degree or another right they're all these different sites that you could attack if you so wanted um and so what you would what do you you know what you can say is that there's some topology to that fraud right there's some peaks and valleys there's some there's some sites that are really really lucrative to the fraud and other ones that you know are really not so much and they're not a good place to be spending your time if you're a fraudster and so you'd think that everybody would converge on the peaks what you think is that almost all the fraud would go towards the very most profitable most uh attackable websites but what you see instead is that fraud is just kind of dispersed everywhere there's just a sort of ambient level of fraud that if you start an e-commerce site chances are you're going to get some amount of fraud why this is weird this should strike you as weird why is it like those fraudsters just don't care about optimizing are they not trying to make the best use of their time is there something that they're misapprehending about the terrain like what's going on that we don't see this convergent behavior within fraud now you know it occurred to me that actually you see the same sort of thing in software you know even just looking at open source solutions there's so many different software packages and so many different open source solutions to many many different problems that are effectively you know trying to do the exact same thing why it seems like software doesn't converge either when you when you don't get this just one okay this is the best way to solve this problem and everybody converges on that solution uh what you see instead is many many many competing solutions uh and it's not really clear which one is supposed to win and when one does win it's actually really surprising you know like so you know react you could say kind of one front end uis and so far as you can say anyone won front end uis uh and we're genuinely amazed by that on some meta levels like holy someone won no one ever wins anything anymore but react just one front end uis you know and you know maybe maybe that kind of relaxes you because you're like oh hey you know i just as long as i learn react i'll just be employable forever and you know that's fine uh but i think this is this is somewhat counterintuitive that more things aren't like this why is this such a rare story that react wins or sequel wins or one particular way of solving a problem just is clearly the best and we all adopt it so you might have uh an obvious objection to this analysis which would be that well of course you know things aren't going to converge because software doesn't just solve one problem right uh the obvious answer is that there are actually multiple terrains so you know you could say that there's a terrain for crud apps and that looks different and there's a terrain for the payments back ends and then one for social networks and maybe the tools and the solutions that you use for these different apps are different and that's why you see this this you know uh these multifarious solutions to effectively the same problem um but i i don't think that's that's sufficient because even within a single terrain you don't see convergence you know even just looking at crud apps right cred apps are you know they're the majority of what people build the majority of web apps are just you know glorified cred apps maybe with like elasticsearch store on top and like because of the vast majority of what people build there's a vast majority of what people disagree on so it doesn't seem to me like that's satisfactory to explain why we don't see convergence and the thing is you should want convergence convergence is actually good because convergence means that we all see the underlying terrain we all understand it and therefore we all go and do the best thing in the world of software for the most part we're not actually competing with each other we're actually all kind of at least to some degree or another motivated by each other succeeding you know when someone invents a great open source solution actually everybody benefits from that and most software kind of works this way so the question i want to ask is why do these systems not converge i think there are four reasons and i'm going to go through each of them so the first reason why these systems don't necessarily converge is because the terrain is actually unstable the terrain is changing it's not just one configuration that you can just see something out in the distance decide okay that's a peak i'm going to go over there right the fact that the terrain is changing means that you're not really sure if you do uh go in this direction that by the time you get there the terrain isn't going to be different right it also means of course that your terrain is moving underneath you so by the time that you when you originally came into ruby on rails like wow this is the hottest newest shiniest web framework anyone has ever come up with and now it's 2017 and rails is not the shiniest newest coolest framework that anyone's ever come up with uh and so that terrain has changed underneath you and that makes it hard for people to actually converge when there's so much change going on uh there was this article that someone wrote on medium a little while back how it feels to learn javascript in 2016. um you know if you didn't read this article i'm sure you have an intuition of what this thing said right and we we see this becoming more and more uh i don't know if i want to say a problem but more and more of a characteristic of software that things are changing very rapidly and it might be that that pace of change is even increasing which wouldn't be wouldn't be that uh unreasonable to expect actually as technology grows more and more rapidly um and you see the same sort of thing in the world of fraud so say for example uh that you know facebook has some kind of hole some kind of uh things that fraudsters can attack let's say it becomes really easy to spam facebook and you know do some kind of referral quick fraud whatever whatever stuff um so maybe you know if you're a fraudster you're making 20 an hour defrauding facebook doing this well facebook goes in and they patch the hole and now you know instead of making twenty dollars an hour you can only make three dollars an hour um but in order for the fraudsters to disperse and find the next highest thing uh you know they they they might be just incentivized to say like hey i don't actually know how easy it's going to be to defraud anything else because that might get patched too and that might you know uh the terrain might just change out underneath me so you know what i'll just keep defrauding facebook for three dollars an hour uh that's fine like at least i know this is working during the time that i have it so maybe that's that's some part of the explanation why you don't see this optimizing across across the terrain so the second reason why i think you see a lack of convergence in some of these domains is because of high switching costs okay so uh let's say that this is the domain that uh let's say this is software this is the map of the software world uh and let's say that you live uh all the way on the bottom right and the bottom right let's say is railsland and you know you go and talk to some uh wide-eyed bushy-tailed developer who tells you you know what you you just need to learn haskell and if you learn haskell you know all these type errors and mill checks and that all that stuff is just going to go away and you're going to live in a land of just pure programming bliss nothing will ever go wrong your code will be way more performant you know you are just going to yeah whatever things are going to be great uh the problem with that of course is that for you to actually engage in that switch let's say even you see what this person tells you and you agree with them but this peak that they're pointing to is very very far away so in order for you to get there you're gonna have to go down into a valley traverse a really really long space until you can finally actually reap the fruits of what they were claiming was so great and who knows because the terrain is changing by the time you get there it might not even be a peak anymore something else might have become a peak or maybe the place where you were at could have become a peak who knows uh and so this this instability and this uncertainty makes people really unwilling to incur the risks of traversing the terrain and exploring and you see of course the same thing in fraud just like i just mentioned with the facebook example and this whole thing of course is exacerbated by specialization the more specialized you are the harder it's going to be for you to convince yourself to engage in those high switching costs right because really specialization is just basically you finding your way to some local maximum that's what specialization is it's climbing as high as you can onto a local maximum once you're there it just becomes really uncompelling to climb all the way down from that peak of specialization you've arrived at to go find the true global maximum which might be very far away and you know we've already talked about the fact that it's changing and uncertain uh and so this makes it harder the more specialized we are same thing with fraudsters there are fraudsters who are specialized in you know attacking one side as opposed to another and it's hard for them to switch they have to relearn a lot of new things they have to start over in their knowledge so now the third reason why i think you don't see convergence in fraud or in software and these other fields is information sharing is a very important part of how you get convergence right if you think about it not all of us can actually clearly and lucidly see that underlying terrain you know there are these peaks and valleys we know they're there but they're sort of like a fog of war right in that we can't see just kind of beyond our local environment because we just don't know about that much about what haskell mountain looks like or what you know uh some other language or framework that you're not familiar with um so in order for us to really get that sense of what the terrain is like we have to share information with each other about what the terrain is that's that's how we learn uh what happens if we go out far enough into the terrain whether the costs are going to be worth it so different cultures have different amounts of information sharing and that makes it harder or easier for them to converge on different things so uh if you if you imagine this if you imagine a graph of different cultures you can sort of graph them on how close versus open they are in terms of information sharing so if you look at a very very closed system a good example of this is the fraud industry so if you're a fraudster uh then actually it's very hard to learn and get access to the information that you need in order to learn how to become a fraudster right so they're all these underground fraud industries so fraud is an industry in many places in the world uh where basically you can you can you know get access to uh courses you can buy them you can get uh primers on how to hack this site or that site uh either various tools you can buy you can pirate whatever there's all the stuff you need to get up and running as a fraudster and it's not easy to get this stuff uh you actually have to you know make your way into communities you have to prove yourself you have to gain reputation uh you can't just decide hey you know i'm gonna go on amazon and buy a textbook on how to commit fraud right it doesn't exist you can't do that you have to go in through a very specialized way and not all information is actually readily up for grabs there are some fraud rings that just don't share that information with anyone outside of it and it's not for sale and so that makes it very difficult if you're somebody who's wanting to learn more about fraud to actually figure out what is the optimal place for me to be spending my time defrauding people uh now somewhat more on the open side is you could look at a world like poker so poker is kind of a more open system you know so they're all these forums there are different places where people can exchange ideas there's certainly books written about poker that you can just buy if you want to uh but the very best players the very best ideas the very best theories and strategies about poker um they're generally not for sale you know the the people who hold them and the people who profit the most from them tend to keep them close to the chest so you get a lot of uh you get a lot of resources that are okay or that are really crappy that are openly available but the very best stuff sometimes is hard to find and hard to actually gain value from uh and then on the other hand if you look all the way to the right you find the world of software the worm software is in a lot of ways kind of staggeringly open you know you have companies that are just releasing the source code for their entire application uh or security libraries that are again like completely open source and companies will just say yeah we use openssl groovy if you find a weakness in openssl that's a weakness in us right and this is this is really about as open as you can get and there are blog posts there are all these things that are shared about about software that make it seem like wow there's an enormous amount of information sharing that should really make it so that people see really quickly what is actually the best solution for any different problem somehow in the world of software it doesn't really seem like that happens all that well uh and so i don't know i think there's somewhat of an open question of even though there is a lot of sharing on the surface uh if in fact there are some things that people aren't that open about sharing uh like you know i think uh when it comes to what a lot of large companies are doing when they're putting together a lot of open source solutions to solve problems they actually don't immediately go out and tell people oh hey we solved this problem here's how we did it very often the way that companies share this information is pretty selective and pretty strategic and the moment you solve a cutting edge problem you generally don't go out and share it unless you think there's some strategic value in doing so and so that i think to some degree kind of exacerbates the problem of why is it that we don't get this convergence in the world of software so reason number four and i think this is a really interesting reason that kind of goes to sociology is basically the problem of group identities and we heard some this morning about dhh from dhh about the value of group identities and i'm kind of going to go at it from a completely different angle i'm going to talk about more the dangers of group identities so you can imagine that the world of programming is kind of you know it's demarcated into these different kind of arbitrary groups okay one of the groups might be rubios or you know rails programmers or whatever which you can categorize myself in there uh then you have java lovers over here and python is over here and then scalas over there and it's they're kind of these special norms that dictate what you can do inside these different worlds if you want to fit in into these groups you know these groups kind of say like well if you're a java lover uh you can explore this area but you're not really you're not really supposed to go over there that's kind of weird stuff that we don't really do in java land right uh and so you get these kind of arbitrary cuts across the terrain that make it hard for you to just freely traverse and explore this terrain without violating some kind of uh social norm associated with your group turns out uh you get the same kind of thing with fraud rights right so there's a fraud ring that just defrauds facebook that's all they do and all they do is you know they're talking on some sort of you know secret channel where only fraudsters that are part of this group can communicate and they share information about just how to defraud uber or just how to defraud google and if you're one of the members of the other groups maybe you can't get into that group maybe there's just broadcasters have just decided nope we are we're this group and you're that group and you're not going to get our information and so if you want to explore you only get to explore the terrain over there you know and i think in the world of software you kind of see this when uh you know instead of having like an explanation a blog post it's just here's how kafka works there's the blog post kafka for rails engineers right or the blog post you know xyz for rubyists and this again is like kind of uh reinforcing that demarcation that hey you know i know you want to go explore that stuff let me show it to you in the way that's appropriate for our group and i think this is really fascinating to me because you know being somebody who's relatively new to the subculture of software uh i can immediately recognize this behavior and i think it's pretty well explained by this theory in psychology called social identity theory so the idea of social identity theory it's pretty simple uh essentially uh suggests that the way that we construct our identities as human beings is largely as a result of the groups that we adhere to so uh this kind of goes in several stages so the first thing you do is you start categorizing the world into social groups okay so you first have to say okay so these people are the christians these people are the goths these people are uh the meat lovers okay whatever whatever you want to however you want to draw up those boundaries in the space of what people can be so first you have to draw those boundaries next you have to identify which of those groups you belong to you know do i want to be do you want to do i want to be a meat lover do i want to be a pythonista do i want to be somebody who loves red or whatever you have to decide which of those groups you're going to identify with okay then once you do that the last step is social comparison now you have to do the pretty hard work of deciding why the other groups are bad and your group is good you have to make this distinction between your in-group and the group and invent some kind of story or narrative that goes along and reinforces why you're good and they're bad so you know uh there are all these classic examples of this sort of thing where basically there's some arbitrary distinction that you've arrived on as being important to your social identification and you know there's no intrinsic reason why that should be important but you know we're rubyists and they're java lovers and because of that they're bad and we're good and we have to come up with some sort of story why that's the case and they have to do the exact same thing now you might think the social identity theory would suggest that okay well that should mean that all rubious and all rails developers are the same but i don't really feel like that i'm not the same as the people around me you know if you look around other people around like you know you all don't look like a completely homogeneous group of people uh and this is this is true um and so there's this other theory that kind of complements us really well and it's called differential psychology and differential psychology essentially examines the way that people within groups try to make themselves different from each other as a way of somehow strengthening their bond as being a part of that group so for example if you've ever seen the movie west side story so you look at these characters they're all part of a gang in west side story and they all you know they they're all together part of the same group uh and you can tell they kind of have a look right if you just saw these people in the street you'd be like okay these people are doing something together there's some something that somehow unites them um but notice they don't all wear exactly the same outfit and they could they could all wear the exact same outfit they could all style their hair the exact same way but they don't why don't they why don't they do that you'd think that maybe that would strengthen their group identity if they all literally did the exact same thing they would be a stronger part of that group but it turns out there's something intrinsic to us as human beings that even though we're a part of groups it's important for us to differentiate ourselves we actually spend a significant amount of energy just differentiating ourselves within the groups that we're in as a way of almost masking our identities within that group right as a way to not make ourselves feel that like hey i don't have any identity outside this group because i seem to be wearing the exact same thing everyone else is wearing and doing the exact same thing everyone else is doing we'll expend a lot of energy in order not to feel that so uh that's exactly what you see these people in west side story doing and so i want to draw a little bit of analogy here is that there's something kind of similar going on when you look at something like this where we're expanding this energy you know if you imagine this underlying terrain of you know software and let's say this right here is rails mountain uh there's a lot of energy going into making it so that you know it kind of looks like we're exploring this big terrain but really everything is actually still in the bounds of this group even though we're talking about you know kafka or elixir or whatever it is we're still keeping you as a part of this group and that identity is actually reinforced by you being here in every single way even the fact that you're going to these different talks talking about different technologies uh you're still seeing it as a rails developer and i think this is bad i think this masks the the severity of the problem of social identities making it harder for us to actually converge and actually find what is genuinely optimal it allows us to kind of distract ourselves with a story but like hey we're exploring these different things but really underlying it we're not so i think we should really want to find convergence we should want to find the true global maximum so as software engineers what what are we to do about this and i don't know that i have perfect answers to this i think these are all really intrinsically hard problems but i do have a couple of pieces of advice that uh hopefully might be instructed to some degree so the first piece of advice is uh an adage from paul grammer initially where he said keep your identity small and really kind of what this means is to as much as you can as much as possible jettison the labels that you've that you've very easily come to identify yourself with and so that's to say you know don't think of yourself as a rails developer or as a ruby developer but instead think of yourself as a software engineer such that whatever ends up being the right tool for the job and that tool might be rails my tools might be ruby might be something else uh that's what you fundamentally use you solve problems in the world of software and right now it might be very beneficial for you to go climbing up this hill of learning more about rails or more about ruby uh but eventually you will not be you can imagine 10 years from now you'll be working on something and rails probably won't be the tool you want to use 10 years from now in fact i would right now i actually consider myself to be uh i love ruby i love rails i think they're really awesome and wonderful tools uh but i would be probably pretty disappointed myself if i was a rails developer 10 years from now and that was what i considered myself to be i'm a rails developer you know when when dhh was talking about uh the article about cobalt programmers that you know they're people still making money for banks working on these super antiquated cobalt uh uh applications you can bet that there will still be rails apps 10 years from now and i'm sure that you'll probably be able to fetch a pretty penny uh you know basically managing these 10 year old 12 year old 15 year old rails apps uh but is that fundamentally what you want to be doing or is what you want to be doing to solve problems with software however those problems end up changing and whoever those tools end up changing the second piece of advice i want to give besides keeping your identity small is pretty obvious which is to explore the terrain and exploring the terrain to me it means more than just kind of you know paying lip service to different things they'll be like okay i don't know what kafka is i'm going to go to this talk or i'm going to go to that talk it means fundamentally to do things you've never done before okay it means to do things that are kind of scary to you it means to take real risks and when i say real risks i'm juxtaposing that against fake risks which i think are a real thing and something you should caution yourself against a fake risk is one where you actually retain all of your safety all of your comfort all of your prestige all of your knowledge all of your abilities where it's like you know what i'm still a really awesome person everyone respects me i know everything that i'm doing uh but i'm also taking this risk no you're not no you're not right the risk comes when you give up you actually walk down the hill and walking down a hill is uncomfortable it's scary it makes you nervous and if you're not actually doing that then you're not really taking a genuine risk uh another way of saying this is go to djangocon uh what i don't actually go to djangocon it probably sucks but there's probably like if if if railsconf is the only conference you're going to this year reflect on that reflect on what that means whether it means that you are actually taking risks that are important so uh and finally of course i think the most important part of exploration is just to have fun and when you let go of the idea that you constantly need to be moving up and then in fact it's okay to move down and to uh take risks in a way that potentially make it harder for you to get your job done but that's okay and that and allow yourself to have fun doing it and i think that makes the whole process a lot easier so uh that's it for me i'm haseeb qureshi a software engineer on risk at airbnb uh if you're a senior software engineer or a data scientist we're always hiring and yeah thanks for listeningtoday i'll be giving a talk on why software engineers disagree about everything okay uh my name is haseeb qureshi i'm a software engineer at airbnb i work on the risk team i'll be talking a lot more about that and a lot more about kind of my own history and the different uh different things that i've done in my past uh because that really informs a lot of what this talk is about uh i don't know that i'll totally answer the question of why software engineers disagree about everything but i think uh this talk is gonna be kind of an exploration of this question from a number of different angles and another things can happen in this talk is i'm going to disagree pretty strongly with dhh's keynote so if so that might be interesting but we'll hopefully i'll at the very least if not convince you of what i'm saying uh you know maybe get you thinking about some things so uh this talk in large part is going to be about philosophy so i know uh you know it's a railsconf usually people come up here and they talk about controllers or i guess someone else is talking about sorting and augmented reality uh i'm not gonna be offended if you get up right now and want to go to another talk that's totally cool but uh we are gonna go way into the weeds today because that's that's what i'm all about so uh specifically i want to talk about the field of philosophy known as epistemology and epistemology the easiest way to define epistemology is epistemology is the nature of knowledge of justification and of the rationality of belief uh so this actually this diagram here is uh of uh phrenology which is this old science of you know trying to figure out what parts of the brain responsible for what which is a very old and ancient version of epistemology which we've now thankfully displaced but essentially in epistemology are the two questions how does anyone know anything and how do they know that they know it uh i mean these two questions are kind of navel-gazy but that's that's fine um so let me tell you about me so i studied english in philosophy in school and i before i ever came into the tech world i used to be a professional poker player and i did that for about five years uh so you know i kind of took a very different path into programming than most people uh and then i worked as a programming instructor i taught programming at a coding bootcamp and for the last a little bit over a year i've been working as an engineer uh on the risk team at airbnb uh basically fighting fraud and so because of this i've kind of you know snaked my way through many different subcultures i've been something of a of a chameleon in that i've learned the different norms and the different beliefs and the different knowledge systems that many different worlds uh have and try to assert on you if you become a part of that world and one thing that i've noticed and you can't help but notice the more worlds you step in and out of is that knowledge is deeply cultural so in the world of programming you know the kind of things you might hear people tell you when you come into this world they might tell you that full stack javascript is the future they know this for sure uh they'll tell you that everyone should know c if you don't know c what are you doing go out there crack open a book and learn some c they'll tell you rust is the best language for systems programming they'll tell you that relational databases don't scale they'll tell you that tdd is a fantasy and nobody serious tries to do it anymore it's it's not it's not practical so now all these are interesting questions and they're all things that people disagree on obviously uh but they're people they're things that people hold with very very strong conviction that this is clearly and just irrefutably true now i'm not interested in convincing you of any of these claims each of these could probably be great talks themselves what i'm interested in is why we disagree about them why isn't that we all converge on an answer to these questions now it's funny because when i was in the poker world i remembered this this was very familiar to me when i was learning how to be a poker player because in the poker world people will tell you things like no limit is dying you have to learn mixed games or they would tell you everybody needs to use a hub or they would say only fish play loose passive styles they would say gto is a fantasy nobody actually plays like that and there's a wonderful analogy between these these two worlds not just in that people argue a lot and then people are maybe kind of whiny and excessive but that you know they're they're all these things that people fundamentally disagree on and each side of them holds tremendous conviction that their side is clearly and obviously correct um and so when most people hear stuff like this when they hear someone say hey everybody should know c and if you don't know c you're not a real programmer their natural reaction is oh god is that true if that's true what should i do about it uh but i for me having having gone through this song and dance so many times my reaction is why do they think they know that and why what is it that implanted so much confidence in them so much conviction that the thing they're saying is actually true and universally true and that's very interesting to me it's almost like i'm asking uh i tend to ask a more evolutionary question than a question about the world as it is right now you know the thing that's kind of weird about programming in particular is that nobody agrees you know there's so many things that people disagree on they disagree about functional programming object oriented programming tdd old you know robust proven frameworks shiny new ones that solve new problems in different ways serverless event driven blah blah blah right you guys you guys know all this uh nobody agrees why that's really weird now you might think it's obvious that people don't agree that's that's an understandable reaction you might say like well you know disagreement is a normal part of a society and like you know the way that we uh you know this this divergence is a source of discourse and that's how we end up having a healthy society so you might think that's obvious so of course people shouldn't agree um i think it's weird i think it's weird that people don't agree and i want to develop in you an intuition to also believe this is weird that people don't agree because in a way it makes more sense for people to agree than for them to disagree another way of putting what i'm saying is that systems in general tend to converge when you see a system you should assume that over time it's going to converge on what's optimal what's the sort of the optimal state for that system uh let me give you an example so i'm going to give you an example related to poker but you don't need to know anything about poker in order to understand this i can explain it in a couple sentences so in poker there's a strategy called set mining okay and set mining is very simple so in texas hold them you get two cards and if you want to set mine what you do is you wait for two cards that make a pair so let's say you get delta pair of threes then what you do is you wait to see if you make three of a kind if you make three of a kind you bet it really aggressively and if you don't you just fold that's set mining super super simple um so set mining was a strategy that was pretty simple and pretty stupid uh almost anybody could do it and it worked it worked really really unreasonably well given how simple the strategy was uh and so pretty soon what you would see in the world of online poker is that pretty soon once people started talking about this almost everybody started set mining at low to mid stakes full ring games the strategy just kind of took over like wildfire and what you could say is that the game converged on set mining everybody saw that set mining was the high ground and they all moved in that direction and uh so this is this is kind of one of the features that happened to poker uh after internet poker took off so internet poker took off really around 2004 2005 uh maybe a little bit before that as well and if you think about it before internet poker ever existed uh there was live poker right but playing poker in a brick and mortar casino and live poker was fundamentally really different than online poker in ways that people didn't really predict before it happened so you can imagine uh live poker you know let's say in the 80s and 90s you know uh there are some people playing poker out in phoenix maybe there are people playing poker in vegas that people play poker in dallas people playing poker in london um and these different groups of poker players weren't really communicating with each other you know ideas that were germinating in london weren't you know going over the pond and uh showing up in card rooms in dallas right people were kind of isolated to their little groups and they were just playing poker and figuring things out as they went but with the advent of online poker and the communication that it enabled for a lot of people uh sorry there was like a flyer running around with the advent of that communication what happened was it allowed the system to converge where suddenly somebody has an idea for a strategy somewhere in you know dallas or new hampshire or whatever and suddenly they can share it and everybody can learn about this strategy very very quickly uh and so what this meant was that the the state of poker strategy for a long time it was pretty static the way that people played in the 40s and 50s was not that different from the way people played in the 80s and 90s you know information just wasn't able to evolve that effectively but with the advent of online poker suddenly if you if you look at the curve of the complexity of poker strategy it just it just takes off right after 2003 when online poker comes to town uh and so right here uh is a graph uh that audio's not playing but that's fine he's saying look at this graph uh this is a joke uh that didn't play audio that's fine we can we're gonna roll with it um so this is actually a graph of uh this is actually just a graph that goes up into the right there's nothing to do with what i'm talking about but i thought it would give a sense legitimacy to the slides like oh this guy knows what he's talking about look at this graph um cool so the point is the point is online poker converged in a way that that live poker never did it was never able to um and this to me is it makes a lot of sense like of course online poker is going to converge right when you have these people able to communicate to see what each other is doing and move around in this terrain of course they're going to settle on what's optimal um and nature is full of convergent systems like this so let's say for example you're you're walking downtown and uh you you buy a loaf of bread maybe you're french i don't know uh and uh let's say you you eat half a loaf of bread and you toss the rest on the street well what happens is pretty soon a bunch of pigeons just it's not like they're all in one place standing over watching you throw the loaf of bread right but pretty soon one pigeon comes three pigeons come and then the a chorus of pigeons from presumably all over town just come in and start you know just just binging on the piece of bread that you've left uh and this this kind of makes sense right like the pigeons are able to very quickly come down to whatever's the optimal place for them to be they they eat as much food as they can and they disperse back to wherever was they came from maybe to the next best piece of food that they can find somewhere downtown um so the example of a convergent system so uh you know there's also uh this great ram scene album that actually i loved when i was a kid um it's not this is actually just cell membrane in german because that was the one i could find like 10 minutes ago uh but uh the the the uh cell membrane you guys might remember from high school chemistry you know there are water molecules on one side they permeate through this membrane and pretty soon uh you have the exact same pressure the exact same uh density of water molecules on each side of the membrane the system converges and you see the same sort of thing in stock markets a lot of natural phenomenon uh it seems like this sort of thing is everywhere so when i started working as a risk engineer at airbnb working in the fraud space naturally i started looking for convergence too because it just seems like one of those things that any any sufficiently uh complex system should eventually have some optimal state and it should find convergence so uh working in in the fraud industry is uh is its own little subculture its own little world uh and the subculture is not just the subculture of people working against fraud which is which is interesting but that wasn't really what fascinated me so much i mean it is very interesting but the really fascinating thing about fighting fraud is that you're actively fighting against a community of people you're actively fighting against a culture that's optimized to take you down and to basically exploit your defenses as effectively as they possibly can uh and so you know really there is actually a subculture that i can't see and don't have direct access to that's organizing and trying to attack all these you know major online companies uh and they're trying to make money and that's really interesting uh and so i wondered you know how do fraudsters figure out what to do i mean what they're what they're learning is pretty non-trivial right like they're learning how to how to script how to do you know sometimes cross-site requests forgery stuff xss whatever uh they come up with all these different ways to try to probe our defenses and attack us and ideally try to make some money how do they learn this and how do they communicate this to each other you know how does how does this knowledge spread in the world of fraud fundamentally the question i want to ask is is there convergence in fraud and this actually uh pretty soon after the new macbook pro replaced the escape cube we saw fraud go way up so that was uh one person after that that was all right there we go there we go a little just a little bit that's all i need in order to keep going uh cool uh awesome so i just want to run with you through a quick example of kind of what uh a standard fraud scheme looks like this is this is some site called chipmunk does anyone know what chipmunk is anyone familiar with this with this esteemed company great then we're gonna on them that's great uh so chipmunk uh let's say they cut some corners they kind of implement a feature they don't really know uh exactly uh you know they're not really thinking about fraud when they implement this right like some product manager is like hey you know we've got the story we've got to get up the end of the week uh we're gonna all look good once we fill this thing out great engineers who work on this just push it out don't really think too much about it so this feature is essentially uh some micro deposit uh where essentially they verify that you own this account because you they deposit two random amounts you report them back right pretty standard for some financial institutions to do this sort of thing um so let's say that they don't implement rate limiting easy thing to overlook uh well you know they go to sleep that night feeling great that they've launched this new feature and pretty soon someone gets alerted uh you know let's say at 2am that night that uh somehow we've lost a hundred thousand dollars which really should not have happened and the only way you could lose that amount of money obviously is if fraudsters are at scale repeatedly hitting this api repeatedly uh micro depositing money to their account and then just keeping it and so people scramble you know people like oh my god i can't believe this is going on they go and they try to patch this hole implement some rate limiting do some rules maybe reverse whatever transactions they can this is a common type of attack that you know a very easy vector that if you're a site influence something like this uh a kind of attack you would be vulnerable to if you didn't implement rate limiting so let's say they get that under control that's fine now what happens is that once you patch that vulnerability of course the fraudsters disperse they don't keep hitting it maybe they maybe they try to verify that in fact it is patched they try a few ways around it doesn't work cool you've patched the vulnerability and the fraudsters disperse but kind of like the pigeons they don't just go randomly to different places and just kind of you know leave your platform what ends up happening is that they then go to the next best place to defraud you so what they'll do is they'll find okay that endpoint wasn't that good but there's another thing that we can do like this may be longer more complex or more expensive fraud scheme uh and we're gonna go do that and basically what you see is that the fraudsters head down to the next highest peaks in this terrain of fraud which you know this this totally makes sense this is uh this is convergent behavior right this is exactly what you'd see if the terrain suddenly changes peak became a valley and they would go towards the next high speed so this seems to me like convergent behavior it seems like fraudsters converge on what's optimal but the more that i thought about this the more i realized that this didn't totally make sense and the reason that this totally didn't make sense because i asked the question why are they defrauding us at all why are any fraudsters attacking chipmunk this might seem like a weird question right like of course they're doing it so that they can make money and that's what fosters are motivated by but if you think about it the the terrain of all the companies that they can defraud is huge there are many many many companies that are vulnerable to fraud and pretty much all of these companies experience fraud to one degree or another right they're all these different sites that you could attack if you so wanted um and so what you would what do you you know what you can say is that there's some topology to that fraud right there's some peaks and valleys there's some there's some sites that are really really lucrative to the fraud and other ones that you know are really not so much and they're not a good place to be spending your time if you're a fraudster and so you'd think that everybody would converge on the peaks what you think is that almost all the fraud would go towards the very most profitable most uh attackable websites but what you see instead is that fraud is just kind of dispersed everywhere there's just a sort of ambient level of fraud that if you start an e-commerce site chances are you're going to get some amount of fraud why this is weird this should strike you as weird why is it like those fraudsters just don't care about optimizing are they not trying to make the best use of their time is there something that they're misapprehending about the terrain like what's going on that we don't see this convergent behavior within fraud now you know it occurred to me that actually you see the same sort of thing in software you know even just looking at open source solutions there's so many different software packages and so many different open source solutions to many many different problems that are effectively you know trying to do the exact same thing why it seems like software doesn't converge either when you when you don't get this just one okay this is the best way to solve this problem and everybody converges on that solution uh what you see instead is many many many competing solutions uh and it's not really clear which one is supposed to win and when one does win it's actually really surprising you know like so you know react you could say kind of one front end uis and so far as you can say anyone won front end uis uh and we're genuinely amazed by that on some meta levels like holy someone won no one ever wins anything anymore but react just one front end uis you know and you know maybe maybe that kind of relaxes you because you're like oh hey you know i just as long as i learn react i'll just be employable forever and you know that's fine uh but i think this is this is somewhat counterintuitive that more things aren't like this why is this such a rare story that react wins or sequel wins or one particular way of solving a problem just is clearly the best and we all adopt it so you might have uh an obvious objection to this analysis which would be that well of course you know things aren't going to converge because software doesn't just solve one problem right uh the obvious answer is that there are actually multiple terrains so you know you could say that there's a terrain for crud apps and that looks different and there's a terrain for the payments back ends and then one for social networks and maybe the tools and the solutions that you use for these different apps are different and that's why you see this this you know uh these multifarious solutions to effectively the same problem um but i i don't think that's that's sufficient because even within a single terrain you don't see convergence you know even just looking at crud apps right cred apps are you know they're the majority of what people build the majority of web apps are just you know glorified cred apps maybe with like elasticsearch store on top and like because of the vast majority of what people build there's a vast majority of what people disagree on so it doesn't seem to me like that's satisfactory to explain why we don't see convergence and the thing is you should want convergence convergence is actually good because convergence means that we all see the underlying terrain we all understand it and therefore we all go and do the best thing in the world of software for the most part we're not actually competing with each other we're actually all kind of at least to some degree or another motivated by each other succeeding you know when someone invents a great open source solution actually everybody benefits from that and most software kind of works this way so the question i want to ask is why do these systems not converge i think there are four reasons and i'm going to go through each of them so the first reason why these systems don't necessarily converge is because the terrain is actually unstable the terrain is changing it's not just one configuration that you can just see something out in the distance decide okay that's a peak i'm going to go over there right the fact that the terrain is changing means that you're not really sure if you do uh go in this direction that by the time you get there the terrain isn't going to be different right it also means of course that your terrain is moving underneath you so by the time that you when you originally came into ruby on rails like wow this is the hottest newest shiniest web framework anyone has ever come up with and now it's 2017 and rails is not the shiniest newest coolest framework that anyone's ever come up with uh and so that terrain has changed underneath you and that makes it hard for people to actually converge when there's so much change going on uh there was this article that someone wrote on medium a little while back how it feels to learn javascript in 2016. um you know if you didn't read this article i'm sure you have an intuition of what this thing said right and we we see this becoming more and more uh i don't know if i want to say a problem but more and more of a characteristic of software that things are changing very rapidly and it might be that that pace of change is even increasing which wouldn't be wouldn't be that uh unreasonable to expect actually as technology grows more and more rapidly um and you see the same sort of thing in the world of fraud so say for example uh that you know facebook has some kind of hole some kind of uh things that fraudsters can attack let's say it becomes really easy to spam facebook and you know do some kind of referral quick fraud whatever whatever stuff um so maybe you know if you're a fraudster you're making 20 an hour defrauding facebook doing this well facebook goes in and they patch the hole and now you know instead of making twenty dollars an hour you can only make three dollars an hour um but in order for the fraudsters to disperse and find the next highest thing uh you know they they they might be just incentivized to say like hey i don't actually know how easy it's going to be to defraud anything else because that might get patched too and that might you know uh the terrain might just change out underneath me so you know what i'll just keep defrauding facebook for three dollars an hour uh that's fine like at least i know this is working during the time that i have it so maybe that's that's some part of the explanation why you don't see this optimizing across across the terrain so the second reason why i think you see a lack of convergence in some of these domains is because of high switching costs okay so uh let's say that this is the domain that uh let's say this is software this is the map of the software world uh and let's say that you live uh all the way on the bottom right and the bottom right let's say is railsland and you know you go and talk to some uh wide-eyed bushy-tailed developer who tells you you know what you you just need to learn haskell and if you learn haskell you know all these type errors and mill checks and that all that stuff is just going to go away and you're going to live in a land of just pure programming bliss nothing will ever go wrong your code will be way more performant you know you are just going to yeah whatever things are going to be great uh the problem with that of course is that for you to actually engage in that switch let's say even you see what this person tells you and you agree with them but this peak that they're pointing to is very very far away so in order for you to get there you're gonna have to go down into a valley traverse a really really long space until you can finally actually reap the fruits of what they were claiming was so great and who knows because the terrain is changing by the time you get there it might not even be a peak anymore something else might have become a peak or maybe the place where you were at could have become a peak who knows uh and so this this instability and this uncertainty makes people really unwilling to incur the risks of traversing the terrain and exploring and you see of course the same thing in fraud just like i just mentioned with the facebook example and this whole thing of course is exacerbated by specialization the more specialized you are the harder it's going to be for you to convince yourself to engage in those high switching costs right because really specialization is just basically you finding your way to some local maximum that's what specialization is it's climbing as high as you can onto a local maximum once you're there it just becomes really uncompelling to climb all the way down from that peak of specialization you've arrived at to go find the true global maximum which might be very far away and you know we've already talked about the fact that it's changing and uncertain uh and so this makes it harder the more specialized we are same thing with fraudsters there are fraudsters who are specialized in you know attacking one side as opposed to another and it's hard for them to switch they have to relearn a lot of new things they have to start over in their knowledge so now the third reason why i think you don't see convergence in fraud or in software and these other fields is information sharing is a very important part of how you get convergence right if you think about it not all of us can actually clearly and lucidly see that underlying terrain you know there are these peaks and valleys we know they're there but they're sort of like a fog of war right in that we can't see just kind of beyond our local environment because we just don't know about that much about what haskell mountain looks like or what you know uh some other language or framework that you're not familiar with um so in order for us to really get that sense of what the terrain is like we have to share information with each other about what the terrain is that's that's how we learn uh what happens if we go out far enough into the terrain whether the costs are going to be worth it so different cultures have different amounts of information sharing and that makes it harder or easier for them to converge on different things so uh if you if you imagine this if you imagine a graph of different cultures you can sort of graph them on how close versus open they are in terms of information sharing so if you look at a very very closed system a good example of this is the fraud industry so if you're a fraudster uh then actually it's very hard to learn and get access to the information that you need in order to learn how to become a fraudster right so they're all these underground fraud industries so fraud is an industry in many places in the world uh where basically you can you can you know get access to uh courses you can buy them you can get uh primers on how to hack this site or that site uh either various tools you can buy you can pirate whatever there's all the stuff you need to get up and running as a fraudster and it's not easy to get this stuff uh you actually have to you know make your way into communities you have to prove yourself you have to gain reputation uh you can't just decide hey you know i'm gonna go on amazon and buy a textbook on how to commit fraud right it doesn't exist you can't do that you have to go in through a very specialized way and not all information is actually readily up for grabs there are some fraud rings that just don't share that information with anyone outside of it and it's not for sale and so that makes it very difficult if you're somebody who's wanting to learn more about fraud to actually figure out what is the optimal place for me to be spending my time defrauding people uh now somewhat more on the open side is you could look at a world like poker so poker is kind of a more open system you know so they're all these forums there are different places where people can exchange ideas there's certainly books written about poker that you can just buy if you want to uh but the very best players the very best ideas the very best theories and strategies about poker um they're generally not for sale you know the the people who hold them and the people who profit the most from them tend to keep them close to the chest so you get a lot of uh you get a lot of resources that are okay or that are really crappy that are openly available but the very best stuff sometimes is hard to find and hard to actually gain value from uh and then on the other hand if you look all the way to the right you find the world of software the worm software is in a lot of ways kind of staggeringly open you know you have companies that are just releasing the source code for their entire application uh or security libraries that are again like completely open source and companies will just say yeah we use openssl groovy if you find a weakness in openssl that's a weakness in us right and this is this is really about as open as you can get and there are blog posts there are all these things that are shared about about software that make it seem like wow there's an enormous amount of information sharing that should really make it so that people see really quickly what is actually the best solution for any different problem somehow in the world of software it doesn't really seem like that happens all that well uh and so i don't know i think there's somewhat of an open question of even though there is a lot of sharing on the surface uh if in fact there are some things that people aren't that open about sharing uh like you know i think uh when it comes to what a lot of large companies are doing when they're putting together a lot of open source solutions to solve problems they actually don't immediately go out and tell people oh hey we solved this problem here's how we did it very often the way that companies share this information is pretty selective and pretty strategic and the moment you solve a cutting edge problem you generally don't go out and share it unless you think there's some strategic value in doing so and so that i think to some degree kind of exacerbates the problem of why is it that we don't get this convergence in the world of software so reason number four and i think this is a really interesting reason that kind of goes to sociology is basically the problem of group identities and we heard some this morning about dhh from dhh about the value of group identities and i'm kind of going to go at it from a completely different angle i'm going to talk about more the dangers of group identities so you can imagine that the world of programming is kind of you know it's demarcated into these different kind of arbitrary groups okay one of the groups might be rubios or you know rails programmers or whatever which you can categorize myself in there uh then you have java lovers over here and python is over here and then scalas over there and it's they're kind of these special norms that dictate what you can do inside these different worlds if you want to fit in into these groups you know these groups kind of say like well if you're a java lover uh you can explore this area but you're not really you're not really supposed to go over there that's kind of weird stuff that we don't really do in java land right uh and so you get these kind of arbitrary cuts across the terrain that make it hard for you to just freely traverse and explore this terrain without violating some kind of uh social norm associated with your group turns out uh you get the same kind of thing with fraud rights right so there's a fraud ring that just defrauds facebook that's all they do and all they do is you know they're talking on some sort of you know secret channel where only fraudsters that are part of this group can communicate and they share information about just how to defraud uber or just how to defraud google and if you're one of the members of the other groups maybe you can't get into that group maybe there's just broadcasters have just decided nope we are we're this group and you're that group and you're not going to get our information and so if you want to explore you only get to explore the terrain over there you know and i think in the world of software you kind of see this when uh you know instead of having like an explanation a blog post it's just here's how kafka works there's the blog post kafka for rails engineers right or the blog post you know xyz for rubyists and this again is like kind of uh reinforcing that demarcation that hey you know i know you want to go explore that stuff let me show it to you in the way that's appropriate for our group and i think this is really fascinating to me because you know being somebody who's relatively new to the subculture of software uh i can immediately recognize this behavior and i think it's pretty well explained by this theory in psychology called social identity theory so the idea of social identity theory it's pretty simple uh essentially uh suggests that the way that we construct our identities as human beings is largely as a result of the groups that we adhere to so uh this kind of goes in several stages so the first thing you do is you start categorizing the world into social groups okay so you first have to say okay so these people are the christians these people are the goths these people are uh the meat lovers okay whatever whatever you want to however you want to draw up those boundaries in the space of what people can be so first you have to draw those boundaries next you have to identify which of those groups you belong to you know do i want to be do you want to do i want to be a meat lover do i want to be a pythonista do i want to be somebody who loves red or whatever you have to decide which of those groups you're going to identify with okay then once you do that the last step is social comparison now you have to do the pretty hard work of deciding why the other groups are bad and your group is good you have to make this distinction between your in-group and the group and invent some kind of story or narrative that goes along and reinforces why you're good and they're bad so you know uh there are all these classic examples of this sort of thing where basically there's some arbitrary distinction that you've arrived on as being important to your social identification and you know there's no intrinsic reason why that should be important but you know we're rubyists and they're java lovers and because of that they're bad and we're good and we have to come up with some sort of story why that's the case and they have to do the exact same thing now you might think the social identity theory would suggest that okay well that should mean that all rubious and all rails developers are the same but i don't really feel like that i'm not the same as the people around me you know if you look around other people around like you know you all don't look like a completely homogeneous group of people uh and this is this is true um and so there's this other theory that kind of complements us really well and it's called differential psychology and differential psychology essentially examines the way that people within groups try to make themselves different from each other as a way of somehow strengthening their bond as being a part of that group so for example if you've ever seen the movie west side story so you look at these characters they're all part of a gang in west side story and they all you know they they're all together part of the same group uh and you can tell they kind of have a look right if you just saw these people in the street you'd be like okay these people are doing something together there's some something that somehow unites them um but notice they don't all wear exactly the same outfit and they could they could all wear the exact same outfit they could all style their hair the exact same way but they don't why don't they why don't they do that you'd think that maybe that would strengthen their group identity if they all literally did the exact same thing they would be a stronger part of that group but it turns out there's something intrinsic to us as human beings that even though we're a part of groups it's important for us to differentiate ourselves we actually spend a significant amount of energy just differentiating ourselves within the groups that we're in as a way of almost masking our identities within that group right as a way to not make ourselves feel that like hey i don't have any identity outside this group because i seem to be wearing the exact same thing everyone else is wearing and doing the exact same thing everyone else is doing we'll expend a lot of energy in order not to feel that so uh that's exactly what you see these people in west side story doing and so i want to draw a little bit of analogy here is that there's something kind of similar going on when you look at something like this where we're expanding this energy you know if you imagine this underlying terrain of you know software and let's say this right here is rails mountain uh there's a lot of energy going into making it so that you know it kind of looks like we're exploring this big terrain but really everything is actually still in the bounds of this group even though we're talking about you know kafka or elixir or whatever it is we're still keeping you as a part of this group and that identity is actually reinforced by you being here in every single way even the fact that you're going to these different talks talking about different technologies uh you're still seeing it as a rails developer and i think this is bad i think this masks the the severity of the problem of social identities making it harder for us to actually converge and actually find what is genuinely optimal it allows us to kind of distract ourselves with a story but like hey we're exploring these different things but really underlying it we're not so i think we should really want to find convergence we should want to find the true global maximum so as software engineers what what are we to do about this and i don't know that i have perfect answers to this i think these are all really intrinsically hard problems but i do have a couple of pieces of advice that uh hopefully might be instructed to some degree so the first piece of advice is uh an adage from paul grammer initially where he said keep your identity small and really kind of what this means is to as much as you can as much as possible jettison the labels that you've that you've very easily come to identify yourself with and so that's to say you know don't think of yourself as a rails developer or as a ruby developer but instead think of yourself as a software engineer such that whatever ends up being the right tool for the job and that tool might be rails my tools might be ruby might be something else uh that's what you fundamentally use you solve problems in the world of software and right now it might be very beneficial for you to go climbing up this hill of learning more about rails or more about ruby uh but eventually you will not be you can imagine 10 years from now you'll be working on something and rails probably won't be the tool you want to use 10 years from now in fact i would right now i actually consider myself to be uh i love ruby i love rails i think they're really awesome and wonderful tools uh but i would be probably pretty disappointed myself if i was a rails developer 10 years from now and that was what i considered myself to be i'm a rails developer you know when when dhh was talking about uh the article about cobalt programmers that you know they're people still making money for banks working on these super antiquated cobalt uh uh applications you can bet that there will still be rails apps 10 years from now and i'm sure that you'll probably be able to fetch a pretty penny uh you know basically managing these 10 year old 12 year old 15 year old rails apps uh but is that fundamentally what you want to be doing or is what you want to be doing to solve problems with software however those problems end up changing and whoever those tools end up changing the second piece of advice i want to give besides keeping your identity small is pretty obvious which is to explore the terrain and exploring the terrain to me it means more than just kind of you know paying lip service to different things they'll be like okay i don't know what kafka is i'm going to go to this talk or i'm going to go to that talk it means fundamentally to do things you've never done before okay it means to do things that are kind of scary to you it means to take real risks and when i say real risks i'm juxtaposing that against fake risks which i think are a real thing and something you should caution yourself against a fake risk is one where you actually retain all of your safety all of your comfort all of your prestige all of your knowledge all of your abilities where it's like you know what i'm still a really awesome person everyone respects me i know everything that i'm doing uh but i'm also taking this risk no you're not no you're not right the risk comes when you give up you actually walk down the hill and walking down a hill is uncomfortable it's scary it makes you nervous and if you're not actually doing that then you're not really taking a genuine risk uh another way of saying this is go to djangocon uh what i don't actually go to djangocon it probably sucks but there's probably like if if if railsconf is the only conference you're going to this year reflect on that reflect on what that means whether it means that you are actually taking risks that are important so uh and finally of course i think the most important part of exploration is just to have fun and when you let go of the idea that you constantly need to be moving up and then in fact it's okay to move down and to uh take risks in a way that potentially make it harder for you to get your job done but that's okay and that and allow yourself to have fun doing it and i think that makes the whole process a lot easier so uh that's it for me i'm haseeb qureshi a software engineer on risk at airbnb uh if you're a senior software engineer or a data scientist we're always hiring and yeah thanks for listening\n"