The Intersections of Privilege and Participation in Online Communities: A Linguistic Analysis

In this discussion, we explored the ways in which privilege and participation intersect in online communities, particularly with regards to language models. The paper's author notes that these models are often trained on vast amounts of data, including social media platforms like Reddit, which can perpetuate existing biases and inequalities. However, the study also reveals that those who participate in these online spaces are not representative of the broader population.

The authors point out that the users most likely to be participating in these communities are those with more privilege and resources. Furthermore, they note that this select group is often predominantly comprised of young people, particularly men, which further narrows the scope of representation. This highlights the importance of considering power dynamics and social inequalities when analyzing online interactions.

One of the key insights from the paper is that linguistics, as a field, must be mindful of its own limitations and biases. The authors emphasize the need for descriptive approaches, rather than prescriptive ones, in understanding language use. However, they also caution that machine learning models, which are often used to analyze language data, can have profound impacts on society.

In particular, linguists recognize that these models are not just passively describing language, but are instead actively affecting it. This raises important questions about the role of descriptive linguistics in relation to design and engineering, as well as normative thought. The authors argue that a more nuanced approach is needed, one that takes into account both the complexity of human communication and the potential consequences of technological solutions.

The discussion also touched on the value of specificity in language analysis, particularly when it comes to mitigating biases and promoting inclusivity. The authors suggest that context-specific solutions can be just as effective, if not more so, than general-purpose approaches. By focusing on specific contexts, researchers and developers can better understand the social implications of their work.

Furthermore, the study highlights the importance of considering the trade-offs involved in bias mitigation. While it is tempting to simply say "don't use racist or sexist language," this approach can have unintended consequences, such as perpetuating stereotypes or limiting the scope of what is considered acceptable. The authors emphasize the need for a more nuanced approach, one that balances the desire to promote inclusivity with the risk of over-broadening or oversimplifying the issues.

In conclusion, the intersections of privilege and participation in online communities are complex and multifaceted. By recognizing the limitations of language models and the importance of context-specific approaches, we can work towards creating more inclusive and equitable digital environments. As the authors noted, linguistics is not just about describing language, but also about understanding its social implications and making informed decisions about how to use language data.

Parting Thoughts from the Q&A Session

The discussion also touched on the idea of specificity having real value in language analysis. The author emphasized that, as researchers, we often prioritize scale and generalization over context-specific approaches. However, this can lead to a loss of nuance and a failure to account for the unique social implications of digital technologies.

The author also highlighted the importance of considering the trade-offs involved in bias mitigation. While it is tempting to simply say "don't use racist or sexist language," this approach can have unintended consequences, such as perpetuating stereotypes or limiting the scope of what is considered acceptable.

Finally, the discussion emphasized the need for a more nuanced approach to understanding language data and its social implications. By recognizing both the complexity of human communication and the potential consequences of technological solutions, we can work towards creating more inclusive and equitable digital environments.

"WEBVTTKind: captionsLanguage: enall right everyone i am here with meg mitchell ai researcher and emily bender professor at the university of washington and we are here to talk about their recent paper on the dangers of stochastic parrots can language models be too big meg and emily welcome to the swimwell ai podcast thank you thanks welcome thanks super excited to jump into this conversation let's start out like we always do with having you share a little bit about your backgrounds meg we'll have you go first uh yeah i um basically studied as a computational linguist like emily i got my phd in computer science um and then i've worked at johns hopkins microsoft research um and most recently google research um i've worked on computer vision as well as natural language processing computational linguistics and more recently issues of bias and fairness and ethics and ai awesome and emily you were on the show not too long ago when we talked about is linguistics missing from nlp research give us a little bit of your background and maybe a catch up on what you've been doing for the past year all right thanks and i just had to close my window against the sun um so like you said i'm a linguist studied linguistics at uc berkeley and then stanford and then i got to come to the university of washington and start the professional master's program in computational linguistics where meg was a student many years ago now and i got to know her then and so i work you know largely in linguistics and computational linguistics but since about 2016 i've also been working in the space of um i've tried to avoid the phrase actually ethics because it sends people down these paths into philosophy that i find somewhat less helpful but societal impact of nlp and in that context a lot of the same discussions are relevant that are relevant to other things that fall under the umbrella what gets called ai um and yeah in the past year um have continued that work and had this great opportunity to work with meg and other members of her team and in particular dr timmy gabriel and a phd student of mine angie macmillan major on a paper looking at the impact of large language models and you know we started this in september and submitted in october and it's the first paper i've ever written that has been far more work after it was finished than writing it in the first place maybe it's kind of success we should probably come right out and say that there has been a lot of conversation about this paper a lot of that unfortunately for reasons that we're not going to go into here um but you know we're going to focus on the the paper and its importance but i'd love to hear kind of the back story and how it all came together what was the the genesis of this this work so the the genesis for me sort of where i came into it was a twitter direct message from tim neat who said hey have you ever written about this or do you know of anyone who has um and i said no sorry don't have any um particular papers that i've written or anything to point to you know why and she said well i keep having these conversations where people would like resources about what are the possible dangers of large language models and i've been pointing them to your tweets i thought okay which tweet since she sent me some links um but then i i said well you know here's the dangers that i can think of and it started feeling like a paper and so a couple days later i later i sent her an outline for the paper and i said hey you want to write this and this is early september it might be good to submit to fact deadline in early october kind of close and she said i don't know that's kind of close but let's see and so we put together this overleaf document and tim neat uh brought in some people from her team and i brought in my phd student and everyone just started contributing to the outline um and filling bits in and it was this group of people that had a really richness of diversity of scholarly backgrounds and so we were able to draw on literature from many different fields and it came together just remarkably quickly um and we got it submitted in time and you know i guess the rest is history but that's so that's how it's gone from my side how about you meg uh yeah i mean so on my side um timmy and i you know have both we're both working google internally on uh opera operationalizing uh ai ethical procedures basically and part of that is figuring out harms and risks of different technologies and being able to transparently report them so i've had this thing i've worked on called model cards where you have to have some sense of intended use and also unintended use and the sort of potential harms associated with those emily has something similar with with data statements and we were really trying to get to a point where we could provide good consulting on these kinds of technologies openai was asking for information um internally there was just a ton of desire to have basic technologies associated to benefits harms and risks and so i really thought that this is language modeling is so fundamental to ai we're seeing it more and more across the board we really need to do a due diligence task for the harms and risks this will help us with further operationalizing ethics internally this is critical for any self-regulation argument that a corporation wants to make about ai they have to demonstrate due diligence for harms and risks for various technology um and also i wrote my phd on language generation so if me didn't include me i was gonna kill her yeah so that was what happened on my end so the the kind of framing for the paper is on language models and the the increasing size of language models um yeah maybe let's start with that in terms of you know for backgrounds you know tell us a little bit about when you think about language models and kind of the the size you know what are the examples that come to mind and um you know where do you see the trajectory going yeah so i think the first thing i want to do is give a little bit of historical context so language modeling is a really old core technology in natural language processing um you know it goes back to ideas that shannon had about the distribution of words and texts and the probabilities of given a substring what comes next and you had language models as a core component of speech recognition and then the very first statistical machine translation systems also used language modeling with this so the using the noisy channel model which goes back to shannon this is a wonderful ridiculous idea where um with speech recognition it's like okay the person had some underlying thing they wanted to say in their mind it got pushed through this noisy channel of the vocal apparatus and then the acoustics in the air and then our ability to capture that let's guess what the most probable underlying string was given that noisy output and it's done in two parts looking at the probabilities of a underlying string giving rise to certain acoustics and then also just the probabilities of that string being a good string in the first place and that probability of string being a good string that's the language model part in machine translation it's kind of ridiculous the metaphor was okay let's take this idea from speech recognition and apply it to machine translation so those people speaking french really had english in mind and it went through a noisy channel and came out as french instead what's the most likely underlying english which is a stupid way to think about translation actually you know was useful and worked well so that's that's like old school language models and initially it was just you know um okay probabilities of one word at a time unigrams bi-gram probabilities given one previous word what's the distribution of likelihoods of the next one up to three four five you guys sort of maxed out at about five or six meg do you know with the engram language models yeah yeah it starts to become uh it looks somewhat fluent at four but then you realize that it's like syntactically uh and semantically completely nonsense and then five yeah yeah and and it just gets really really hard to um just hold on to all of that data if that's the model that you're using and so that was sort of an important piece of things in these tasks that involved manipulating language form and coming up with fluent sounding strings and so just generating from a language model like mike says you sort of like sequences of any any window of four words kind of makes sense but longer than that it loses coherence but if you're using that to select among outputs that came from something else like in translation or speech recognition it's really quite useful and helpful for smoothing things out um the big change that happens though is neural nets come into this space and so instead of just saying what's the probability of you know this the next word given the previous one through four um it starts instead being this task of predicting words using a much more elaborate predictive structure and i've say this is outside my expertise like i know the sort of from the outside what you can put into a neural net when you can get out of it but to describe what it's doing on the inside that's not mine maybe you want to say a few words there um i just i was i was just blown away by how efficiently and elegantly you explain the noisy channel model i was i was just still appreciating over here how well you did that um but yes i you know i don't know that i actually have much more to add than that i think that you captured a lot of the important basics and i don't know that we need to turn this into a tutorial on transformers because that'll take more time than we have all right so transformers exist and the result of that is that you can talk about words in terms of their embedding right so the transformer allows you to say okay for each given word i'm going to talk about it not as a string but as a thing in vector space that represents what other words it co-occurs with and then the next generation transformer is not only that but i'm going to represent it slightly differently depending on the context i'm looking at it in so we get more and more elaborate representations of words that represent more and more information about what other words they've co-occurred with and then these language models go from being something that's useful for basically re-ranking outputs to a useful representation of words in place of any other spot where you would use a bag of words model and so they start being just ubiquitous starting around 2016 or so in the nlp literature so i was program committee co-chair for culling which is one of the international conferences in the space in 2018 and by then just the vast majority of the submissions were using word embeddings in some way in the in the text um starting with like word devec yeah exactly where divec and glove and there's one other big one in that space um and pretty quickly like as that's coming in you've also got the start of people looking at okay but what kind of biases are these picking up like there was there was not a lot of lag between everybody starting to use them and a few people going hold on what are we learning um uh quote-unquote learning about the quote-unquote world by representing words in this way um and so the bullet glasses at all papers in 2016 and the calisthenic all papers in 2017. so that that happens pretty quickly and then there's one more step with the large language models which is the um the title the gpd it's something like um language models are few shot learners so we go from the old style engram language models being just a component that re-ranks things in the noisy channel model setup to language models providing us word embeddings that give us representations of words across many many tasks to the language model does the task and for the language model to do the task the task has to be recast into a language generation problem given some substring which is the prompt what comes next is the answer um and that is sort of a neat parlor trick like people were really impressed with gpt3 but it is uh parlor i say pilot trick and we say stochastic parrots because you can throw these large language models at tasks that are meant to test for understanding and they can be write an interesting amount of the time without actually having understood anything and they get there because of scale and the scale is both the amount of training data that they've ingested and also the sheer size of the number of parameters that allows them to model that training data very very closely so that for you know a surprisingly um high amount of the time given all those inputs and the and the parameters to model them they can spit things out so that's sort of the trajectory about size but concurrent with that the domain of application of the language models has expanded and we've got these you know very large language models now that take you know many weeks to train on you know gigantic uh data sets and one of the critiques that you mentioned in the paper is one that uh we've covered here on the show as well and the interview with ms jubel is the kind of energy cost of creating these things what did you learn about the the energy elements of this um so i what we've what we learned was i mean so we looked at at um the so strabala 2019 and a couple more recent papers also making similar points so echoing um what trebel had all said which is this is big enough now that we need to think about it that it's not just let me pull out my calculator and you know use a little bit of the energy that's in that battery there to ask a question but rather it's it's at a scale where if you're going to start one of these experiments it's worth asking okay is this worth it right what it's this is this is this has a cost associated with it and the trouble is that the cost is in the form of an externality so the company running the experiment yes they have to pay for the energy but they are not um required by law in the u.s to pay for the carbon costs i know that many of the companies do strive for to be carbon neutral and so they are probably doing offsets somewhere but the engineer making that decision doesn't necessarily know that right and i think the main thing that we contribute in this space is to bring in the the discourse about environmental racism so this is a cost-benefit analysis where the people paying the costs and people getting the benefits are not the same people and in particular in the case of language models there's this added angle to it which is that we aren't building these ginormous language models for each of the seven thousand languages on earth right they're getting built for english chinese maybe a couple other languages um and so even at that scale you can see okay who's going to benefit from this who uses this technology and we you know point to some of the places in the global south that are getting hit the hardest with the you know current effects of climate change the people there are not anywhere close to getting any benefits from this technology but you don't even have to go look so far afield right i mean environmental racism exists within the us as well and if you look at language models and what language varieties they target they target the privileged standardized language variety and not the language that is you know belongs to the communities that are getting the hardest hit with the environmental impacts so that's i think the new thing that we added was bringing those discourses together and saying this matters here too there's a comparison in the paper i don't remember if it came from emma's paper uh about um training uh training a burp model and comparing it to like a cross-country flight in terms of its uh carbon output um did you come across any sense or do you think we have any way to know you know so that's an individual training you know how much of this is happening you know at scale across companies part of what these models are offering is um kind of the transfer learning idea that you can train them once and then lots of people can have the benefit from them um do we know the you know as opposed to kind of a relative sense of the energy cost or environmental cost like what the absolute cost looks like i'm i'm i'm afraid i don't know i mean we have we have some numbers that we've quoted in the paper and those come from strabel at all it's not our own work and we also you know didn't this paper was you know co-authored by people then at google and people at the university of washington but we did not try to do any investigation internal to google about you know energy use or things like that we were just making the broader point that um it's time to think about this and not just assume that this is something that happens only in some abstract space it actually happens you know in the physical world as well and that has impacts yeah yeah i'll just maybe add that um part of why this is important it fits into this um growing trend in talking about ethical ai or aligning ai to human values about understanding the context around usage um and so we're you know we're talking about who's using it and how they're using it and what's happening and that's where you start seeing things like environmental considerations as well as societal impact because now you're thinking outside of the model itself outside of the learning itself and so this is also reflecting just sort of this general trend that i think a lot of us are aligning on as we think about you know creating more ethical technology which is what is the context and what are the considerations of that context um and that's part of what we are trying to do you know by pointing to this dribble paper and the details there um the uh so you talk a little bit about this environmental piece the uh you you reference that they're you know part of the the what's been requested or asked of industry is to just make clear like what the cost of these various experiments uh is and it doesn't sound like there's been a ton of progress in that regard and uh companies kind of volunteer voluntarily publishing the or companies or or even in academia folks volunteer voluntarily publishing the energy or financial costs of these models they're running uh any updates uh in that regard i think this might help the case i mean i worked forever trying to get google to publish um evaluation numbers across different subgroups and after years finally got uh you know finally was able to make enough inroads where google was doing that but that was only helped by people externally and activists showing that this is what we need um so i think that like you know perhaps unintentionally google has now focused a lot of attention on the need for transparent reporting of this kind of stuff so i imagine we'll start seeing it soon yeah and i just want to say that there's there's you know some levers so for um the knackle 2021 i'm actually co-chair of the ethics committee and we put together this faq these are things you should be considering and one of them is environmental costs so if you're running something that involves a lot of compute you know describe that and say something about the you know what is the actual environmental cost of this approach that you've taken so that somebody else picking it up would know um so at the point of publishing a paper there's this notion of okay we need transparency here the problem is that not all experiments become papers and so getting to sort of an accounting of all the things we try that didn't work like where does that go and so i think they're um having uh both sort of the general you know reports from companies about their efforts are becoming carbon neutral um and not just becoming carbon neutral but the overall energy use because one of the points that comes up is you can do a whole bunch of computation based on solar electricity um and so yeah your computation has not put any carbon into the world but if you weren't doing that someone else could have used that solar power to you know do something that is actually life critical as opposed to another tweak on the experiment oh we forgot that parameter let's try it this way kind of a thing so yeah it makes me think of uh you know there are you know in pretty much any you know subcategory of industrial activity there's some analyst firm out there that's you know building up a model for you know what the total addressable market of this thing is often kind of bottoms up looking at individual companies and what their revenues are things like that um it seems like someone could and should be doing the same thing to try to give us a sense for you know what is the total economic impact or environmental impact rather of the ai industry i've not seen anything like that but it would be fascinating to you know even if like you know other models it was wrong it would be insightful to get a sense for what the the scope and scale of it might be yeah absolutely startup idea uh so one of the the biggest contributions of the paper is really talking about the the bias implications of these kinds of models um you know meg why don't you get us started off there what uh you know how do you think about the you know large language models and the the bias implications of them yeah i mean one of the one of the things to keep in mind is that you have errors in these models that then are deployed at scale all over the world um so you know uh i think one way to understand this is similar to um the guerrillas incident that happened at google where that might have just been one mistake but it's one mistake that's the same mistake all over the world uh reflecting a very problematic horrible historical prejudice and so you have the same sort of thing with language models where even if it's one mistake it can affect billions of people and in the same way so you're very clearly driving a line between um you know the the normative sort of world we want and then these specific uh biases and stereotypes and and aspects of racism and things like that um and so uh this is one of the reasons why um you know you have to have like a critical look at how well these things work in different situations and what kind of things they they produce in order to have some sort of sense of what's going to happen when you deploy it the problem is which is part of the paper is that these things are so big that it's really hard to know so you kind of don't find an issue until suddenly people are being harmed by it um and so this is why we need to think very critically about how we build up these language models so that we have some sense of how it might perform and particularly with with the respect to uh things like racism and sexism and what it might you know further refi uh unintentionally one of the kind of recurring arguments in ai twitter and we've talked about it on the the podcast as well as kind of this idea um around you know bias in in these bottles coming only from the data versus the you know the other aspects of the the system the architecture of the models themselves emily you and i may have talked about this last time um in in the case of language models we talk a lot about the the data being the source of of bias are there you know how is that kind of conversation of the you know the broader um the broader sources of bias does how does that play out in terms of language models so i think that first of all i just want to go on a record of saying that i think that this know it's only in the data move that sometimes people make is really just a deflection and a way to say um to carve out to try to carve out a space where if all you're doing is working on architecture then none of this is your responsibility and it's just not true right so um the if you want to think about you know how does um learned bias in some sort of um machine learning model cause harm in the world then all the pieces matter so what's in the data how the data was collected how the data was documented can you actually tell what's in the data the model itself so what's learning from the data to what extent is it sensitive to you know outliers versus high frequency things you know there's model parameters there what were you optimizing for as you were training the model and designing the model and then how is the model sitting in its deployment context so at every step of the way there is part of the story of how bias in the data can end up as harm in the world and so yeah to a certain extent if if we could come up with a completely clean and this does not exist completely clean unbiased data set then what the rest of the steps did wouldn't matter as much but also if you could come up with a model that was somehow impervious to learning bias which is equally impossible to coming up with an unbiased data set then it wouldn't matter what bias is in the data set so you know it's it's everyone's got a responsibility and trying to say that's not my problem is being part of the problem i just realized that for people working on uh architectures and machine learning and optimization who say it's the data it's the same argument as oh it's the pipeline problem for diversity in time firing and diversity that's what happened before i have no part this is from before but also speaking speaking to that um to what emily's saying and the interplay between data and model you know your choice of loss function minimally is packing in normative decisions about how the model should behave so you use something like l1 loss l2 loss which is super common um sort of across statistics and especially in machine learning you're making uh you're making the value judgment that uh different kinds of errors are basically the same because they use absolute measurements um so even these very very basic fundamentals of machine learning pack in value judgments about how to treat different kinds of errors and that really affects what the model will learn and how it will propagate different kinds of issues yeah i'd love to maybe dig in a little bit deeper and make some of the harms of or potential harms of language models more explicit i think before we got rolling we talked a little bit about you know bias being a bit of this ginger word that we use that isn't particularly pointed about uh the outcomes of the bias um one of you want to elaborate on that who wants to jump in first i'll have a go with this one because i'm i'm i feel strong about i know meg does too and so she'll probably jump into you'll always see things better than me so um thank you i don't think it's true but thank you um so so what do we mean by bias in the language data first of all right and that is um basically a wide range of things starting with you know slurs and overt hate speech to um sort of the the subtle um framings of things um that can include you know the discourses that we have in this country where you can see that different um uh so people who are experiencing oppression will come up with new words to describe their experience that push back against what the oppressors are saying and so you see this in pairs for example between undocumented immigrant or undocumented person versus illegal immigrant or divorce illegal aliens so those those pairings um you also see things like you know how how often outside of the twitter account man who has it all do you see the phrase male doctor right compared to female doctor so there's these phrasings that basically say this is what's normal or normative and this is what's marked um so these are like this full range of things that shows up in the language data put that into a language model and think about how you're using the language model are you using it to um randomly generate text in a fun chat bot well is that fun chatbot going to be either spewing overt hate speech or just sort of going along with the various systems of oppression and saying things that are sort of vaguely racist or vaguely misogynist but a little bit dog whistle under the radar and then the people who are reading that are either experiencing the direct effects of that denigrating language because it's their own identity that's under attack or if it's not their own identity and they're not very alert to what they're seeing and sort of ready to look at it with a critical eye might be just sort of getting things subtly reinforced and then be prepared to go out in the world and say similar things themselves or act on these ideas that are harmful so that's sort of one version of it but also language models get embedded into other technology right um and they can become part of classification systems that basically then say well you know we associate um all of these high paying high prestige professions with being male and so therefore we are going to find this resume a better match for this high paying profession because it's got these other markers that are associated with being male it's one example um but also i mean this is what what um safiya noble was documenting in her algorithms repression book and i'm just looking down here because i have it put in a plug for her book there it is um so you know on the cover of this book she illustrates what happens with search engines by putting in the search prefix why are black women so and then um the search engine that she's using which based on the colors of the textbook is probably are the the colors of the book cover it's probably google um gets complicit with angry loud mean attractive lazy annoying confident sassy insecure and so you so sort of see all of these societal stereotypes that have been pulled into the search engine and exposed through this you know let's what might you be searching for here thing um and i want to say just sort of as a footnote that i don't know for sure how this product is created and so the search autocomplete might be using language models differently or independently of how the underlying search engine is but there's word list now so angry not gonna have angry all right um so anyway these these are ways that um the these biases that could either be overt or subtle can happen and there's one more that i want to flag that william agnew pointed out on twitter and i thought this was was really smart and fortunately he pointed it out while we were still working on the paper so we were able to incorporate the idea and credit him which is there's this list may says stop wordless this is a different one i think the list of what is it obscene offensive and otherwise very bad words or something and it was initially created by an engineer somewhere who wanted to avoid this problem wanted to avoid um search strings coming up and shocking people in the context of i think music recommendation i'm not entirely sure the back story there and so it is it's got a few racial slurs and then a whole bunch of words that basically whoever created the list thought would be associated with sites to do with pornography and if this is now actually used in filtering the underlying data for large language models in english basically use this word list to drop any website that has these words on it um which okay good first pass at getting rid of you know the overtly white supremacist websites okay that that helps like that that was not a negative thing in itself but in addition there's words like twink on this list and so you end up ruling out all of these spaces where people are speaking from their own you know lgbtq identity and speaking positively about their lived experience or positively about their identities and negatively about things that they've experienced and that stuff gets removed from the language models and so you've got this now uh sort of amplification of the rest of the language that is either indifferent to or negative towards lgbtq people getting sort of boosted in the training data and then you know downtrend effects you know what does it mean for someone to do this what kind of search results are we going to see what kind of um sorry search results are my only example top of mind right now but everywhere you get a language model and better what kind of mistakes are we going to see in speech recognition right where certain words are super low probability or certain sequences are certain low very low probability because of what was taken out of the training data and now we can't recognize it and portray it accurately and so instead we're going to say something ridiculous that in context could end up being quite offensive i only have maybe a slight add-on um you know emily talked about um ways of of talking about people that can be sort of othering uh like like saying illegal immigrant um and i think one of the um things that really sticks out for me in that is how when you add different kind of modifiers to words uh especially words about identity how that can be very othering or you might call it sort of marked so saying that this is an asian woman or uh you know a black person instead of a person or a woman uh gives it this like not quite a person but a certain type of person vibe that isn't there uh with a white default um and so you see this coming out in things like captioning models um you know any sort of generating from images where these sort of trends uh if you're trained on uh data that is predominantly from from white websites and people you know um with white experiences writing about their experiences then you're going to call white people people and you are going to call black people black people and so these even just very subtle things can really create an uttering uh in society that you don't even really know if you're and you're not paying attention and i think that othering ties right into the work that people from marginalized identities um have to do to just prove their validity as humans all the time this is something i hear over and over and over again is i am spending all my energy proving to people that i am a person and therefore have a right to have an opinion and exist and be respected and that is directly connected to this other thing that meg is talking about and we don't need computers helping us do more of that all right all right uh you also referenced in the paper the um kind of even you know from the the title of the papers to catholic parents this idea that um you know we tend to think that these models are we give them qualities that they don't have like understanding and reasoning and the like um and uh what what do you think are the implications of of that how does that play out in the use of these models so meg i want you to take this one sure yeah so so the human brain is amazing um and we are very good at anthropomorphizing there's a there's a famous illusion called um i think it's called the hyder symbol illusion where if you have this little video of a triangle and a circle and the triangle is moving around all rough and the circle is moving around all slowly and and lightly you create this whole narrative about the triangle as a bad guy and trying to attack the circle and circle's a good guy but these are just basic lines right and so if we can do that with basic lines imagine what we're doing when we see something that looks like actual language we really believe there's a mind there even when we tell ourselves rationally there's not we still are imbuing it we're still seeing it and this isn't um just because we're good at anthropomorphizing per se but because language in particular is something we've evolved with as a way of communicating and understanding one another's intent uh so we're in this position where we've actually evolved to uh to see this illusion and to experience it in a very human way that this is fundamental to our understanding um so it can be a pretty tricky issue there especially when it comes to things that could be like bullying harassment other kinds of really problematic language that could come out of language models and you start to really think it's real you start to feel that way improper medical advice you know there's so many things that you can now misinterpret misunderstand have wrong information about feel terrible about because you have a sense that there's some mind here that's telling you something but it's not are most of the uses of language models that you've seen um kind of in the wild noted that it's a language model i guess i you know i'm thinking a little bit to um conversations around like uh you know we in part um we impart a level of authority to computers and robots and automation bias yeah yeah um and uh but i think on the other hand that most uses of language models like you don't really know like even a chat bot like you know we name our chat bots we give them personalities and we try to hide the fact that they're computers um you know does that mitigate the you know this effect that that we're talking about here or is it accentuated it exacerbates it totally i mean the um so as meg was saying we are prepared as soon as we see language to imagine a mind on the other side of that and what's happening is the that mind doesn't exist except that we've created it right the the listener is the one who creates the the phantom mind there and um it's yeah it it is really there's great potential for harm there's a startup in france that ran the study asking whether gpt3 could be used for medical applications and sort of ran a whole bunch of different tests of ways they might imagine using it and one of them was a mental health chatbot right why you would want like you don't want even an untrained person handling sensitive you know discussions with somebody who's in distress right let alone a machine who's just going to randomly say things based on previous patterns and of course it comes out with suggesting self-harm right and my reaction to reading this blog post was what gave anybody the idea that this might be a reasonable application of this and there i think it really comes back to ai hype that people build these things and um it's a cute parlor trick and we are way too fast to say look what this thing can do and way oversell what's actually going on and this is what i was talking with you about last summer also but i think the i think the article that i gt3 must have been after that i don't think that this thing from france had been out at that point but it's really um really really important to keep both the specific instances labeled as this is a computer that that either knows nothing or only has access to these possible actions that can handle for you um and then in general having a very clear idea to the public about um this is not in fact intelligence of any sort it is just pattern recognition at scale and we need to think about what are the decisions that we want made based on pattern recognition and what are the consequences of reproducing the patterns that it will recognize even if they're not there because it's reproducing the patterns from this training data i am i want to also like piggyback on something you said emily which is talking about how using something like a language model for health information is a function of ai hype um i it's i've worked a little bit and somewhat familiar um with crisis lines and uh it's not just hype there's a lack of people you need if you know if you want to have sensitive conversations with someone um and just be available on the spot you need a lot of very very highly trained people and maybe that's possible in denver um you know but it but it might not be possible uh outside of you know more outside of like very well-funded cities right so um the idea with automation in at least like crisis helplines is that you can start standardizing bringing in people who are less trained but they can have access to potential responses and sort of standardize across them and so now you can have more of a sort of help chat where you press different buttons on the when you're the um sorry the helper you press different buttons and then different things will be generated um so there is an actual need for helping professionals in these contexts and then the problem is what is the foundation from which you're helping them and that's where the language model insidiousness then comes in right because you're going to be drawing from some language model in constructing these larger systems used for very sensitive purposes and there might be a person involved if you're lucky and even then they're not necessarily going to be making the right decisions if they're already given really problematic um input to choose from right and they might therefore themselves be subject to this machine bias well the machine suggested this this might be a good response right um so yeah i think it all comes back around to there's definite uses for pattern recognition and places where we can use automation to fill in the gaps or we don't have enough resources to hire enough humans to to do the work or the humans i mean it's also really difficult work right you could have all the resources in the world and you might not find enough people who are willing to do it so the question is how do we create something that is well fit to its context and you know really built with the population that we'll come in contact with in mind both the you know in this case the humans that are choosing among the responses and the people that are calling in and talking to them or chatting in i suppose and talking to them um you know is this is this system you know based on a bunch of actual therapy chats that involved um you know only white people or only wealthy people and then you deploy it as the you know municipal helpline and you get a much broader range of people who've got very different lived experiences and are going to talk about it in different ways and you know has it been tested actually in that context so that it can be as helpful as possible so the the paper is primarily asking questions and prompting people to think about these models but are have you come across examples of you know practices that you think will help or you know examples that give you hope or things that we should take inspiration from and what do those things look like is it curation is it uh you know carbon offsets what are those things yeah so one of one of the sort of common points that we started from in doing this work so as meg mentioned at the beginning um she had worked on um model cards and tim neat was on that paper too right and team ninten are you on the data sheets paper as well meg are you both okay so timmy had also worked on data sheets and at the same time i was working with baja freedom on friedman on data statements so all of this is about um documenting um underlying data sets and transparency exactly documentation and transparency and one of the practices that enables documentation and transparency is keeping things scaled to the size where you can actually do it and so we talked in the paper about a concept of documentation debt and i was feeling really special for having come up with that term that i thought i bet you're not the first emily so i went and did a search and sure enough people talk about documentation debt it's a thing um but the idea there is if you get to a place where your underlying data set or your model trained on the data set are so large that it can't be documented post-hoc then you are putting yourself at all kinds of risk for doing harm that you can't predict that you can't trace back that you can't fix um and so you know one of the practices that we suggest is budgeting for documentation at the start of a project and only collecting as much data as you can document and that's that's the appropriate size and that's our question when people say well okay how big is too big you're telling us that they can be too big where's the where's the bar and it's not one single number it's too big is too big to document um because once it's documented then you have this hope of saying okay how's this going to fit in my use case is this appropriate for the decisions it's going to be making the context is going to be in the people it's going to be coming in contact with so that's documentation is one set of practices um but there's others as well so this idea of a pre-mortem um and meg can you speak to that one a little bit oh so so a pre-mortem is when you think of what could go horribly wrong and then you walk through what you know chain of events had led could have led uh to to that horrible ending and try and find solutions to that before they happen it's also there's also a similar idea called the dirt test um in programming um so yeah sorry is that yeah that makes me think of uh you know amazon famously writes these press releases you know before they write it before they start developing a product um that kind of guide the development of the product uh the pre mortem sounds like maybe the ethics version of that where you write that's right yeah what could have gone wrong and you know use that to guide how you avoid that yeah exactly so you write the you write the scandal that would hit the papers and then figure out how to avoid that don't do that scandal yeah yeah and this is um i think connected to this idea from value sensitive design called um design noir and value scenarios so this is let's write stories let's think about you know what could happen and the and the practice there is to say all right what if you hit the jackpot and your product becomes completely persuasive and everybody's using it right how does that change because once it becomes part of the infrastructure of society what changes follow um but also what happens to you know think about specific vulnerable populations um if everybody's using this product then um you know what happens to somebody who you know doesn't have a smartphone or what happens to somebody um who's is all voice activated but somebody's got dysarthria and they can't speak clearly enough for the system can they still get on the bus or like you know sort of thinking through these things so that's and then there's there's one thing that i keep seeing on twitter and i don't know who to cite for this but this is like for every piece of technology one use case that you should keep in mind and try to thwart is what would happen if this was how would somebody who is an abusive ex partner use this to stalk like what's the potential there and just like that that use case is frequent enough it's a it's a frequent enough sort of downfall of various kinds of technology um that it's it's worth just sort of thinking that through early on now i don't know how to connect that to language models but going back to your question of practices a lot of this has to do with um sort of situating everything in context what are we trying to build who are we building it for who else might use it how do we plan to deploy it and sort of taking that mindset rather than the hey fun we're playing let's build something and figure out how to make money off of it later kind of approach there's like uh this point you made about being too big to document and that being part of the problem um i think there's also an important point to make here about uh normative versus uh descriptive uh approaches in language modeling where you could think of a way to create a data set being you define the requirements of that data set what are the characteristics that that data set should have let's be very clear about these and now let's get the data that matches the requirements that's sort of a normative approach a descriptive approach is like let's just take whatever and just call it the data we need which is actually the state of state of the art approach right now let's just take whatever the idea of having data requirements is new yeah and so i think it's really important to sort of keep in mind with this idea of being too big to document that part of that is that uh we're just collecting things from wherever we don't really know what it's conveying we don't really know what's going on there's no sense of getting things that specifically you know um are useful for some normative considerations it's rather let's have the internet define what's important what they go and there's this idea that the internet is everything and everybody yeah right absolutely the perfect representation of all of the world yeah and and it's absolutely not like if you dig into the studies and this is part of what we do in the paper i think it's section four and basically saying okay who's actually participating in the internet yeah that is already people who have more privilege and more resources and then who is participating in the sites that are chosen for the starting point so a lot of this actually starts from reddit not reddit texts itself but websites that are linked from reddit as well these must be high quality they have been linked by predators um and there's a study that we cite in the u.s that looks at um sort of a survey of redditors in the u.s and finds that it's like mostly people in the like 18 to 34 age group and also mostly men so you've already sort of narrowed your view of whose views are we taking on um but then on top of that who gets to participate comfortably in online forum right who's subject to harassment who's going to get harassed off of sites versus the people who um tend to like get away with harassment and stick around right so there's further narrowing and then that list of obscene et cetera and otherwise very bad words i was talking about is yet one more narrowing and so this notion that we are just grabbing what's out there that's just how the world is we're only reflecting that back to you just isn't true all right and on top of that so linguists very much like to be descriptive and not prescriptive but we're very much about you know language is a natural phenomenon people talk the way they talk all language varieties have inherent value and you can quit it with your stop splitting negatives type prescriptive rules um but that's because linguistics is engaged in describing language as it is and machine learning models that are deployed in context that are either making decisions or helping people make decisions are not just describing things they are affecting things and so a even if the we've just grabbed a perfectly representative sample thing worked we still wouldn't want to deploy that because that's not a job for description it's a job for careful engineering and some you know normative thought wow there's so much i got so many more directions i want to take this in but i'm also looking at the time here and want to be respectful of your time how about a round of kind of parting thoughts or words for the audience things that you'd like them to consider so one thing that came up in the the fact q a so the fact conference q a about this paper um that i that i've been sort of sitting with and i enjoy is this idea that uh specificity has real value there's a lot of value placed both in academia and an industry on scale and generalization and um trying to make things bigger and bigger and bigger and applicable applicable across more and more contexts and i think that it would be worth reallocating some value to building solutions that work really well in their specific context where the context includes the task and the users and the other people who are affected by the use of it because when you're working at that scale you can bring in the sort of care and considerations of impacts that really just get overwhelming if you're trying to do a thorough analysis of what are all the possible specific harms we need to mitigate in this one enormous general purpose thing so i want to put in a plug for context specificity and working at smaller scales yeah i guess i guess maybe maybe one of my imparting thoughts here can be about um considering the trade-offs when you try and mitigate different types of biases so you know a naive uh takeaway from this discussion might be oh well we should figure out uh how racism is expressed how sexism how ableism is expressed and then tell language models not to do that the problem you have to immediately start thinking about then is dual use once you have explicit details about racism and sexism particularly as it relates to language models then that in itself can be weaponized so there's always a very difficult balance between being clear on the sort of racist and sexist things you don't want uh your technology to do on the one hand and b making explicit the exact things that can be harmful to people uh in a way that can also be deployed at scale or even applying stereotypes in order to categorize them as such so there really is a tension here about um doing the right thing by mitigation versus coming up with other metrics or specific uh requirements and data collection so um you know naively just saying don't say racist things has with it a lot of drawbacks as well awesome awesome well emily meg thanks so much for taking the time to chat with us about the paper lots of interesting conversation and definitely one that folks should should read and think about thanks for having us on the paper's open access so um it should be available to everybody and of course we'll link it in the show notes uh and so go take a look remember the parrots thank you awesome thank you all right thanks thanks bye\n"