**Aerial Images Reveal the Evolution of Spatial Apartheid in South Africa**

A recent study by researchers at the Dare Institute has shed light on the persistence of spatial apartheid in South Africa, a legacy of the country's tumultuous past. Using aerial images and computer vision techniques, the researchers have been able to analyze the evolution of neighborhoods in townships, revealing the clear delineation between formal residential areas and informal settlements.

The study highlights the need for more inclusive and equitable urban planning practices, particularly in the face of rapidly changing demographics and economic realities. "It's not just a matter of looking at these aerial images," said one researcher, "but also considering the social and historical context in which they were created." The researchers are using their findings to advocate for policy changes that will allow for more accurate data collection and analysis, ultimately informing more effective solutions to address spatial apartheid.

**The Importance of Data Availability**

One of the significant challenges facing researchers studying spatial apartheid is the lack of available data on townships. In order to conduct meaningful research, it is essential to have access to reliable and comprehensive datasets that accurately reflect the demographics and socioeconomic characteristics of these communities. Unfortunately, many countries, including South Africa, have historically limited access to such data, which can lead to a lack of visibility for marginalized communities.

For example, in Brazil, Google Maps did not include Favela neighborhoods until recent years, highlighting the significant gaps in data availability that exist. Similarly, in South Africa, the government's own census data does not accurately reflect the demographics and socioeconomic characteristics of townships, which are often lumped together with formal residential areas.

**The Role of Independent Research Institutes**

In an effort to address these gaps, researchers at the Dare Institute are working on establishing independent research institutes that can provide critical data and analysis on issues such as spatial apartheid. By leveraging funding from public sources and partnering with local communities, these initiatives aim to prioritize transparency and accountability in their research methods.

"We want to ensure that our research is not only rigorous but also accessible and actionable," said another researcher at the institute. "That's why we're committed to building a team of researchers who are not only experts in their fields but also passionate about social justice and community engagement."

**Fellowships and Internship Opportunities**

In order to achieve this goal, the Dare Institute is currently hiring for various positions, including full-time researcher, intern, and fellow. The institute recognizes that these opportunities will be highly competitive, given the limited number of spots available.

"We want to create a pipeline of talented researchers who are committed to addressing spatial apartheid," said one researcher. "That's why we're prioritizing outreach and recruitment efforts to ensure that our program is inclusive and representative of diverse backgrounds."

**The Future of Research**

In the coming years, researchers at the Dare Institute plan to continue their work on analyzing the evolution of neighborhoods in townships. They also hope to expand their research to other countries, exploring how these issues affect communities globally.

"We're excited about the potential for our research to inform policy changes and drive positive social change," said a researcher. "That's why we're committed to building strong partnerships with local communities and working collaboratively to develop evidence-based solutions."

**Conclusion**

As researchers continue to explore the complex issue of spatial apartheid, it is clear that there are significant challenges ahead. However, by leveraging cutting-edge technology, independent research institutes, and community-led initiatives, it is possible to create meaningful change and drive positive social outcomes.

At the Dare Institute, researchers are committed to addressing these issues through rigorous research, inclusive partnerships, and a commitment to social justice. By working together with local communities and prioritizing transparency and accountability, we can build a more equitable and just society for all.

"WEBVTTKind: captionsLanguage: enlike i said your experience teaches you a lot more than what anybody else writes or says is that if you don't have the right institution and the right structure there's just no way that you can do things fairly alright everyone welcome to another episode of the twiml ai podcast i am of course your host sam cherrington and today i'm joined by a very special guest none other than timnit gebru founder and executive director of dare the distributed artificial intelligence research institute and of course a great friend of the show before we dive into today's conversation be sure to take a moment to head over to apple podcast or your listening platform of choice and if you enjoy the show please leave us a five star rating and review tim knit it is wonderful to have you back on the show it has been a bit i think uh this is actually your fourth time-ish because you did a meet-up that you probably don't remember oh i do remember back in january 17 about your uh google street view work uh and then your first time on the show is in january of 18 episode number 88 we're probably at 588 or something like that now and of course you helped us cover trends in fairness and ai ethics in uh january of 20 kind of looking back on 19. wow it's been a long two and a half years i can't even believe it why don't we get started by uh having you share a little bit about what's been going on for you welcome back yeah i can't even it's it's really interesting being back you know because i remember our first black night workshop you all had you were at like a a hotel room you had a whole setup it was just like it just feels like such a long time ago yeah that was long beach yeah yeah yeah um and it's very interesting it's kind of like chronicling a journey you know every time i come back here um well i have to say that you know right now i'm focused on dare as you mentioned um and i'm i'm trying to take the time to calm down a little bit and and also think about you know just take take a step back so one of the things i wanted to do was think about you know there are all of the issues that we're talking about ray fairness ethics labor issues etc and but what does the right model for doing things look like right what does the right institute look like what do the right incentive structures look like how should we approach the way we do research and and what we build what we don't build and i i'm just kind of trying to take the time to to figure those out at this uh right now with dare is it fair to ask you to give a 30 000 foot 30 second overview of your recent experiences folks get some at google to help folks get some context uh if they've not well heard any of the previous uh where do we start well so uh well i got fired from google or as uh some of my former teammates have called it actually sami banjo uh he coined the term being resignated he was like in french he said in french you know you have this word where like someone resigns you that's you know and so like they call it being resignated so i was resonated from google and it was a whole to be honest with you i still have not processed it because um i i don't you know it was in the middle of a pandemic in the middle of you know a war that just started um in ethiopia the most horrible war i have ever seen that is not really being talked about that also gets us has gotten me to see all of the issues on social media and in a way that i've never seen before you know people talk about these issues and it's it's like uh you never learn about it as much as when you experience it um and so in the middle of that whole thing um and um i wrote you know this paper on the dangers of large language models and the way this actually happened believe it or not was not because i wanted to write a paper but um i saw that people at google were basically saying you know why are we not the leaders in like large language models you know this is we should be the ones doing these giant models and you know you see this race just people are so fixated on having larger and larger models and i was i was very worried about that because it seemed to be this rush to a bigger thing without clarity on like why that bigger thing and also what are the issues um and so i asked um emily bender and i said hey you know do you have papers on this that you've written before that i can cite because right now i'm citing your tweets and if i could cite a paper that you've written that i can send to people because people are also internally at google asking me what are things we should worry about and so she said hey why don't we write something together and i'm like well i don't know what i'd contribute you know um and so then i and we each pulled in other people i pulled in meg and other people from that play team and we wrote this paper and honestly i never thought it would be contra it wasn't you know i i just thought it was just gonna be this paper and that's it right um i i didn't think they would love i didn't think the google people were gonna be like super happy about it but i didn't think they were gonna just you know do what they did obviously and so long story short um i found myself uh basically disconnected from my corporate account um in the middle of my supposed vacation and i found out from my direct report that i had apparently sent in my resignation and that sort of a whole you know very very stressful few months because then you know there was all this harassment online there was all of this you know you have to make sure you're safe um there are literally like people from the dark web who made it a point like a point to to to harass me come to all the talks i'm giving and you know just kind of harass anybody who was coming to my defense you know a lot of other people found themselves writing documents having to talk to lawyers and things like that people who don't even know me by the way just because see people who just you know were coming my defense on twitter or something like that just because of that i found myself being uh thirst thrust into the public space um and so then that also just that fact itself um brings in more attention from more people um and then i was like really worried about my team and what was gonna happen to them but then you know my my co-lead meg mitchell was also fired so it was a whole few months and it was a whole thing and you know that's what i mean by i didn't have a chance to really process what has happened in the midst of that of course i was thinking what is the next what is the thing i could do next because i really you know couldn't get myself to think about being at another large tech company and do that fight again i also know that i i would not there would be some companies that would be uh unwilling to hire somebody like me after all of that there's you know some members of my former team their office where we signed it from some places like after um this publicity and it's real you know it really is real that um people can you know by speaking up just destroy their entire careers and any options but um you know i had been thinking about creating a distributed independent research institute i've been even thinking about like creating a university why can't we have a distributed kind a different kind of universe you know i've been thinking about these things but if i um hadn't been fired probably what i would have done is slowly start something you know maybe start something from on the side and grow it very very slowly not not like the way you know we just started this so anyhow um and after that i decided to start there the distributed ai research institute that's awesome and so what's the how do you think about the charter for dare what's kind of in the in the zone in the scope versus out of scope yeah so you know there is a ai research institute like you know like any other research institute that you you can think of um the the thing that we are is we're an interdisciplinary research institute um so you know alex hannah recently joined as our director of research she's a sociologist um and um the distributed aspect was very important for me because i saw it even at google in the ethical ai team you know meg was very good at retaining and distributed a team and you know one of the last people we hired was mahdi who's a moroccan and he was raising the alarm on social media like no other person and he was doing all this research his friends were in jail they're journalists and i could see that nobody you know even the people in ethics or whatever could not really grasp this the gravity of the situation and if if you didn't have that person with that experience there's no way you would you would you know find out about that issue and look into it right and that showed me the importance of of having people you know like that and not forcing them to move to silicon valley or whatever i don't want to you know i i what i'm thinking about is how not to consolidate power right not how to further kind of contribute to the brain drain um of different other locations so um so that's why the first word that came to my mind was distributed and i called you know i told eric sears who's a program uh officer at a director at macarthur the macarthur foundation i was like hey look you know the first word that came to my mind is distributed i want to call it dare like does it sound weird you know um it's like no it's it's it's cool um and so so that's that's there and so when you say what's in scope versus out of scope um you know that's honestly something that we're still trying to figure out because it i'd like it to be kind of a combination of of course we have a few top down directions but i i really feel strongly that it's very important to have a bottoms up approach to research because you can't be the all-knowing person who knows like what the next important thing is right so it's important to let other people drive that too but um the thing we're focused on right now is you know what is our research philosophy and what what do we care about right and so first of all we care very much about not exploiting people in the research process one of the most um one of the things that is super clear in research in general and especially when you look at this field where you you know there's a lot of knowledge that's extracted from people a lot of data and different forms that's extracted from people without compensation without you know um acknowledgement etc right like you have that also in the fairness space for instance you have a group of researchers you know they get tenure and they're ascending based on work on fairness or something and who are the subjects that they talk about oh they'll talk about formerly incarcerated people or people in prison currently they'll talk about like different groups of people who are harmed by this technology who are not you know getting the money you know for the research or the fame or or you know many times their lives are not changing because of this work but they're subjects of it right right and so we're trying to figure out how how do we not do that you know how do we do the opposite of that uh what does it mean to to have research that that that incorporates um these people and actually is led by many times people like that and how do you funnel resources um and um and so one of our research fellows um who just joined mila is actually um one of the things she's doing is helping us figure that out right what is our research philosophy and how do we operationalize it so um in terms of you know what's in scope and out of scope so there's a self selection going on there where the people you know who do want to do research out there are in people who care about these kinds of things are somehow embedded in community building not just you know um like uh research that is that has nothing to do with that um and you know like for instance if you want to work on you know low um you know so i i'm cool i'm advising on a workshop which i coordinated before on practical machine learning um and you know for developing um countries or practical machine learning in low resource scenarios so if you want to you know kind of think about what about like small data and small compute right like that i think you might want to join you know we might want to think about working out there but if you're interested in like even larger models and even larger or something then i don't understand what we would you know provide in that sense so that's kind of how i'm thinking about it right now well what i'm hearing in part is that um the the areas that you've traditionally been working in a researcher ethics fairness and that you're probably best known for that is not necessarily a research focus for dare but more like a undercurrent or a foundation and dare is going to be broader uh and encompass uh uh you know like you said all the things that another research institute might like a mila might be interested in depending on you know who it is that comes and starts at research programs there exactly so like a lot some people describe dare as like an ai ethics research institute right and i'm like no um it's it's like yeah that's not what we're we're hoping to do and by by virtue of who we are we will so there's two ends of the spectrum that we we're looking at right and i think our advisory committee members when you look at safiyan noble and shira uh maina they they encompass those two ends of the spectrum so the first end is how do you develop how do you do this research in a way that we think is beneficial to to the groups of people that we care about and actually when you say what's in scope and out of scope our focus is you know we're starting with um thinking about you know people in in africa and the african diaspora right like so you know um you know there's no kind of question like i don't have i don't know if i have to explain why but like you know uh black people in general around the world who are very much harmed by this technology and not necessarily benefiting from it so um when you look at shira he's in the area he's in he's in kenya and a lot of his work is on how to you know work on um climate change um and data science right he analyzes bird migration patterns to to that tells you something about the the climate and how it's changing he he was at the first black neon workshop he probably covered his work um food security um and conservation he works on stuff like he co-founded data science africa right so it's kind of like you know how to work on the quote-unquote data science or related fields in a way that is beneficial to certain you know to the groups of people that he cares about on the other end you have sophia who's um um in you know in in the us and she is more on the other end of the spectrum how to you know raise the alarm um uh when we know there are issues um that with technology that's already been built right so we and you know she's uh more uh from the social sciences side right so like for me that encompasses sort of what i want to build with dare right um interdisciplinary have different groups of people um to to be able to work on research that you know we think is beneficial to our communities um and um in a way that's not exploiting the people who are actually you know who might not have phds or whatever but have a lot of knowledge about the these systems and how they're impacting them so i liked what you said yeah it is an undercurrent right of like how do we do this work is is that's how we're building this foundation i mean this institute one of the things that we chatted about before we started recording was that a lot of your focus right now is on institution building for obvious reasons you're building an institution like uh i'm curious what that means for you and also well afterwards i want to relate that back to your experience at google and and the idea around you know how to how to ethics organizations inside large companies like how do we build those so that they have teeth so to speak so that they can be effective yeah that's a very good question um and so i've been going on this fairness rabbit hole as you know and you know i've been like i've worked on things related to math and or documentation or auditing community building like black ai power building you know all the different um kind of ways in which i think you can attack the problem and i have kind of just kind of come to the conclusion like many and of course this is not something new that i'm saying it's just like i said your experience teaches you a lot more than what anybody else writes or says is that if you don't have the right institution and the right structure there's just no way that you can do things quote unquote fairly right so um so that's why i'm i'm kind of working on institution building right i've i've had experiences in academia i've had experiences in industry and when i after i got fired from google i was thinking you know a lot of people were saying well you obviously won't have academic freedom in industry if you want that you should go to academia and i was like that's not true right to me it's a pyramid scheme up here at the top of the you know somebody just tweeted the other day that graduate students make 36 000 a year perhaps right and you know it's like they're in this weird position are they students are they are they workers like do they get vacation or not but they're in this situation for years right very similar to college athletes oh abs 100 which also should get paid exactly so that's where we are right and so um yes and it makes absolutely no sense it's i think it's very very exploitative and so imagine you're doing that work as a graduate student and your advisor controls your life and then you're gonna tell them you know whatever research they're doing is not fair you should have a different sort of direction you you're you should stop how are you going to do that you you will lose your your money you will lose your um career like your future prospects because they won't write you a recommendation if people are on visas you will lose your visa so so ex so um how are we telling people to do the right thing when we know we're not setting them up right with the incentive structure to do the right thing and it's the same thing at work too right like um again what did i i spoke up i got fired so um then why why would anybody do something differently then right like and so so that's why i really believe we have to think about um the um incentive structures and it's not just about for instance labor practices that we're talking about right it's about what kind of work is valued and what kind of work is not valued um you know so i i think you have marielle gray well so her and sudarsa 3 have this book called ghost work how silicon valley is creating a global underclass and they're talking about data labor right so all of this automation that we talk about is sort of pseudo it's not you know real automation is that there's a lot of people behind it labeling data you know doing all sorts of things but they're being exploited they're not being paid right um and so in in in graduate school if you're telling your phd student that they should spend all of this time working on data related work data labor that's the very the most important thing you should think about how you're gathering and annotating data take the time to do this right but then they can't publish their work or they it's not valued or they can't get a job after they graduate again that's an incentive structure and institution building issue right so now there's some people working on journals for instance to be able to for people to be able to um publish on data and there was this new rips um this new europe's um data sets and benchmarks track where we actually um published a paper to for dare so that's what i mean like this is exactly why i'm thinking about the is the the incentive structures right because there's no way you could you know do quote-unquote the right thing if you're in the wrong incentive structure yeah yeah we i did an interview with safe savage uh who researches that area as well um that was a future of work for the invisible workers in ai exactly back in episode 447 you know if you kind of you know chart your path as experimenting with different institutional structures to try to see what works is your decision to start dare uh you know can we infer from that that support organizations aren't enough internal organizations aren't enough there needs to be just an independent uh alternative to kind of traditional research structures what i'm exactly what i'm thinking is so let's say if i went to academia and i just told you i'm trying to spend the time to think about the meta questions how do we build something etc how am i gonna survive like i have to publish tomorrow my students have to you know i'm no 10 year old sorry what are you working on like it's not even an option right so so mike so my hope is that yes um we start these smaller independent institutes where we can actually say stuff um alex was telling me about a talk that she gave the other day and that might not have made a number of people happy but she's like well that's fine because i'm not looking to get tenure and i'm like yeah that's the kind of stuff you can do when you're not looking to get tenure so i think you know it gives us the opportunity to actually advocate for things that we think are important and maybe um slowly those other larger institutions might change or you know have pressure to to do things differently if they know that there are different options like like our institute and if other people create other institutes and honestly um if you look at you know even how i started black nai before black nai i had been involved in a lot of other organizations like for diversity or for this or for that and i was like you know there's no way i can convince you all to do the things that i think we need to do for black people you know i just i fought i tried i this and that and i was like let's start something new and do it the way we think it should be done and this is kind of similar to that right i try this i tried that i tried inside the organizations i tried appealing to you know higher ups i do whatever but you know i found that that's not you know it's not working and so what i want is to have an alternative and even when you look at black and ai right what has you know there's now um latinx and ai queer and ai indigenous disabilities and ai you also have a lot of black and x um i'm lacking robotics black and nero black and physics but i don't even know like there's so many of them right and we can then you know build there is like a network now you have you build power and you can advocate for things collectively um and so hopefully that's what i'm hoping with dare right it's kind of an alternative to what we have right now um and hopefully you know other people can kind of replicate it in a way that not exactly replicated but you know in a way that works for their context um and you know so with dare i can do things like you know think about funding right where is our funding coming from it you know honestly sometimes it feels like pick your poison like there's no really clean money like you know i'm learning all these things but you know you i'm thinking about right like again like i said the meta questions right if we're thinking about ai and where the money comes from you think about technology in general when the government really invests in technology right um it's during warfare or when they're interested in something to do with warfare like so the transistors in silicon valley right um in world war ii you think about machine translation why people were interested in their research has to do with russia cold war you think about darpa and self-driving cars it wasn't because they were like oh we need you know to make cars more accessible you know we need to make sure that blind people can uh very freely move around so let's build that's not what they said they said we we care about you know autonomous warfare right and so how do we expect to come to a different conclusion when from the very beginning our funding our incentive structures every the paradigm that we're using has something to do with warfare and it's the same with industry too like if if all you're thinking about is how to make money for this large huge humongous company that that you know affects the entire world controls the entire world how do we have the space to think about a different paradigm right so like we're hoping to to think about a different paradigm i'm sure like you know not that these paradigms don't exist other people are doing it too but like you know kind of take the time for ourselves to think about what paradigm should we follow starting from the funding um to how we do research um to you know who we are hoping to serve you know we both have a lot of kind of colleagues in industry that are working within larger organizations trying to help them use ai responsibly you know what does it say about that work is it you know futile is it for not is it you know a pessimistic view or is it you know do you do you have examples of that process working correctly that you refer to uh and you know you're just offering an alternative or do you think that that is um you know yeah you know there's fundamentally there is this paper called um what is it the grey hoodie project from the university hoodie project yeah from people at the university of toronto and um and they talk about they say how big tech's tactics are close to big tobacco so they talk about how and they give examples of how like you know the tobacco industries would give lots of money to certain academics who talk who write about how well you know it's not it's unclear if smoking causes cancer or something like that or they would then internally retaliate against people who actually have those kinds of funding i mean of conclusions right or fossil fuel industry whose scientists knew about climate change way back but they were suppressing it and so why you know why wouldn't big tech be like that i mean what is their incentive not to be like that so i i have seen it myself how they capture how they you know people talk about industry capture how they use um research in order to like fight back against regulation so i do believe honestly that the number one um reason that these large tech companies want to have these clinical ethics teams is to um in order to like fight back against regulations so after i got fired you know uh members of congress and representatives sent a letter to google and there was a number of letters they sent first of all they sent letters about you know um you know the the number of black people they have in these ai divisions do they have special like uh training uh in ai except you know a racial equity or any training or impact or anything and they write back and they say oh we have you know these um ergs or whatever you know employee resource groups and we have lots of black people we have this event that event and similarly um they uh wrote a letter to them about lara's language models and their impacts et cetera and they're like we have had hundreds of papers in ethics and fairness you know what i mean but i know for a fact they are actually suppressing more papers about the dangers of large language models you would think that they've learned from their lessons so when they're doing this they they are freely allowed to suppress and persecute people with certain kinds of works but not others you have to ask why and that's that becomes more of a propaganda than research right so i do think that this is their goal but so the people inside could know that and try to fight that right and um i think the way they can fight that is through collective organizing like people before them have done right um polaroid or uh workers organized against polaroids on partnership with apartheid south africa right and that was that had a big impact so it's not that i don't think that people on the inside cannot um cannot change things but it's that they have to be vigilant to understand why they want them there and how their work is being used right if you're meg mitchell used to call it fig lift right she's like fig a fig leaf she's like i don't want to do a fig leaf work you know because like they already do everything else and you do you do the fig leaf work right like you're like just stamping what they say oh we're gonna write your ethical consideration section or we're get we're just not changing the course the direction that you're doing um but we'll we'll sort of do a fig leaf thing that can do much more harm than good do you are there examples in the industry that you look to as you're creating dare um well i mean there's examples of what i don't want it to be like and what that was a huge motivation for you know like the open ai type stuff is not what i want like when open ai was announced i and i have been so clear about this i've never hid kind of um i remember when opening it was announced i was at new york's and i think it was in 2015. and that was before the name change and i had just gotten harassed at some you know bro compa like um a google uh party i was harassed i was having a horrible time i just like i don't ever want to come back to this conference and you know and it was after the whole google gorillas incident and they announced this company that's a this that's supposed to save humanity from ai like the whole world it's all about elon musk and peter thiel and all these people they had 10 deep learning people 100 no interdisciplinary whatever eight of them white men uh one white woman one asian woman um and i know for i like i knew for a fact this was not gonna save humanity and fast forward what are we talking about we're talking about gpt3 we're talking about the dangers of large language models we're talking about how we're worried about things we're talking about how you know there's unsafe products out there etc right so of course like you start at the instep this is exactly what i'm thinking about institution building right so you start at the inception who was at the table where did the funding come from where you know and so um unless you think about that you can't not arrive at uh at the kind of the state that we're in today there are a lot of those kinds of models i am finding out about institutes left and right right now one working on agi one working on some other thing that has like 50 million dollar endowment or whatever you know i get irritated i'm like where is that 50 million dollars coming from i wanted to dominate so i don't have to think but you know um but then i would have to compromise on of course probably where we get the money or something like that so that is you know on the one hand i have models of what i don't want but i do have models of the kinds of grassroots organizing that i've seen that i'm i'm really excited about right so for instance i gave you an example of masakani uh which is a network and it was really beautiful to see right because it grew up grew out of the deep learning in daba uh which is a convene right and so then they create a people there who met their created masakani network and it's really cool it's a whole bunch of people focused on working on natural language processing tools for african languages and their values are very much kind of in line with the kinds of values i'm i'm thinking about for dare um and they grew it super slowly you know and i think now they have a foundation i don't think they have any full-time people um before the show i was i was talking to you about um the this article this wired article i had read about the maori uh who created speech related technology to benefit their community so they they had this um competition uh using um their local radio to for people to send in kind of annotated speech um for like speech to text and other kinds of speech related you know language related technology and they had like hundreds of hours of data right and um then all of a sudden this company i think was called lions land bridge or something this american company wanted to license their data and they you know said no and they they published their um their reasoning and they said you know we think this is the last frontier for colonization um they beat the language out of our grandparents literally were not allowed to speak this language and they were beat up for speaking it and why is this company interested in like you know buying stuff for now it's not it's obviously not because they want to benefit this community so they're like we want to make sure that whatever we do with this data and how you know is is something that benefits us so those are the kinds of models i'm looking at i'm like oh that's awesome like i like that you know how they're doing data stewardship you know um and you know masakani i like their approach for grassroots organizing migente right is another grassroots organizing they're doing such great work i just read their report on for instance border technology um and they're talking they're educating people about what are the different companies involved in this like digital border digital border walls how should we organize they drove palantir out of palo alto right and they had this no tech for ice campaign so so i'm looking at that um them too and and how they're they've been able to be so successful with their grassroots organizing so i'm looking at different kind of you know different kind of models to see what what it is that i like about each of these models and what makes sense for dare yeah yeah one of the things that you mentioned as we were chatting before getting started was this realization that you had that you can't reduce fairness to a mathematical problem like in i think you saw that experience that i see a ton of that elaborate on that a bit and kind of your journey to realizing that and where you see it um how it occurs for you out in the industry you know i mean like my sisters have been saying you know doctors of fianna has been saying this forever dr ruja benjamin has been saying this forever simone brown and you know like the people not trained as engineers and computer scientists have been saying this for a long long time and unfortunately i was um reading philip agrees um it was the most depressing thing in the 90s he wrote um so there was even actually a washington post article about him he was um an ai and then kind of uh became much more of a critic of it he was a became a professor and he was like talking about a number of issues that for instance like people are gonna share their data much more freely with you know for various applications right this is way before social media and stuff and he was like it's not gonna be so much of a big brother kind of thing but people are just going to like share it without knowing without thinking carefully you talked about face recognition and so i was reading this like lessons from trying to reform ai or something like that i'm like you got to be kidding me i don't remember when i mean but it was like lessons from china from ai talks about how the field is not reflexive how it's arrogant you can't get people to think about disciplinary norms and whatever and i'm like oh my god like it's true right and that's what it is it's that when the field feels like it's better than other fields has a lot of power and money thrown at it at this point you have money from the government money from industry money from everywhere you don't have to think about what anybody else says even though these people have been saying this stuff forever um and so uh what my own experience showed me of course you know we can talk all we want about we we we want to reduce fairness um to mathematical equations because first there's again it takes me back to that incentive structure when you think about how you ascend in the academic world in computer science or in you know in um you know engineering in general there is this hierarchy of knowledge i gave a whole talk about the hierarchy of knowledge right certain kinds of knowledge and contributions are valued so if you spend five years working on data related stuff first of all in these conferences if you have a data track it's already inferior oh it's just a data set paper or whatever you know where's the engineer you know that's how they talk um and actually kiri wagstaff gave uh um a keynote uh at html 2012 called machine learning that matters which was basically about this kind of stuff and how what conferences are valuing versus not so that to me makes it such that you want to reduce everything to the algorithm to the math to the whatever you want to not look at it as a socio-technical problem as you know many people in sds have said and when you do that um like the paper there's a paper called fairness and abstraction um they they do a really good job of giving examples of what kind of issues might arise when you do that right you're just like looking at the system in isolation um not thinking about it as as you know as part of a larger system which is like how is it being used what domain is it being used and who is be using it against whom etc then your analysis becomes very different and much more complex but you're not incentivized to do that where are you going to publish what you know you're not going to ascend the people who are going to want to give you 10 you're going to be like oh whatever that's just data or that's just that's just fluffy whatever you know that's what i'm telling is how they talk and so so because of that you're incentivized to be like oh you know what what does fairness mean i have no idea what fairness means right and um you know mimi i know actually was sita pena i think who said um she came to fact and and she um gave a talk about some of her observations there at some other conference workshop and she said what what why you know what does it mean to make systems fair that are punitive that their job is to just be punitive right so um for instance when people talk about risk assessment um they jump to you know using the compound compass data set and then you know christine lom and others like have written about the uh like all the issues with that data set and why people shouldn't just jump to use it um and then they they say okay like you know we we looked at that data set and we have this new algorithm and it makes you know this other metric higher by x percent right what is exactly what does that mean in reality right like what you're doing is you're still locking people up like you know and and you're trying to figure out how you know what does fear mean in this case like you're locking this other person at the same amount so a lot of people don't you know abolitionists don't even think that whole system should exist right so when you're not looking at the entire system and you're just focusing on this math um it's even it's unclear like what uh you know if that's even something that will help or not and many times it can be very harmful i've seen this in the face recognition discussion you know um after joy and i wrote the paper like gender shades a lot of people are like oh okay you know microsoft came out with an announcement saying now that like they've changed their training data you know data and now it's much better now it's all you know accurate it doesn't have um you know the uh for darker skinned women the error rates are not as high but then when you look at all of the scholarship from especially trans and non-binary scholars they talk about how automatic gender recognition should not even exist it shouldn't even be a thing so why are you jumping to making it quote unquote fair right that's because you're not incentivized to look at the whole system so you know and then even if you want to do the right thing even if you want to do the right thing like i i try to do at google if you if you're you're going to get fired then what what does that mean right like why are we writing papers about what to do if everybody's going to get fired if they try to do the right thing so that's what i mean you just cannot um reduce it to this mathematical thing but you could you keep on doing it and our field people keep on doing it because that's what they're incentivized to do and that's what they're rewarded for i think what i love so much about talking to you is that you have this very clear view of all of the challenges the systemic systematic challenges that are kind of inherent and baked into um you know all of these systems that uh you know we struggle against and you know that doesn't uh you don't get jaded you just oh let me try something else let me try something else i'll try something else honestly i think sometimes it's because i don't really think about you know the other option i think is way too depressing is is what i think right the other option of like you know being like i guess this is too big of a problem we can't do anything i just you know it's too depressing right um and then sometimes like when i started dare now i think about oh my god i was doing i was doing this um visualization of the plots of like how much money you have your burn rate when you're there's a red line when you run out of money i don't and i made a mistake and that red line was like literally like this october and i just my heart was just like oh and i and i knew i made a mistake but i'm like oh my god i'm doing something right now where this could be this could be the scenario and it's not just my job it's like all of these people's jobs that are on the line you know and um so when i think about those things i'm like oh my god what am i doing but you know if i don't really if i don't really then you we have to be and and i love this um quote by mariam cabba and i heard it um from ruja benjamin saying it that hope is a discipline it's it's you know we we have to yeah what's the alternative what do we you know and it's not like a lot of times i think we think that things are so far away they're not going to touch us but we're seeing that that is not true right with the pandemic with the wildfires in california with you know all of the ways in which we're all connected in the world like you know if you want a just even a better a better world for ourselves i'm not even thinking about the next generation who should sue the hell out of all of the prior generations for leaving them like a world with cata you know with the climate catastrophe even if we want a different alternative you know i think we should work for it and for me i it makes me feel better to do that right to feel like at least you know i'm trying this other thing um you know this other alternative you know um otherwise it's just too depressing i don't i don't know how to not do that you know yeah yeah yeah um so the initial funding for dare came through macarthur have you identified funding sources beyond that or how that all is going to work yeah that's the big question that i'm working on right now so the initial funding came from macarthur and ford um and also the kapoor center um gave us a gift and um rockefeller foundation and open society foundation so it's all these foundations right now and so now we're applying for you know project-based funding for grants like based on specific projects that we're working on and just like other people you know if there's an nsf grant we'll look into that and you know see if we can apply but i am extremely worried about having a whole institute that is only based on grants so one of the things i'm doing right now is trying to figure out how do we have our own revenue stream and what does that look like um i'm really hoping to have some things that we can experiment with in the next few months because you know we have a bunch of people with expertise um and i think we can provide that expertise in different ways that like are valuable to people um and that help us kind of generate revenue for our institute in a way that gives us a little bit more freedom and independence and flexibility right imagine right now i see something wrong that one of the f you know one of the funders doesn't like they're all know each other and then everybody can just be like sorry bye like you know that that that can happen um and um it's really interesting you know the nonprofit world you realize you know i mean it is because of wealth inequality that this world even exists it's actually really sad um and it's all the people i ran away from like you know eric schmidt you know chan zuckerberg besos and these are all the people who have these large foundations that want to fund tech related stuff so it's you know um so that's kind of what i'm thinking about right now like we're identifying different funding sources um thinking about how to diversify our fun our funding sources our revenues what what would our own revenue stream look like um and once i figure especially the revenue stream part and um we have a few things to experiment with i'll be much happier i'll be you know i'll feel much better about it nice nice now how far along are you are are there uh are there folks that are dare affiliated that have research projects that are spun up and uh things that you can talk to so we have uh we have alex hannah as a director of research dylan uh baker who used to be under me at google too um as a as a researcher slash engineer um we have i think uh two research fellows uh mila and russ sedra mila just joined like this week um and one person who's uh probably gonna join us full-time and in the next month or so and uh yeah so we have for instance one of the projects um that uh i have been working on what we've been working on with uh rasha is this project to analyze spatial apartheid the impacts of spatial apartheid using satellite images uh and computer vision techniques um and that's a project where again all the issues i talked about appear like it takes you a long long time to the innovation is on figuring out the data right like um how to get the data and how to process it how to annotate it but that's very hard to what does that mean spatial apartheid oh spatial apartheid is is basically like segregation but um but it was mandated in 1950 by the group areas act uh in south africa so it's it's a a big um like it's a feature of apartheid you know and um so uh people of european descent could live in certain areas and and everybody else had to live in in you know other areas like townships and the budget allocation was a lot lower for townships of course um and so the question is you know supposedly apartheid has ended legally right but when you look at these aerial images it's so clear like the the delineation is so clear and so the question is can we analyze the evolution of these neighborhoods and how things are changing because we know right it passes the smell test and that you can look at these things visually um and and do an analysis it's we're not just trying to do magic right so um so the question is you know how do how can we use computer vision techniques to do that um when speaking of you know exploitation versus not etc rasa just someone who grew up in a township i mean so this is a very personal project for her so it's like she's you know investigating her own you know like stuff that's related to her it's not like this when people say parachute science right um so that's one of the projects we're working on we just um had a paper on it's um we're working on releasing the data that's one of the things i like about being at dare because i you know we didn't just stop you know publish the paper and like really quickly release the data and we're done we're like okay how do we release the data how do we create visualizations how do we allow people to interact with the data what are what follow-up work are you know we're writing an article for africa as a country i don't know if you know that outlet it's uh it's one of my favorite outlets about about the work and one of the things we want to say is that actually the south african government has to um label townships if they want to analyze the impacts of spatial apartheid what they do is they in the in the census they lump it with just suburbs as formal residential areas but you know you that doesn't allow you to to because townships were created as because of apartheid that doesn't allow you to and this is interesting it's part of a larger kind of issue um was talking about how some of her work i think it's called data voids or something like that talks about how for instance google maps didn't have fabelas in brazil you know right that's a huge huge huge community of people so it's part of this larger thing about you know who whose data is is visible um anyhow but yeah like but that's an example of a project that um we're working on and there's a few others too well how uh how can the community support what you're doing well you know follow us a dare um we're gonna uh and you know on twitter um i think we're gonna also have more stuff on our website just about more stuff we're working on um and you can donate to dare if you're interested we're going to have you know fellowships for people that we're you know we have to think through how to do these fellowships too um and yeah i think that's it you know um and advocate for more funding for these kinds of independent research institutes i i don't want to have to cater to like a billionaire to to get you know uh funding for our institute i would rather apply for a grant that comes from you know public you know tax payers and you know be accountable to that so that's another way i think in which people can advocate for these things and are you still hiring are you bringing on additional researchers yeah i mean we have a lot of requests for hiring and so we have to figure out like i said we have to first build the initial foundational team um and so before i we open it up for like uh applications that will fly like but we've had like hundreds of um people asking about internships and volunteer and and you know full-time jobs so after we set up the initial team then we're gonna be think you know thinking very carefully about what kind of internship fellowship opportunities we'll have what kind of you know other full-time opportunities we'll have i mean that that's the thing about having a small research institute and having to think about funding sources is i can't grow it really fast right i can't like so that's that's the sad part but um and uh the thing about volunteer opportunities that i'm thinking about very carefully is who does that prioritize right a lot of people can't do volunteer stuff because they have to work so i think i feel strongly about people being compensated for for their work very cool very cool well tim knit it has been wonderful as always connecting reconnecting with you and learning a little bit about dare and what you're building there thank you for having me it's it's a lot of fun to come back periodically and kind of reminisce on like how much stuff has changed you know yeah we'll have to be sure to uh to schedule the next one not quite as far out yeah yeah yeah youlike i said your experience teaches you a lot more than what anybody else writes or says is that if you don't have the right institution and the right structure there's just no way that you can do things fairly alright everyone welcome to another episode of the twiml ai podcast i am of course your host sam cherrington and today i'm joined by a very special guest none other than timnit gebru founder and executive director of dare the distributed artificial intelligence research institute and of course a great friend of the show before we dive into today's conversation be sure to take a moment to head over to apple podcast or your listening platform of choice and if you enjoy the show please leave us a five star rating and review tim knit it is wonderful to have you back on the show it has been a bit i think uh this is actually your fourth time-ish because you did a meet-up that you probably don't remember oh i do remember back in january 17 about your uh google street view work uh and then your first time on the show is in january of 18 episode number 88 we're probably at 588 or something like that now and of course you helped us cover trends in fairness and ai ethics in uh january of 20 kind of looking back on 19. wow it's been a long two and a half years i can't even believe it why don't we get started by uh having you share a little bit about what's been going on for you welcome back yeah i can't even it's it's really interesting being back you know because i remember our first black night workshop you all had you were at like a a hotel room you had a whole setup it was just like it just feels like such a long time ago yeah that was long beach yeah yeah yeah um and it's very interesting it's kind of like chronicling a journey you know every time i come back here um well i have to say that you know right now i'm focused on dare as you mentioned um and i'm i'm trying to take the time to calm down a little bit and and also think about you know just take take a step back so one of the things i wanted to do was think about you know there are all of the issues that we're talking about ray fairness ethics labor issues etc and but what does the right model for doing things look like right what does the right institute look like what do the right incentive structures look like how should we approach the way we do research and and what we build what we don't build and i i'm just kind of trying to take the time to to figure those out at this uh right now with dare is it fair to ask you to give a 30 000 foot 30 second overview of your recent experiences folks get some at google to help folks get some context uh if they've not well heard any of the previous uh where do we start well so uh well i got fired from google or as uh some of my former teammates have called it actually sami banjo uh he coined the term being resignated he was like in french he said in french you know you have this word where like someone resigns you that's you know and so like they call it being resignated so i was resonated from google and it was a whole to be honest with you i still have not processed it because um i i don't you know it was in the middle of a pandemic in the middle of you know a war that just started um in ethiopia the most horrible war i have ever seen that is not really being talked about that also gets us has gotten me to see all of the issues on social media and in a way that i've never seen before you know people talk about these issues and it's it's like uh you never learn about it as much as when you experience it um and so in the middle of that whole thing um and um i wrote you know this paper on the dangers of large language models and the way this actually happened believe it or not was not because i wanted to write a paper but um i saw that people at google were basically saying you know why are we not the leaders in like large language models you know this is we should be the ones doing these giant models and you know you see this race just people are so fixated on having larger and larger models and i was i was very worried about that because it seemed to be this rush to a bigger thing without clarity on like why that bigger thing and also what are the issues um and so i asked um emily bender and i said hey you know do you have papers on this that you've written before that i can cite because right now i'm citing your tweets and if i could cite a paper that you've written that i can send to people because people are also internally at google asking me what are things we should worry about and so she said hey why don't we write something together and i'm like well i don't know what i'd contribute you know um and so then i and we each pulled in other people i pulled in meg and other people from that play team and we wrote this paper and honestly i never thought it would be contra it wasn't you know i i just thought it was just gonna be this paper and that's it right um i i didn't think they would love i didn't think the google people were gonna be like super happy about it but i didn't think they were gonna just you know do what they did obviously and so long story short um i found myself uh basically disconnected from my corporate account um in the middle of my supposed vacation and i found out from my direct report that i had apparently sent in my resignation and that sort of a whole you know very very stressful few months because then you know there was all this harassment online there was all of this you know you have to make sure you're safe um there are literally like people from the dark web who made it a point like a point to to to harass me come to all the talks i'm giving and you know just kind of harass anybody who was coming to my defense you know a lot of other people found themselves writing documents having to talk to lawyers and things like that people who don't even know me by the way just because see people who just you know were coming my defense on twitter or something like that just because of that i found myself being uh thirst thrust into the public space um and so then that also just that fact itself um brings in more attention from more people um and then i was like really worried about my team and what was gonna happen to them but then you know my my co-lead meg mitchell was also fired so it was a whole few months and it was a whole thing and you know that's what i mean by i didn't have a chance to really process what has happened in the midst of that of course i was thinking what is the next what is the thing i could do next because i really you know couldn't get myself to think about being at another large tech company and do that fight again i also know that i i would not there would be some companies that would be uh unwilling to hire somebody like me after all of that there's you know some members of my former team their office where we signed it from some places like after um this publicity and it's real you know it really is real that um people can you know by speaking up just destroy their entire careers and any options but um you know i had been thinking about creating a distributed independent research institute i've been even thinking about like creating a university why can't we have a distributed kind a different kind of universe you know i've been thinking about these things but if i um hadn't been fired probably what i would have done is slowly start something you know maybe start something from on the side and grow it very very slowly not not like the way you know we just started this so anyhow um and after that i decided to start there the distributed ai research institute that's awesome and so what's the how do you think about the charter for dare what's kind of in the in the zone in the scope versus out of scope yeah so you know there is a ai research institute like you know like any other research institute that you you can think of um the the thing that we are is we're an interdisciplinary research institute um so you know alex hannah recently joined as our director of research she's a sociologist um and um the distributed aspect was very important for me because i saw it even at google in the ethical ai team you know meg was very good at retaining and distributed a team and you know one of the last people we hired was mahdi who's a moroccan and he was raising the alarm on social media like no other person and he was doing all this research his friends were in jail they're journalists and i could see that nobody you know even the people in ethics or whatever could not really grasp this the gravity of the situation and if if you didn't have that person with that experience there's no way you would you would you know find out about that issue and look into it right and that showed me the importance of of having people you know like that and not forcing them to move to silicon valley or whatever i don't want to you know i i what i'm thinking about is how not to consolidate power right not how to further kind of contribute to the brain drain um of different other locations so um so that's why the first word that came to my mind was distributed and i called you know i told eric sears who's a program uh officer at a director at macarthur the macarthur foundation i was like hey look you know the first word that came to my mind is distributed i want to call it dare like does it sound weird you know um it's like no it's it's it's cool um and so so that's that's there and so when you say what's in scope versus out of scope um you know that's honestly something that we're still trying to figure out because it i'd like it to be kind of a combination of of course we have a few top down directions but i i really feel strongly that it's very important to have a bottoms up approach to research because you can't be the all-knowing person who knows like what the next important thing is right so it's important to let other people drive that too but um the thing we're focused on right now is you know what is our research philosophy and what what do we care about right and so first of all we care very much about not exploiting people in the research process one of the most um one of the things that is super clear in research in general and especially when you look at this field where you you know there's a lot of knowledge that's extracted from people a lot of data and different forms that's extracted from people without compensation without you know um acknowledgement etc right like you have that also in the fairness space for instance you have a group of researchers you know they get tenure and they're ascending based on work on fairness or something and who are the subjects that they talk about oh they'll talk about formerly incarcerated people or people in prison currently they'll talk about like different groups of people who are harmed by this technology who are not you know getting the money you know for the research or the fame or or you know many times their lives are not changing because of this work but they're subjects of it right right and so we're trying to figure out how how do we not do that you know how do we do the opposite of that uh what does it mean to to have research that that that incorporates um these people and actually is led by many times people like that and how do you funnel resources um and um and so one of our research fellows um who just joined mila is actually um one of the things she's doing is helping us figure that out right what is our research philosophy and how do we operationalize it so um in terms of you know what's in scope and out of scope so there's a self selection going on there where the people you know who do want to do research out there are in people who care about these kinds of things are somehow embedded in community building not just you know um like uh research that is that has nothing to do with that um and you know like for instance if you want to work on you know low um you know so i i'm cool i'm advising on a workshop which i coordinated before on practical machine learning um and you know for developing um countries or practical machine learning in low resource scenarios so if you want to you know kind of think about what about like small data and small compute right like that i think you might want to join you know we might want to think about working out there but if you're interested in like even larger models and even larger or something then i don't understand what we would you know provide in that sense so that's kind of how i'm thinking about it right now well what i'm hearing in part is that um the the areas that you've traditionally been working in a researcher ethics fairness and that you're probably best known for that is not necessarily a research focus for dare but more like a undercurrent or a foundation and dare is going to be broader uh and encompass uh uh you know like you said all the things that another research institute might like a mila might be interested in depending on you know who it is that comes and starts at research programs there exactly so like a lot some people describe dare as like an ai ethics research institute right and i'm like no um it's it's like yeah that's not what we're we're hoping to do and by by virtue of who we are we will so there's two ends of the spectrum that we we're looking at right and i think our advisory committee members when you look at safiyan noble and shira uh maina they they encompass those two ends of the spectrum so the first end is how do you develop how do you do this research in a way that we think is beneficial to to the groups of people that we care about and actually when you say what's in scope and out of scope our focus is you know we're starting with um thinking about you know people in in africa and the african diaspora right like so you know um you know there's no kind of question like i don't have i don't know if i have to explain why but like you know uh black people in general around the world who are very much harmed by this technology and not necessarily benefiting from it so um when you look at shira he's in the area he's in he's in kenya and a lot of his work is on how to you know work on um climate change um and data science right he analyzes bird migration patterns to to that tells you something about the the climate and how it's changing he he was at the first black neon workshop he probably covered his work um food security um and conservation he works on stuff like he co-founded data science africa right so it's kind of like you know how to work on the quote-unquote data science or related fields in a way that is beneficial to certain you know to the groups of people that he cares about on the other end you have sophia who's um um in you know in in the us and she is more on the other end of the spectrum how to you know raise the alarm um uh when we know there are issues um that with technology that's already been built right so we and you know she's uh more uh from the social sciences side right so like for me that encompasses sort of what i want to build with dare right um interdisciplinary have different groups of people um to to be able to work on research that you know we think is beneficial to our communities um and um in a way that's not exploiting the people who are actually you know who might not have phds or whatever but have a lot of knowledge about the these systems and how they're impacting them so i liked what you said yeah it is an undercurrent right of like how do we do this work is is that's how we're building this foundation i mean this institute one of the things that we chatted about before we started recording was that a lot of your focus right now is on institution building for obvious reasons you're building an institution like uh i'm curious what that means for you and also well afterwards i want to relate that back to your experience at google and and the idea around you know how to how to ethics organizations inside large companies like how do we build those so that they have teeth so to speak so that they can be effective yeah that's a very good question um and so i've been going on this fairness rabbit hole as you know and you know i've been like i've worked on things related to math and or documentation or auditing community building like black ai power building you know all the different um kind of ways in which i think you can attack the problem and i have kind of just kind of come to the conclusion like many and of course this is not something new that i'm saying it's just like i said your experience teaches you a lot more than what anybody else writes or says is that if you don't have the right institution and the right structure there's just no way that you can do things quote unquote fairly right so um so that's why i'm i'm kind of working on institution building right i've i've had experiences in academia i've had experiences in industry and when i after i got fired from google i was thinking you know a lot of people were saying well you obviously won't have academic freedom in industry if you want that you should go to academia and i was like that's not true right to me it's a pyramid scheme up here at the top of the you know somebody just tweeted the other day that graduate students make 36 000 a year perhaps right and you know it's like they're in this weird position are they students are they are they workers like do they get vacation or not but they're in this situation for years right very similar to college athletes oh abs 100 which also should get paid exactly so that's where we are right and so um yes and it makes absolutely no sense it's i think it's very very exploitative and so imagine you're doing that work as a graduate student and your advisor controls your life and then you're gonna tell them you know whatever research they're doing is not fair you should have a different sort of direction you you're you should stop how are you going to do that you you will lose your your money you will lose your um career like your future prospects because they won't write you a recommendation if people are on visas you will lose your visa so so ex so um how are we telling people to do the right thing when we know we're not setting them up right with the incentive structure to do the right thing and it's the same thing at work too right like um again what did i i spoke up i got fired so um then why why would anybody do something differently then right like and so so that's why i really believe we have to think about um the um incentive structures and it's not just about for instance labor practices that we're talking about right it's about what kind of work is valued and what kind of work is not valued um you know so i i think you have marielle gray well so her and sudarsa 3 have this book called ghost work how silicon valley is creating a global underclass and they're talking about data labor right so all of this automation that we talk about is sort of pseudo it's not you know real automation is that there's a lot of people behind it labeling data you know doing all sorts of things but they're being exploited they're not being paid right um and so in in in graduate school if you're telling your phd student that they should spend all of this time working on data related work data labor that's the very the most important thing you should think about how you're gathering and annotating data take the time to do this right but then they can't publish their work or they it's not valued or they can't get a job after they graduate again that's an incentive structure and institution building issue right so now there's some people working on journals for instance to be able to for people to be able to um publish on data and there was this new rips um this new europe's um data sets and benchmarks track where we actually um published a paper to for dare so that's what i mean like this is exactly why i'm thinking about the is the the incentive structures right because there's no way you could you know do quote-unquote the right thing if you're in the wrong incentive structure yeah yeah we i did an interview with safe savage uh who researches that area as well um that was a future of work for the invisible workers in ai exactly back in episode 447 you know if you kind of you know chart your path as experimenting with different institutional structures to try to see what works is your decision to start dare uh you know can we infer from that that support organizations aren't enough internal organizations aren't enough there needs to be just an independent uh alternative to kind of traditional research structures what i'm exactly what i'm thinking is so let's say if i went to academia and i just told you i'm trying to spend the time to think about the meta questions how do we build something etc how am i gonna survive like i have to publish tomorrow my students have to you know i'm no 10 year old sorry what are you working on like it's not even an option right so so mike so my hope is that yes um we start these smaller independent institutes where we can actually say stuff um alex was telling me about a talk that she gave the other day and that might not have made a number of people happy but she's like well that's fine because i'm not looking to get tenure and i'm like yeah that's the kind of stuff you can do when you're not looking to get tenure so i think you know it gives us the opportunity to actually advocate for things that we think are important and maybe um slowly those other larger institutions might change or you know have pressure to to do things differently if they know that there are different options like like our institute and if other people create other institutes and honestly um if you look at you know even how i started black nai before black nai i had been involved in a lot of other organizations like for diversity or for this or for that and i was like you know there's no way i can convince you all to do the things that i think we need to do for black people you know i just i fought i tried i this and that and i was like let's start something new and do it the way we think it should be done and this is kind of similar to that right i try this i tried that i tried inside the organizations i tried appealing to you know higher ups i do whatever but you know i found that that's not you know it's not working and so what i want is to have an alternative and even when you look at black and ai right what has you know there's now um latinx and ai queer and ai indigenous disabilities and ai you also have a lot of black and x um i'm lacking robotics black and nero black and physics but i don't even know like there's so many of them right and we can then you know build there is like a network now you have you build power and you can advocate for things collectively um and so hopefully that's what i'm hoping with dare right it's kind of an alternative to what we have right now um and hopefully you know other people can kind of replicate it in a way that not exactly replicated but you know in a way that works for their context um and you know so with dare i can do things like you know think about funding right where is our funding coming from it you know honestly sometimes it feels like pick your poison like there's no really clean money like you know i'm learning all these things but you know you i'm thinking about right like again like i said the meta questions right if we're thinking about ai and where the money comes from you think about technology in general when the government really invests in technology right um it's during warfare or when they're interested in something to do with warfare like so the transistors in silicon valley right um in world war ii you think about machine translation why people were interested in their research has to do with russia cold war you think about darpa and self-driving cars it wasn't because they were like oh we need you know to make cars more accessible you know we need to make sure that blind people can uh very freely move around so let's build that's not what they said they said we we care about you know autonomous warfare right and so how do we expect to come to a different conclusion when from the very beginning our funding our incentive structures every the paradigm that we're using has something to do with warfare and it's the same with industry too like if if all you're thinking about is how to make money for this large huge humongous company that that you know affects the entire world controls the entire world how do we have the space to think about a different paradigm right so like we're hoping to to think about a different paradigm i'm sure like you know not that these paradigms don't exist other people are doing it too but like you know kind of take the time for ourselves to think about what paradigm should we follow starting from the funding um to how we do research um to you know who we are hoping to serve you know we both have a lot of kind of colleagues in industry that are working within larger organizations trying to help them use ai responsibly you know what does it say about that work is it you know futile is it for not is it you know a pessimistic view or is it you know do you do you have examples of that process working correctly that you refer to uh and you know you're just offering an alternative or do you think that that is um you know yeah you know there's fundamentally there is this paper called um what is it the grey hoodie project from the university hoodie project yeah from people at the university of toronto and um and they talk about they say how big tech's tactics are close to big tobacco so they talk about how and they give examples of how like you know the tobacco industries would give lots of money to certain academics who talk who write about how well you know it's not it's unclear if smoking causes cancer or something like that or they would then internally retaliate against people who actually have those kinds of funding i mean of conclusions right or fossil fuel industry whose scientists knew about climate change way back but they were suppressing it and so why you know why wouldn't big tech be like that i mean what is their incentive not to be like that so i i have seen it myself how they capture how they you know people talk about industry capture how they use um research in order to like fight back against regulation so i do believe honestly that the number one um reason that these large tech companies want to have these clinical ethics teams is to um in order to like fight back against regulations so after i got fired you know uh members of congress and representatives sent a letter to google and there was a number of letters they sent first of all they sent letters about you know um you know the the number of black people they have in these ai divisions do they have special like uh training uh in ai except you know a racial equity or any training or impact or anything and they write back and they say oh we have you know these um ergs or whatever you know employee resource groups and we have lots of black people we have this event that event and similarly um they uh wrote a letter to them about lara's language models and their impacts et cetera and they're like we have had hundreds of papers in ethics and fairness you know what i mean but i know for a fact they are actually suppressing more papers about the dangers of large language models you would think that they've learned from their lessons so when they're doing this they they are freely allowed to suppress and persecute people with certain kinds of works but not others you have to ask why and that's that becomes more of a propaganda than research right so i do think that this is their goal but so the people inside could know that and try to fight that right and um i think the way they can fight that is through collective organizing like people before them have done right um polaroid or uh workers organized against polaroids on partnership with apartheid south africa right and that was that had a big impact so it's not that i don't think that people on the inside cannot um cannot change things but it's that they have to be vigilant to understand why they want them there and how their work is being used right if you're meg mitchell used to call it fig lift right she's like fig a fig leaf she's like i don't want to do a fig leaf work you know because like they already do everything else and you do you do the fig leaf work right like you're like just stamping what they say oh we're gonna write your ethical consideration section or we're get we're just not changing the course the direction that you're doing um but we'll we'll sort of do a fig leaf thing that can do much more harm than good do you are there examples in the industry that you look to as you're creating dare um well i mean there's examples of what i don't want it to be like and what that was a huge motivation for you know like the open ai type stuff is not what i want like when open ai was announced i and i have been so clear about this i've never hid kind of um i remember when opening it was announced i was at new york's and i think it was in 2015. and that was before the name change and i had just gotten harassed at some you know bro compa like um a google uh party i was harassed i was having a horrible time i just like i don't ever want to come back to this conference and you know and it was after the whole google gorillas incident and they announced this company that's a this that's supposed to save humanity from ai like the whole world it's all about elon musk and peter thiel and all these people they had 10 deep learning people 100 no interdisciplinary whatever eight of them white men uh one white woman one asian woman um and i know for i like i knew for a fact this was not gonna save humanity and fast forward what are we talking about we're talking about gpt3 we're talking about the dangers of large language models we're talking about how we're worried about things we're talking about how you know there's unsafe products out there etc right so of course like you start at the instep this is exactly what i'm thinking about institution building right so you start at the inception who was at the table where did the funding come from where you know and so um unless you think about that you can't not arrive at uh at the kind of the state that we're in today there are a lot of those kinds of models i am finding out about institutes left and right right now one working on agi one working on some other thing that has like 50 million dollar endowment or whatever you know i get irritated i'm like where is that 50 million dollars coming from i wanted to dominate so i don't have to think but you know um but then i would have to compromise on of course probably where we get the money or something like that so that is you know on the one hand i have models of what i don't want but i do have models of the kinds of grassroots organizing that i've seen that i'm i'm really excited about right so for instance i gave you an example of masakani uh which is a network and it was really beautiful to see right because it grew up grew out of the deep learning in daba uh which is a convene right and so then they create a people there who met their created masakani network and it's really cool it's a whole bunch of people focused on working on natural language processing tools for african languages and their values are very much kind of in line with the kinds of values i'm i'm thinking about for dare um and they grew it super slowly you know and i think now they have a foundation i don't think they have any full-time people um before the show i was i was talking to you about um the this article this wired article i had read about the maori uh who created speech related technology to benefit their community so they they had this um competition uh using um their local radio to for people to send in kind of annotated speech um for like speech to text and other kinds of speech related you know language related technology and they had like hundreds of hours of data right and um then all of a sudden this company i think was called lions land bridge or something this american company wanted to license their data and they you know said no and they they published their um their reasoning and they said you know we think this is the last frontier for colonization um they beat the language out of our grandparents literally were not allowed to speak this language and they were beat up for speaking it and why is this company interested in like you know buying stuff for now it's not it's obviously not because they want to benefit this community so they're like we want to make sure that whatever we do with this data and how you know is is something that benefits us so those are the kinds of models i'm looking at i'm like oh that's awesome like i like that you know how they're doing data stewardship you know um and you know masakani i like their approach for grassroots organizing migente right is another grassroots organizing they're doing such great work i just read their report on for instance border technology um and they're talking they're educating people about what are the different companies involved in this like digital border digital border walls how should we organize they drove palantir out of palo alto right and they had this no tech for ice campaign so so i'm looking at that um them too and and how they're they've been able to be so successful with their grassroots organizing so i'm looking at different kind of you know different kind of models to see what what it is that i like about each of these models and what makes sense for dare yeah yeah one of the things that you mentioned as we were chatting before getting started was this realization that you had that you can't reduce fairness to a mathematical problem like in i think you saw that experience that i see a ton of that elaborate on that a bit and kind of your journey to realizing that and where you see it um how it occurs for you out in the industry you know i mean like my sisters have been saying you know doctors of fianna has been saying this forever dr ruja benjamin has been saying this forever simone brown and you know like the people not trained as engineers and computer scientists have been saying this for a long long time and unfortunately i was um reading philip agrees um it was the most depressing thing in the 90s he wrote um so there was even actually a washington post article about him he was um an ai and then kind of uh became much more of a critic of it he was a became a professor and he was like talking about a number of issues that for instance like people are gonna share their data much more freely with you know for various applications right this is way before social media and stuff and he was like it's not gonna be so much of a big brother kind of thing but people are just going to like share it without knowing without thinking carefully you talked about face recognition and so i was reading this like lessons from trying to reform ai or something like that i'm like you got to be kidding me i don't remember when i mean but it was like lessons from china from ai talks about how the field is not reflexive how it's arrogant you can't get people to think about disciplinary norms and whatever and i'm like oh my god like it's true right and that's what it is it's that when the field feels like it's better than other fields has a lot of power and money thrown at it at this point you have money from the government money from industry money from everywhere you don't have to think about what anybody else says even though these people have been saying this stuff forever um and so uh what my own experience showed me of course you know we can talk all we want about we we we want to reduce fairness um to mathematical equations because first there's again it takes me back to that incentive structure when you think about how you ascend in the academic world in computer science or in you know in um you know engineering in general there is this hierarchy of knowledge i gave a whole talk about the hierarchy of knowledge right certain kinds of knowledge and contributions are valued so if you spend five years working on data related stuff first of all in these conferences if you have a data track it's already inferior oh it's just a data set paper or whatever you know where's the engineer you know that's how they talk um and actually kiri wagstaff gave uh um a keynote uh at html 2012 called machine learning that matters which was basically about this kind of stuff and how what conferences are valuing versus not so that to me makes it such that you want to reduce everything to the algorithm to the math to the whatever you want to not look at it as a socio-technical problem as you know many people in sds have said and when you do that um like the paper there's a paper called fairness and abstraction um they they do a really good job of giving examples of what kind of issues might arise when you do that right you're just like looking at the system in isolation um not thinking about it as as you know as part of a larger system which is like how is it being used what domain is it being used and who is be using it against whom etc then your analysis becomes very different and much more complex but you're not incentivized to do that where are you going to publish what you know you're not going to ascend the people who are going to want to give you 10 you're going to be like oh whatever that's just data or that's just that's just fluffy whatever you know that's what i'm telling is how they talk and so so because of that you're incentivized to be like oh you know what what does fairness mean i have no idea what fairness means right and um you know mimi i know actually was sita pena i think who said um she came to fact and and she um gave a talk about some of her observations there at some other conference workshop and she said what what why you know what does it mean to make systems fair that are punitive that their job is to just be punitive right so um for instance when people talk about risk assessment um they jump to you know using the compound compass data set and then you know christine lom and others like have written about the uh like all the issues with that data set and why people shouldn't just jump to use it um and then they they say okay like you know we we looked at that data set and we have this new algorithm and it makes you know this other metric higher by x percent right what is exactly what does that mean in reality right like what you're doing is you're still locking people up like you know and and you're trying to figure out how you know what does fear mean in this case like you're locking this other person at the same amount so a lot of people don't you know abolitionists don't even think that whole system should exist right so when you're not looking at the entire system and you're just focusing on this math um it's even it's unclear like what uh you know if that's even something that will help or not and many times it can be very harmful i've seen this in the face recognition discussion you know um after joy and i wrote the paper like gender shades a lot of people are like oh okay you know microsoft came out with an announcement saying now that like they've changed their training data you know data and now it's much better now it's all you know accurate it doesn't have um you know the uh for darker skinned women the error rates are not as high but then when you look at all of the scholarship from especially trans and non-binary scholars they talk about how automatic gender recognition should not even exist it shouldn't even be a thing so why are you jumping to making it quote unquote fair right that's because you're not incentivized to look at the whole system so you know and then even if you want to do the right thing even if you want to do the right thing like i i try to do at google if you if you're you're going to get fired then what what does that mean right like why are we writing papers about what to do if everybody's going to get fired if they try to do the right thing so that's what i mean you just cannot um reduce it to this mathematical thing but you could you keep on doing it and our field people keep on doing it because that's what they're incentivized to do and that's what they're rewarded for i think what i love so much about talking to you is that you have this very clear view of all of the challenges the systemic systematic challenges that are kind of inherent and baked into um you know all of these systems that uh you know we struggle against and you know that doesn't uh you don't get jaded you just oh let me try something else let me try something else i'll try something else honestly i think sometimes it's because i don't really think about you know the other option i think is way too depressing is is what i think right the other option of like you know being like i guess this is too big of a problem we can't do anything i just you know it's too depressing right um and then sometimes like when i started dare now i think about oh my god i was doing i was doing this um visualization of the plots of like how much money you have your burn rate when you're there's a red line when you run out of money i don't and i made a mistake and that red line was like literally like this october and i just my heart was just like oh and i and i knew i made a mistake but i'm like oh my god i'm doing something right now where this could be this could be the scenario and it's not just my job it's like all of these people's jobs that are on the line you know and um so when i think about those things i'm like oh my god what am i doing but you know if i don't really if i don't really then you we have to be and and i love this um quote by mariam cabba and i heard it um from ruja benjamin saying it that hope is a discipline it's it's you know we we have to yeah what's the alternative what do we you know and it's not like a lot of times i think we think that things are so far away they're not going to touch us but we're seeing that that is not true right with the pandemic with the wildfires in california with you know all of the ways in which we're all connected in the world like you know if you want a just even a better a better world for ourselves i'm not even thinking about the next generation who should sue the hell out of all of the prior generations for leaving them like a world with cata you know with the climate catastrophe even if we want a different alternative you know i think we should work for it and for me i it makes me feel better to do that right to feel like at least you know i'm trying this other thing um you know this other alternative you know um otherwise it's just too depressing i don't i don't know how to not do that you know yeah yeah yeah um so the initial funding for dare came through macarthur have you identified funding sources beyond that or how that all is going to work yeah that's the big question that i'm working on right now so the initial funding came from macarthur and ford um and also the kapoor center um gave us a gift and um rockefeller foundation and open society foundation so it's all these foundations right now and so now we're applying for you know project-based funding for grants like based on specific projects that we're working on and just like other people you know if there's an nsf grant we'll look into that and you know see if we can apply but i am extremely worried about having a whole institute that is only based on grants so one of the things i'm doing right now is trying to figure out how do we have our own revenue stream and what does that look like um i'm really hoping to have some things that we can experiment with in the next few months because you know we have a bunch of people with expertise um and i think we can provide that expertise in different ways that like are valuable to people um and that help us kind of generate revenue for our institute in a way that gives us a little bit more freedom and independence and flexibility right imagine right now i see something wrong that one of the f you know one of the funders doesn't like they're all know each other and then everybody can just be like sorry bye like you know that that that can happen um and um it's really interesting you know the nonprofit world you realize you know i mean it is because of wealth inequality that this world even exists it's actually really sad um and it's all the people i ran away from like you know eric schmidt you know chan zuckerberg besos and these are all the people who have these large foundations that want to fund tech related stuff so it's you know um so that's kind of what i'm thinking about right now like we're identifying different funding sources um thinking about how to diversify our fun our funding sources our revenues what what would our own revenue stream look like um and once i figure especially the revenue stream part and um we have a few things to experiment with i'll be much happier i'll be you know i'll feel much better about it nice nice now how far along are you are are there uh are there folks that are dare affiliated that have research projects that are spun up and uh things that you can talk to so we have uh we have alex hannah as a director of research dylan uh baker who used to be under me at google too um as a as a researcher slash engineer um we have i think uh two research fellows uh mila and russ sedra mila just joined like this week um and one person who's uh probably gonna join us full-time and in the next month or so and uh yeah so we have for instance one of the projects um that uh i have been working on what we've been working on with uh rasha is this project to analyze spatial apartheid the impacts of spatial apartheid using satellite images uh and computer vision techniques um and that's a project where again all the issues i talked about appear like it takes you a long long time to the innovation is on figuring out the data right like um how to get the data and how to process it how to annotate it but that's very hard to what does that mean spatial apartheid oh spatial apartheid is is basically like segregation but um but it was mandated in 1950 by the group areas act uh in south africa so it's it's a a big um like it's a feature of apartheid you know and um so uh people of european descent could live in certain areas and and everybody else had to live in in you know other areas like townships and the budget allocation was a lot lower for townships of course um and so the question is you know supposedly apartheid has ended legally right but when you look at these aerial images it's so clear like the the delineation is so clear and so the question is can we analyze the evolution of these neighborhoods and how things are changing because we know right it passes the smell test and that you can look at these things visually um and and do an analysis it's we're not just trying to do magic right so um so the question is you know how do how can we use computer vision techniques to do that um when speaking of you know exploitation versus not etc rasa just someone who grew up in a township i mean so this is a very personal project for her so it's like she's you know investigating her own you know like stuff that's related to her it's not like this when people say parachute science right um so that's one of the projects we're working on we just um had a paper on it's um we're working on releasing the data that's one of the things i like about being at dare because i you know we didn't just stop you know publish the paper and like really quickly release the data and we're done we're like okay how do we release the data how do we create visualizations how do we allow people to interact with the data what are what follow-up work are you know we're writing an article for africa as a country i don't know if you know that outlet it's uh it's one of my favorite outlets about about the work and one of the things we want to say is that actually the south african government has to um label townships if they want to analyze the impacts of spatial apartheid what they do is they in the in the census they lump it with just suburbs as formal residential areas but you know you that doesn't allow you to to because townships were created as because of apartheid that doesn't allow you to and this is interesting it's part of a larger kind of issue um was talking about how some of her work i think it's called data voids or something like that talks about how for instance google maps didn't have fabelas in brazil you know right that's a huge huge huge community of people so it's part of this larger thing about you know who whose data is is visible um anyhow but yeah like but that's an example of a project that um we're working on and there's a few others too well how uh how can the community support what you're doing well you know follow us a dare um we're gonna uh and you know on twitter um i think we're gonna also have more stuff on our website just about more stuff we're working on um and you can donate to dare if you're interested we're going to have you know fellowships for people that we're you know we have to think through how to do these fellowships too um and yeah i think that's it you know um and advocate for more funding for these kinds of independent research institutes i i don't want to have to cater to like a billionaire to to get you know uh funding for our institute i would rather apply for a grant that comes from you know public you know tax payers and you know be accountable to that so that's another way i think in which people can advocate for these things and are you still hiring are you bringing on additional researchers yeah i mean we have a lot of requests for hiring and so we have to figure out like i said we have to first build the initial foundational team um and so before i we open it up for like uh applications that will fly like but we've had like hundreds of um people asking about internships and volunteer and and you know full-time jobs so after we set up the initial team then we're gonna be think you know thinking very carefully about what kind of internship fellowship opportunities we'll have what kind of you know other full-time opportunities we'll have i mean that that's the thing about having a small research institute and having to think about funding sources is i can't grow it really fast right i can't like so that's that's the sad part but um and uh the thing about volunteer opportunities that i'm thinking about very carefully is who does that prioritize right a lot of people can't do volunteer stuff because they have to work so i think i feel strongly about people being compensated for for their work very cool very cool well tim knit it has been wonderful as always connecting reconnecting with you and learning a little bit about dare and what you're building there thank you for having me it's it's a lot of fun to come back periodically and kind of reminisce on like how much stuff has changed you know yeah we'll have to be sure to uh to schedule the next one not quite as far out yeah yeah yeah you\n"