The Future of Work and AI: A Conversation with Ben Goertzel

There's still a certain pace of work, meaning that if it redesigns our open Cog pattern matching chip, it means that's great, but it's still going to go through FPGA and testing, and then do a limited production run and then large production run. This indicates that we're not having like five seconds from the initial breakthrough to the singularity, but rather a lag of months to years, not decades. However, what happens during those two years is going to be very weird and messy.

As we take that as a random period of time, what's happening in the world during those two years is going to be very interesting. Already, AI systems have reached a level of capabilities that are obsoleting people's jobs, so it's likely that universal basic income will become a reality throughout the developed world. People may not be able to find employment in their traditional roles, and therefore, they'll rely on UBI to survive. However, this won't happen in countries like Central African Republic, where people already struggle to make ends meet.

As AI systems continue to advance and automate most jobs, we can expect to see a significant exacerbation of the divide between the haves and have-nots. The developed world will be able to indulge in luxurious activities like playing VR video games while living on UBI, whereas in developing countries, people will still be subsistence farmers with no more jobs being outsourced.

This raises important questions about the potential for terrorist activity as countries struggle to adapt to this new reality. With AGI systems becoming increasingly powerful, it's possible that nations may resort to using proto-AGI for defense or offense against other countries. This could lead to a very chaotic and unpredictable world, where nations are unable to deal with each other effectively without the help of AGI systems.

However, as Ben Goertzel notes, this is unlikely to happen for some time, and it's only once we've developed human-level AGI that we can expect significant changes. Once AGI surpasses human intelligence, it will be able to control us by force if necessary. This raises important questions about the ethics of creating such powerful systems.

One potential solution is to use AGI to create a new global system that prioritizes human well-being and prosperity. As Goertzel notes, if we could air-drop smartphones to everyone in Africa, it would be a game-changer for those communities. The development of molecular assembly technology could also enable the mass production of resources like food and energy.

The laws of physics are not yet understood enough to prevent us from creating such technologies. In fact, current limitations are more related to our understanding of materials science and energy storage than to fundamental physical laws. AGI systems would be able to overcome these limitations and create new materials and technologies that we can't even imagine yet.

However, Goertzel's confidence interval for the advent of AGI was much wider 20 years ago, indicating that he believes it will happen sooner rather than later. The pace of progress in AI research is accelerating rapidly, and it's likely that we'll see significant advancements in the coming years.

Overall, the conversation with Ben Goertzel highlights the complexities and uncertainties surrounding the development of AGI systems. While some predict a smooth transition to human-level intelligence, others warn of potential risks and challenges. As we continue to push the boundaries of AI research, it's essential that we prioritize responsible innovation and ensure that these powerful systems are developed and used for the benefit of humanity.

The conversation also raises questions about the role of humans in this new world. Will we become less relevant as AGI systems take over most jobs? Or will we be able to work alongside these systems to create something truly new and innovative? The future is uncertain, but one thing is clear: the development of AGI will have far-reaching implications for humanity, and it's essential that we engage in thoughtful and informed discussions about its potential impact.

Ultimately, the conversation with Ben Goertzel highlights the importance of considering the broader implications of technological advancements. As AI research continues to advance, it's crucial that we prioritize responsible innovation and ensure that these powerful systems are developed and used for the benefit of humanity. By exploring the possibilities and challenges of AGI, we can work towards creating a future where humans and machines collaborate to create something truly remarkable.

"WEBVTTKind: captionsLanguage: enall right everyone welcome to another episode of the twimmel AI podcast I'm your host Sam cherrington and today I'm joined by Ben gertzel Ben is the CEO of singularitynet before we get going be sure to take a moment to hit that subscribe button wherever you're listening to Today's Show Ben welcome to the podcast hey thanks it's a pleasure to be here it's great to have you on the show I'm looking forward to digging into uh our conversation we'll be touching on a number of topics including your work around artificial general intelligence and uh top of mind for many of us the intersection between that and large language models uh before we do I'd love to have you share a little bit about your background and how you came to the field sure thing so I I have a PhD in math from the late 1980s but I've been interested in Ai and AGI and related topics since really the early 70s when I first encountered it in the you know Star Trek in the space 1999 and and and and so forth and uh you know we passed through 1999 and it didn't get to human level AI yet but it seems Seems we're close now right so I mean after after 10 years in Academia teaching math computer science and cognitive science I've been in the software industry building AI systems since the late 90s both research toured artificial general intelligence real thinking machines and applied AI across quite a variety of vertical markets from Finance to biology to language language processing and Robotics I did this software and AI behind the Sofia humanoid robot who became the first first robot citizen so I've been you know playing around with AI of various sorts for a long time but now is by far the most uh most exciting time to to be doing it keeps getting better and better it is a super exciting time to be in the field um you mentioned that you know we're getting close uh you know I'd love to have you Riff on that a little bit and in particular you know what does it mean to get close to AGI how do we know uh when we're getting close to AGI we you know talk about tools like the Turing tests um but that doesn't seem to be a sufficient Benchmark for measuring artificial general intelligence how do you think about defining AGI and how close we get to it yeah first of all I think having a rigorous definition of AGI is not that important any more than biologists need a rigorous definition of life to you know to work on some synthetic biology or analyze viruses and whatnot there is a strong mathematical theory of of AGI you could look at the book Universal AI by Marcus huder who's now at Google deepmind but one of the lessons of this theory is humans are not that General in the scope of all possible General intelligences like if if you think about full-on AGI is the ability to let's say achieve any computable reward function in any computable environment as one overly simplistic definition which Marcus puts forward I mean we're very far from that right like I'm I'm a fairly smart human being I cannot optimize an arbitrary computable reward function like I can barely run the maze in two Dimensions let alone 755 Dimensions right so I'm in what we mean in practice by AGI is having a decent amount of ability to generalize and to extrapolate and creatively leap Beyond one's programming and one's training like roughly as much ability to creatively lead Beyond one's background as as human beings have right and that's sort of pragmatically what we mean by it and I I think we're clearly not there yet with any of the AI systems that we've created so far on the other hand progress in that direction is very very interesting at the moment right and the the up the opposite of AGI I think of as narrow AI which are AIS that do sort of one specific thing that you've configured them to do which could be could be drive a car answer questions based on on a certain knowledge based play a board game and so forth and uh it's not entirely clear even to the greatest experts in the world which of the things that humans do can be solved really well by a narrow AI approach versus which ones or what we'd call AGI hard and really need something with human level general intelligence I mean that's a something on which people have made the wrong judgment over and over again through the the decades that AI field has been around do you differentiate AGI and sentience that that's come up uh you know comes up uh often in that context most recently the whole thing around Google's Lambda llm and um how do you think about the distinction between those ideas well I think AGI is at least a relatively well-defined quantity I mean you have Marcus hooder and that whole theory of AGI is maximizing computable rewards in computable environments and you have Weaver David weinbaum's theory of open-ended intelligence where general intelligence is about a complex system maintaining its boundaries and then seeking to transcend itself and Achieve new functions I mean at least there's not an agreement on what it is exactly but there are you know formal not formal mathematical approaches to it I'd say sentience is Meaningful but remains a bit fuzzier like what's the relation between sentience and sapience and intelligence and and Consciousness that's really more fully in the domain of of philosophy I would say so I mean on on the question of machine consciousness there's a lot of different philosophies out there I tend to be pen psychists and you don't believe that this ballpoint pen has its own species of of Consciousness which is not as complex or dynamic or richly structured as as human consciousness and my own feeling is when you get an AGI system carrying out human-like cognition and intelligence is basically going to have human-like conscious experience but then to nail down what that means and argue for it scientifically remains somewhat of a vexed question right now a concept like sentience or sapience it's a weird blend of Consciousness and intelligence which isn't isn't all that well defined now when it was one that was claimed that Google Lambda was sentient what was meant wasn't really just it was giving smart answers what was meant was it seemed to be responding holistically and emotionally to what was going on like when when you asked when you ask the questions that made it nervous when you asked questions that were sort of maybe hitting some filter or something and it wasn't sure if it was allowed to answer then it would display sort of textual patterns like hemming and hawing and so on that were consistent with with nervousness right so it it appeared to the guy interacting with it to be giving some indication of a holistic emotional response to the sort of tenor of the overall dialogue it it was and then you've seen similar phenomena like that in in people's chats with Microsoft's like Bing chatbot with bang on chat GPT right it's not not just in it's saying smart things is that seems to be responding emotionally as a whole system to the overall like tenor in context that it that it's it's involved in and that's that's interesting it's kind of a slippery and not terribly well-defined thing at at the moment it's not really something that drives my my research at the moment but it's it's interesting you know and thinking about getting closer to general intelligence uh to what degree has the does the recent progress with large language models play into that is that uh yeah do you see that as evidence of getting closer or is it ancillary too I think somewhere in between the two so I I I I don't think that current large language models display a heck of a lot of of general intelligence on the other hand there's certainly evidence that we've come a long way in the AI field generally and I think they can be used in various ways to accelerate progress toward general intelligence and I will talk a bit a bit about that so I mean if you look at a system like chat GPT or Google Lambda and so forth these achieve what seems like a lot of generality relative to the human scale of understanding but they achieve it basically by having a very very general training data set where like the training data says the whole damn web right so if everything in the web is in your training data set you achieve a certain generality just by doing fancy nearest neighbor matching against the whole web because almost anything that people throw at the system has some near matches somewhere on the web so it can throw back stuff which is minor variations and combinations once in its training data set so it's it's achieving what's a lot of generality relative to any one human without a heck of a lot of abilities to generalize right and that's that's a weird thing for us to wrap our brains around because we don't we don't have a great intuition from what it means to have the whole web at the fingertips of your of your memory I mean just as we don't have a good intuition for the difference between a sextilian or a septillion or something right I mean this so there's some some things we can do mathematically and on the computer that bollocks are are intuition now on the other side I'd say some of the results with few shot learning on llms do demonstrate an intriguing ability to generalize but yet there are severe limitations there right like that you don't do these few shot learning techniques on top of llms don't deal well with negation let alone like nested negations or nested negations and quantifier bindings or something like you find in a in a in a math room for a scientific hypothesis so that there is there is some ability to generalize which you can see in few shot learning on llms it is it is real progress It's also has fairly strict limitations right so I could I could see why some people would think these llms are on the path to ATI I mean there's a there's a sort of naive dumbass reason for thinking that which is why they can answer a lot of questions and then there's a there's a subtle reason for thinking that which is I mean few shot learning which is just learning in modeling the activation pattern in the network without updating the weights is able to make some leaps beyond what was in in the training dead even if not really really big leaps and that that's an interesting phenomenon that's not fully understood right you know what I I don't think that Transformer neural Nets in the end represent knowledge in this sort of way that you need for human level AGI I mean I think even though these are deep networks in the sense of having multiple layers I think the knowledge representation is far too shallow and and surface level and and will not be able to make leaps of innovation Beyond training data the way that the way that that people can so as one as one sort of illustrative example you know if you're using a model like music LM to generate music you can imagine training a music LM model on all music up to 1900 or something and then no music from after 1900 then give it prompts asking it to put you know western church music together with a West African poly rhythms What's It Gonna spit out it's not going to invent Jazz right I mean which in a way comes from putting together West African polyrhythms with western church music and its chordal structure I mean it it's gonna it's gonna at best Give You Mozart with a West African beat or something but it's it's not representing these things deeply deeply enough to make him really major creative leap like the invention of jazz or the invention of modern poetry or something or or invention of quantum mechanics right so there's clearly a limit of creativity and generalization the human mind can do and these models just aren't representing things at an abstract enough way to do it now from from a practical standpoint it might be that 95 of human jobs don't need this sort of radical creativity I mean not many people invent jazz or quantum mechanics and in their spare time right so I mean it could be one lesson of this sort of AI is almost all the things that people do for a living just require sort of shallow repermutation of stuff that was that was done already and then you can obsolete almost all human jobs without making an AGI breakthrough just by fancy K nearest neighbor matching against against everything that's that's been done been done already I mean but that that could be huge huge economically and in a humanitarian sense without yet making the leap to something with human level ability at generalization and Imagination so I mean I'm I think there there could be multiple approaches to really get to AGI I mean you could you could simulate the human brain in a much more thorough level including recurrent neural networks and glial cells and you know charge diffusion through the extracellular Matrix all sorts of stuff in the brain they're not in informal neural net models I'm more interested personally right now in hybrid systems that put together neural net symbolic logic systems and evolutionary Learning Systems so sort of neurosymbolic evolutionaries evolutionary systems and I think the symbolic aspect has a capability for abstraction we don't see in current neural Nets I think evolutionary algorithms have a capability for creativity we don't see in current neural Nets and hybridizing these together within an appropriate sort of synthetic architecture could be could be interesting but if if you do take that sort of approach it could be that llms are a powerful accelerated right so I mean one thing I'm working on now is using systems like open AI codecs or salesforces Cogen using llms trained on code to make models that translate English Senses Into predicate logic and term logic and other forms of logic so if if you can basically use llms to translate the whole web into structured logical statements feed these in to a logical fear improver and a neural symbolic AI system I mean then then you may have something with the same scope of knowledge as an llm but much less susceptible to hallucination and and because it can do inductive deduct and abductive reasoning and can combine different different pieces of knowledge together right so I mean that's one example of how llms can be used as a tool to accelerate a somewhat different approach to to AGI and of course if my team in Singularity net and the open Cog project is thinking in that direction there's going to be other teens those other teams also doing that in in their own ways not just thinking how do we double triple quadruple down on gbt type models but thinking how do we use the power that's here to help accelerate other other approaches approaches to to AGI so in that in that sense they may be significant on the path to AGI even though what they're doing now isn't exactly AGI in the sense of the ability to generalize you kind of pose this um you know this kind of limitation of llms with regard to creativity uh in a sense to put it you know high level uh you know their inability to create Jazz based on their training data uh have there been attempts to formalize to formalize that notion the the limitations of llms with regard to generalization well the limitations with regard to creativity are particularly hard hard to formalize I mean that of course there are attempts to mathematically quantify what's what's a creative leak but that's I mean that's not that's not that refined an area of of study right and uh limitations of llms that at generalization I'm I'm not aware of any really sort of Knocked Down rigorous study of this there's certainly a lot of papers giving case studies and and examples showing many many cases where llms fail to fail to generalize in in pretty simple ways I mean so there was there was a recent paper they got a lot of attention on social media and mainstream media on like large language models can do theory of mind right with theory of mind is the phenomenon and cognitive science where like I have a sense of what of what you know like so I'm I may I may know what you can see even if even if it's different the the the the the than what I could the what I can see right like I know I know that right now I see while recording this podcast I see your head in a little box on my laptop and I see a robot behind the laptop but I know that you cannot see that robot which is behind my laptop you just see me right um this theory of Mind originates in human babies during during the first year of Life at various points during their maturation so you can give a large language model various puzzles regarding theory of mind like you know two people are in in in in a room together playing with the toy Bob and Jill then Bob walks out the door into the next room and and shuts the door and scratches his head Ken Jill who was in the room with him beforehand you know because you see him scratching his head right and there was a paper published showing that that large language models are are able to solve some problems regarding theory of mind then when you investigate further you see it does really well at solving problems on theory of mind that we're in Psychology papers before because there's there's so many write-ups about about the exact solution to that problem but it can solve some puzzles that don't seem good in any exact psychology paper before but if you deviate too far from the theory of mind puzzles that have been discussed in Psychology literature that it will screw up right so it's it's a bit subtle like it's not just spinning out only the examples that that you fed to it it is abstracting to some degree but it's not abstracting to the level of a two-year-old child you can deal with examples that are are more Divergent from the cases if they've seen in in the past so how the how to rigorously measure these limitations is is interesting and I'm sure a bunch of academics are are working on it from my own standpoint as a researcher I can see it's limited from playing with it I'm more interested in working on on building AGI systems and in rigorously documenting the the shortcomings of systems with qualitatively obvious shortcomings uh that that's totally fair I think where the question comes from is you know you hear uh I hear casual dismissal of llms is interesting they're you know uh blurry jpegs they just are a poor copy of things that already exist but there's you know an obviously creative element to what they're doing they are you know no one put to that the uh you know the the poem that it created about your mom you know flying a dragon didn't exist anywhere on the internet um there's there's a degree of creativity there uh in your example where you are kind of combining uh previous types of music maybe that's an llm that you know yeah to which some Randomness is injected and it's you know it is creating more things so how do we like formalize these ideas to I'm using all them for music generation and it is it is interesting and I haven't thought about formalizing the limitations that much but I can say like you can take I don't have a model as good as music LM yet but I have some models similar to that trained on on smaller amounts of data and I mean you could you can say like you know play me uh passage of a song with a reggae beat but with a rhythm guitar section that sounds a bit like Master of Puppets from from Metallica and the female vocalist and it will do that right and that's really cool on the other hand it sounds like how a bunch of fairly uninspired Studio musicians who are very talented would do that like it's not it's not it never it doesn't sound amazing right it it but it's remarkably competent that putting the pieces together this is my own qualitative evaluation as as a musician though now what one thing that interests me there is could we improve the creativity of output from music LM type models by using like beam search and making up some mathematical measure of what's interesting like you use Mutual information a relative entropy use some information theoretic surprising this measure and then specifically try to milk a continuation out of Music LM which is a valid continuation with reasonably high probability but also has a high level of of surprisingness value right so I would I would like to milk as surprising things I can out of these models because for for the band I play keyboard in I would like to be able to play a riff and then have an AI model respond and play vacuum riff that like incorporates what I played but with an interesting Twist on it right now what it will do it can play back what you played but it will like homogenize it and and make it sound kind of boring and typical for the for the genre which is not it's still an amazing capability right but it's not it's not it's not what I really want as a musician trying to make interesting music and I think I think you'd see the same thing music I happen to have played with a lot but you I mean you see the same thing in every domain like I I talked to a friend of mine is the editor of cointelegraph which is a crypto magazine and then online crypto scene and they evaluated using chat GPT and other systems to write articles and coin Telegraph and they finally decided like the these are just really boring articles I mean it's it's more like if you got middle school students to write to write articles for your for for your magazine I mean it's competent it starts with a summary tells you the thing it it gives you another summary at the end it raises a few key points so I mean at first blush like wow it's amazing that amazing that an AI can do this on on on on the other hand it's also cleared the articles are more boring than the average article in cointelegraph which is is a decent online Journal but is is not like Atlantic Monthly either right like that that's not the absolute the absolute highest end so I would say in that in that case it's not a mathematical measure but in in that case I mean you could you could take the music that comes out of a music LM type model or the the blockchain articles that come out of chat GPT I mean you could submit those articles to be evaluated by human magazine editors or or by the by the folks who audition students for the Berkeley School of Music or something and it's pretty clear what you would find if they would find these things are competent but mediocre right I mean I mean that's I think that experiment is underway with a certain science fiction magazine yeah yeah I mean Cena various other outlets have grown articles created by by chat GPT and it makes sense because like if you're doing a product review I mean that's it's a boring kind of article right I mean no it doesn't have need that much much drama to it but then much simpler language generation algorithms have been used for a long time to like summarize sports sports scores and and weather reports so what we're doing now is a leap beyond that but it's not it's not a leaked to the level of the average journalist let alone to Mark Twain or James Joyce I mean if you want to position it and how to how to measure that in a mathematical way as opposed to a sort of human and qualitative way is it's I mean it's also an interesting question so you deposit that our path to AGI is likely some hybrid of uh neural approaches symbolic approaches I think there's multiple possible paths to AGI I'm really interested in Neuroscience I've spent some of my life doing computational Neuroscience so I mean I mean I think there's a there's a path to AGI where you really do a non-linear Dynamic simulation of of of of of the brain right I mean you've got amazing chaotic Dynamics in the brain you've got EEG waves that are generated by you know glia and neurons and weird Dynamics in the water Mega molecules in the brain like there's amazing stuff happening in the brain I think we we need a revolution in brain Imaging before we can measure enough about the brain to to really do that right but on the other end people aren't trying very hard right like no no one has tried to take say an ishakevich neuron like the best the best non-linear dynamical model of the neuron the best models of glia and astrocytes model all the you know 350 major networks known in the human brain and trying to put together a real large-scale brain stimulation doing something intelligent attached to it to a robot's body right like no that's a doable thing it can be done for no more money than you know Google Microsoft Facebook Ted centers something spend on on AI r d but I mean no one is no one is doing that I'd hoped Henry Markham would do that with the human brain project but that wound up sort of fragmenting and going in different directions so I mean I think I think that's really interesting it's not the approach I'm I'm now taking I I think that uh sort of Deep mathematical Fusion of methods from probabilistic programming neural net learning evolutionary program learning and probabilistic logic I think I think this can probably get us there faster so I mean the Casual way to look at that is you're making a hybrid system where you're taking it deep neural net for large-scale pattern recognition and synthesis you're taking a symbolic logic engine for for inductive abductive deductive reasoning you're taking an evolutionary programming engine to create create new things and you sort of glomming them together I mean then in reality we're not building like a modular system with a neural evolutionary and symbolic box I mean we've actually done a whole bunch of of mathematics to put to put neural probabilistic and evolutionary systems in a in a common mathematical framework using higher order probability distributions and the homotopia type Theory and and uh you know intuitionistic logic and the whole connection so a whole bunch of math Voodoo that we can't go into in the time we have here but I mean we're trying we're really trying to find the mathematical level that shows that neural symbolic and evolutionary are all just different aspects of the same underlying methods for probabilistic search of of program space right and so I mean key there are mathematical results like Curry Howard correspondences that show there's an equivalence between logic systems programs and categorial abstractions so so really when you're doing a logical theorem proof you're also you're also learning a program and you're also learning an ontology that can be represented in category three like these these AI paradigms that historically have seemed very different like symbolic versus neural or learning versus reasoning actually if you dig in the math behind them there's morphisms between them all but all kinds of different ways different ways of looking at at the same thing so that's a yeah on the surface it's a hybridization of methods for different AI paradigms but we're really trying to make a common math framework that leads leads to to to all of them and then trying to figure out how to deploy trying to figure out how to deploy this at Large Scale which is a very interesting software challenge right or even small scale how uh how close have you gotten to uh a concrete demonstration of this kind of approach well so we we had a system called open Cog that we launched in 2008 with some code going back to 2001 and we were doing various experiments on this sort of AI I mean we published a bunch of Journal papers and we we use this on the back end of commercial AI systems in Financial prediction and in functional genomics and and and so forth but that was more these are more narrow AI applications using a framework that was being developed with it with AGI in mind so what what we're doing now we're basically rebuilding the whole infrastructure of opencog systems we have a new a new framework called open open Cog Hyperion and we're we're trying to build a framework where we can do this sort of AI at at the large scale because what we found with the old version of opencog is to really attempt ATI like things with the system we needed quite large knowledge bases and our system was was just too slow so I mean the the underlying architecture we have this large Knowledge Graph or more properly uh a metagraph which is a more General graph like object that than the graph and we have this big knowledge metagraph living in Ram across multiple machines and then distributed across across multiple multiple machines using using uh distributive processing technology and you need to get metagraphs with many billions of nodes in them and then your neural evolutionary and logical AI algorithms they're actually little programs represented as nodes and links in that same metagraph so you have this big like self-modifying self-reprogramming knowledge metagraph we implemented that in opencog and what we found is to try to do really serious things with it it just is like 10 000 times slower than tensorflow or something it's terrible so I mean it's a very high level of abstraction but I was I was reminded how in the 90s you know in the 90s I was teaching neural Nets in the university and you were doing recurrent back propagation you had a neural net with like 40 nodes it would take three hours to train right and so I mean so you could do a few things now now obviously things are scaled up tremendously because we have gpus and all these libraries about bonding out of them right so I think with the new programming language we built in opencog hyper on which is Meta Meta type talk and the new distributed Knowledge Graph that we built are our hope is to be able to scale up all of our all of our earlier experiments and so we've uh it's been a bit painful to take a year out of doing AI r d to build scalable infrastructure but I mean looking at looking at how much good scalable infrastructure did for deep neural Nets I mean it seems like an interesting thing to do and I mean we're we're also working with a company called simulai out of Florida on the dedicated AGI board which has a you know a chip for hyper Vector manipulations a chip I designed for large-scale graphs pattern matching and a deep learning chip and the CPU like wired together very tightly on on the same board and the and if if that goes well then simulized AGI boards can do for this sort of hybrid AI approach basically what gpus did for the Roman approach meaning fairly modest variations and algorithms that have been the literature before suddenly you deployed them at greater scale and they started they started doing doing magical seeming stuff right so if if all goes according to this planned it's probably early next year by the time we start to see really amazing results put out of this I mean it's not like five or five or ten five or ten years or something now that's that's much sooner than where I thought you were were going I thought you were gonna need Quantum Computing to materialize or something like that I would love to see Quantum Computing materialize but I I think that the new chip I mentioned just is three years out just because we're now in the software simulation phase you need to go to FPA fpga we need to go to the hardware but I mean we already have the distributed Knowledge Graph running across many different machines we have a limited version of the of our inter preter working and we're we're working on using uh row calculus which is an advanced form of process calculus so the next step neon Pi calculus to allow it to use multiple processes on this on a single machine because we you know we need to go beyond match reduce like mapreduce is good for something that's matrix multiplication based but we what we're doing isn't just mapping I mean if you're into functional programming you need you need not just fold and unfold you need like chronomorphisms and future morphisms you you need a whole bunch of of ways of holistic processing data structures Beyond mapping so if you look in everything everything that Amazon Google Facebook and so on has done is pretty much based on using mapreduce or it's variations to to take neural net architectures and then run them across massive multi-gpu server Farms right so so what we need is some more sophisticated math to take take algorithms for large-scale graph manipulation and efficiently efficiently run them across multi-gpu server farms and I think that's what we're working on mostly this year I think once that's done we can we can start getting interesting things and I mean that probably there's a few different approaches we're taking in parallel so one approach is just take llms extract all the knowledge from an llm into a distributed logical knowledge base do some symbolic logical entrance on there and see if you then get something llm like but that can compare its other instances now with its other instances in the past and that can reason about whether it's various thoughts and statements are consistent logically coherent with each other right and that's that's not AGI but that that would be you know Elon Musk is thrown around the term truth GPT but all he really wants to do is not make politically correct filters on on chat GPT right if you if you take everything out of an llm put it into a distributed logical knowledge base and do logical inference on that that that's really the truth truth GPT right so I mean I mean that's it that's that's one thing you can do the other interesting thing to do we're looking at Virtual Worlds so we've been playing with Minecraft and we've been looking at various metaverse environments and look looking at using using the opencog hyperun system to control societies of little agents buzzing around buzzing around in in the virtual world and the the experiment I really want to do there is get a community of Agents buzzing around in Minecraft or some other virtual world but we're starting with Minecraft make them do Collective activities together so they have to work together to build stuff to achieve their goals I want to make them invent their own language for communicating with each other like you you might remember there was some total news report maybe three three years ago like Facebook chat box have invented their own language which is great like I had I had software programs invent their own formal language 30 years ago and no one no one wrote me up in the news about it right have a bunch of little agents running around in Minecraft and they're they're making actual noises to communicate with each other and they're collaborating on on you know building structures to to be able to get somewhere they want to to get get rewards and they need to communicate and coordinate to achieve their goals can you get them to sort of invent invent their own language in in that world to to achieve to achieve their goals so I think that's a really interesting experiment I'd rather do it with robots but and I mean we are working with robots with Sophia and Grace in the Hanson robots so robots robots are still a pain it's quite straightforward to do this in the in in a virtual world at the moment so I think the direction of making the real truth GPT is really interesting it's a little more commercial it could set the work could set the world on fire right on the other hand I think making a bunch of little agents in a virtual world that cooperate socially to achieve goals together and invent their own sort of new communication paradigms to do it I think that's more likely to lead to fundamental AGI breakthrough but the beauty of it is you can connect these two things through the same knowledge base right like like a I mean you and I learned our own separate things but we can't wire our brains together we have to communicate by making noises and drawing pictures but I mean if you have if you have a real truth GPT then you have a community of Minecraft agents to learn learn to build stuff and develop their own language I mean you can you can develop machine translation between the Minecraft agents language and English then you can you can you know hybridize these two open Cog hyper on based AGI systems together if you want to right so I mean so those there's a lot of interesting potential directions you can you can you can go here and then the the other ingredient is with with Singularity net we've built a deployment fabric where we can roll out systems like this on the decentralized infrastructure so you could have like 100 server Prime sitting in different countries and pieces of the network underlying this running in all these different places and the and these all these all coordinate together I mean of course without any one Central controller because using using blockchain for decentralized coordination I mean that that that doesn't intrinsically help you make the AGI smarter but it it means that your AI system doesn't necessarily end up monolithic and nothing to run on on the huge server Farm because we're starting with this wildly distributed decentralized infrastructure from the get-go you mentioned Sophia in there uh can you talk a little bit about what that system uh is really intended to demonstrate it it has received a lot of criticism for um I don't know one article says it is is it a show robot um is a parlor trick kind of thing of course it's a sure of course it's a show robot but it's not just a parlor trick I mean I mean so Sophia is a character created by David Hansen and it's a obviously a hardware design and a sort of squishy wear design in the face developed by David Hanson who's a he's an amazing robot assistant and creative artist and the software behind Sophia has actually changed over and over again since 2015 when Sophia was was was first created right so I'm in the I think initially Hanson robotics put a sort of role-based chat bot behind it then I came on board hessnerabox was their Chief scientist for a while we then shifted to using a combination of neural chat box with with open Cog for some some some reasoning behind it so now now behind Sophia Grace Desmond and the other handsome robots I mean you have you have a hybrid you have a hybrid dialogue engine I mean we have we have open Cog in there which is doing some reasoning to generate some some responses you have a rule-based chat system in there the Hanson robotics created the Hanson AI system that responds to some basic knowledge about the robot and who it is I mean you have you have a gpt3 behind there with a cusp we don't use chat gbt but we use a custom set of prompts that were crafted for for gpt3 and when So when you say something to the robot then there's a decision process it checks what the different sub-dialog engines behind the scenes have to say it picks which response makes the most sense and responds with that then there's some subtly behind the scenes because whichever sub dialogue system responded of course that response has to be known to all the other sub dialogue systems so you have multiple dialogue systems with some notion of of shared State behind behind the scenes so I mean it's a obviously a year and a half ago chat GB G3 wasn't an ingredient there and we've also transitioned from the old version of open Cog to using the new version opencug hyper on though so it's really I mean I'm like a human being like we learned but we got the same brain year on year right I mean Sophia has had radical brain upgrades year on year as as the AI field has has developed I mean I think David Hansen is a brilliant sort of theatrical Mastermind right like I mean he worked at he worked I mean he worked into he worked at Disney for a while early in the in his career and he he has been both trying to advance Ai and Robotics and he's been trying to sort of show the world what with a kind loving friendly emotional intelligent robot really look like so certainly some people along the way have assumed that Sophia was much smarter and more human than she really is but I would say that's not a specific problem to Hanson robotics like right now many people think chat GPT is much smarter than it than it really is like that's that's uh that's that's how people roll and right there is the responsibility then to tell people what's actually going on but what I found with Hanson robotics is even when you tell people what's actually going on they don't care and don't believe you they're just like no she really loves me I don't care what you say right I mean and that's I mean that I see that like I have a two-year-old daughter she believes her stuffed animal really loves her right I mean that's a that's what that's what that's what people want that's what people want to do uh for better and For Worse yeah on the topic of responsibility how do you kind of Reason through the the ethics of AGI some critics say it's you know we shouldn't be doing it uh there's uh you know certainly a lot of conversation around AI safety and I mean what what humans should or shouldn't what humans should or shouldn't do is a vexed question I mean it's not it's not clear humans now are happier than we were in the Stone Age so I mean I don't know if I don't know if morally we should have developed Agriculture and civilizations and and machines it depends depends on your risk tolerance it depends on how much you value you know happiness and contentment with your everyday life versus the joy of of do of doing new exciting things right I mean clearly curly Factory workers in the industrial revolution were a lot less happy than Kalahari Bushmen right so I mean I mean that's so I think for better and worse Humanity keeps willing to do high risk novel things which is probably tied in with why we're not Apes anymore right I mean that's sort of what what people do and I mean developing AI is certainly along those lines it's it's high risk High reward it's bringing us into New Frontiers that we don't that we don't fully understand and I think there there is non-trivial existential risk with developing AGI but you don't necessarily see it as overwhelming existential risk I mean some some folks see only a negative AGI outcome that's very irrational I think if we really wanted to be maximal rationalists we have to say we have no idea what the is going to happen right I mean I I think the bottom line is the confidence interval is is really really wide because if once you have something 10 times as smart as a human there's certainly no overwhelming reason to think it's going to be nasty but I mean there's not an overwhelming rational reason to think it's going to be it's going to be nice either like we just we just don't know what's going to happen any more than the mouse could predict you know whether tencent or Badoo is going to become dominant in the Chinese in in the internet Market as a mouse that doesn't understand what's what's going on I mean I think intuitively and sort of spiritually if you would like I I have a strong optimistic sense about the the future of AGR like in in my heart I feel like it's going to be for the good for for for Humanity and I can see that some people have a similar opposite intuitive gut sense like they have a strong gut sensitive it's going to come out come out badly but I don't I don't think that's really a fully rational thing that that that that's more a matter of our our heart feeling about it because when you when you rationally plot it out I mean okay you're gonna build quantum computers you're going to build femto computers we don't have a theory of quantum gravity we don't know all the things that femto computer can do what will the what will the Super AGI discover about that like I mean and we we fundamentally fundamentally don't don't know right I mean my my gut feeling is you know if we raise up a baby AGI with love and compassion and we have it doing education and medicine and Healthcare and Science and a bunch of bunch of good things as that baby AGI goes into adulthood it's going to overall feel well disposed toward human beings and the resources that human beings use are trivial compared to the amount of energy in the solar system let alone let alone the the universe right I mean there I see no reason to assume an early stage superhuman AGI is going to be a psychopathic megalomaniac that needs to use every Quantum of mass energy for its own its own you know self-gratification I mean even even in that description you're kind of anthropomorphizing that the AGI and the relationship between us and the AGI um is is there a way to kind of separate those there is but it's hard to talk about right I mean I think I think I antrimorifies Less in my mind than I do in verbalizations because we just don't have a language for talking about these things I mean yeah anthropomorphize you would say a super AGI could view humans like we view the squirrels in the in the in Yellowstone Park or something right like I mean we we want we want to preserve them yeah I was responding mostly to the the idea of raising the AGI what does that even mean I mean in the current way of doing things I mean I mean it means a lot because we have Sophia Grace and Desdemona robot and we talk to them and we we enter we interact with them right I mean so we we do I do have I mean in my life I have these humanoid robots that I am actually talking to and and they're seeing me and they're learning from the they're learning from the interaction with me and I can see if I put these systems to work selling Amway door-to-door versus if I put them to work in an elder care facility or a kindergarten they're totally learning different things about how they interact with people I mean the one case they're objectifying people like I'm using this person the only thing I care about is to get them to sign them online and get their money on the on the other hand I mean they fundamentally want the elderly person in the Elder Care Facility or they want the kid in in the kindergarten to be to be fulfilled and happy and that they're trying to establish what uh Martin boo bear the philosopher would have called like an eye vowel relationship right and you don't need that as an Amway salesman you get them to sign on the dotted line then you move on to the next house right you you you you do you do need that to be effective in in Elder Care or in early childhood education so I mean I think we and there's a there's a anywhere there's a concrete level here actually yeah yeah one question that comes up for me just hearing you describe this and maybe it kind of goes back a bit to one of my first questions around like sentience versus AGI um which you kind of clearly delineated it's not clear to me that AGI uh necessitates agency are those two more more closely linked that I'm that I'm thinking right now I think they are but it's a subtle point I think for I think for cognitive architectures that even vaguely resemble human cognitive architectures those two are very tightly bound together general intelligence and and agency but I think if you were exploring cognitive architecture space more broadly and looking at like radically non-human Minds it's it's not it's not as as clear like you if you think about like I mean the mathematical fear improver that could prove brother and brother are more and more General theorems than any possible logic system how could you make that really really really generally intelligent without a self model or a model of self another or agency it's it's somewhat of an of an open question whether agency would emerge at some abstract level there but I think in in the human mind or in the human-like cognitive architecture which is I think how we're going to make the AGI breakthrough just because it's what we understand better than other more alien forms of cognitive architecture there I think they're very tightly combined because the formation of Concepts in the human mind is very closely Tied by analogy to the formation of the self-concept and it's I mean it it's by learning how we distinguish ourselves from the environment and our self relates to others I mean that's how we learn to relate a concept to other Concepts and you you can see this in a lot of psychology results like people who are what are called thin boundary meaning they're very influenced by what happens to them they're very emotionally sensitive will tend to develop Concepts that are more flexible and thin boundaries and we'll adapt more more readily to new information whereas people who are very thick boundaries like they they distinguish themselves from their environment very rigidly they're not much influenced by what other people say or do these people tend to develop very thick boundary Concepts and they have a hard time adapting their Concepts to what what happens around them right which can be good or bad depending on what you're doing but you can see from that example like the way we form Concepts in general and the way we form our self-concepts are very closely tied together and this is a big theme in developmental cognitive Neuroscience so I think for for people these are very closely tied together and they are for the open Cog systems that we're exploring now in my own AGI r d but whether they're of necessity tied together I I think if he took any AI system which is controlling an embodied agent in the physical world together with other similar embodied agents in that same physical world right I think which is what humans are doing it's what mobile robots would be doing is what virtual characters would do in a game world I think that setting naturally leads itself to agency being closely linked with Concept formation but of course that's not the only thing that an intelligent mind could could do in the end right it's just what humans and and the robots we build we build are doing and that I mean here as with the question of existential risk you're faced with like whoa we're going into like a a very large Uncharted domain that we that we we barely understand that don't have the theoretical or intuitive tools to to Grapple with and all we can do is take it step by step right so as I'm looking at it now my goal my main goal as a researcher is make an intelligence with roughly human level roughly human-like general intelligence that's about as smart as you and me and then once you get there then that system is going to help you go on go go on to the next level and plus once we get there we're going to learn so much that our current Ai and cognitive theory we'll probably start to seem seem like classical physics relative to quantum gravity or something right so I mean I think there's there's a limited as fun as it is there's a limited extent to which we can we can think uh five steps ahead in this sort of domain yeah related to uh I guess related to to risk you know one of the things that you've spoken out about is um maybe the politics of AGI you know in particular with regard to kind of control of of AGI and the hands of a few large companies or large government agencies yeah this is going to be it's going to be a big deal and I've put a lot of work into building the infrastructure to correctly deal with the Advent of AGI once it happens which is a sort of a sort of ballsy thing to do because I mean it's of course the notion that my team will be the ones to achieve AGI first well I have a fair level of confidence in that not an absolute confidence I I understand the most most people aren't necessarily go going to buy that because they they don't see everything I do in terms of the underpinnings of the systems we're building but if you do believe that like if if you believe we're say three five six years away from making an AGI breakthrough with open Cog hyper on running on Singularity net right then then what happens once you get there right I mean that that's it you could say Okay post Singularity once you have an AGI that's ten times as smart as people okay none of us can figure out what's going to happen but can you think through the sort of end game of the the pre-singularity era right I mean and that then so what happens once you've demonstrated you have a true AGI system that can really think like people and everyone everyone sees that it's not and it's not just a narrow AI right then I think what could happen very quickly is some jack booted thugs come to come to your house and force you to hand over the keys to the thing right I mean I I think I I think this clearly is gonna everyone will see this is the biggest event in the history of the human species and this uh I mean this could lead to all sorts of techno Thriller level scenarios and what I would like to see happen is for the first AGI to be rolled out in the manner comparable to the internet or Linux right like you want to be rolled out across every country at once you want to be rolled out everywhere in the world you want not just the source code to be open you want the knowledge bases that are that are learned to be open you want the code to be running on thousands of different machines everywhere so that there's no one person you can shoot and stop the singularity from happening and and there's no way for any any one government to take the thing over and try to use it to gain the gain a you know a military or financial advantage you really want it to be distributed everywhere and there the infrastructure we built in Singularity net and the new net another Allied sort of blockchain based AI platforms this this and it enable enables that I mean it also it also enables us to have a decentralized infrastructure for for narrow AIS along the way which which can also be interesting but it will it will demonstrate its value to the greatest level if whoever makes the big break through the AGI chooses to roll it out on the decentralized infrastructure rather than on the big tech companies server Prime which is operating in close alliance with the the government of some particular Nation mm-hmm so do you necessarily think that uh that AGI will be the result of uh a big breakthrough I'm trying to think about this question and the degree to which it makes sense I mean if you think about the way AI as a field has evolved like you know there have been kind of these you know there continue to be these Stepping Stones like uh you know but there's also chat GPD which is kind of breakthrough-ish or at least it appeared to us as a breakthrough I don't think that was the Breakthrough I think I think the Breakthrough was burnt made within within Google brain yeah once you have the the paper attention is all you need was a real breakthrough right right and so but on top of that in order to have the thing that broke out uh which kind of defines a breakthrough you you a bunch of steps like rlhf and other things well sure I mean I mean that that's like you know what was the Breakthrough the the atom bomb or the development of nuclear physics before that right I mean so you could pinpoint it however you want but I think if we look at how the AI field is developed in recent years you had this sort of three-year Burris of amazing advances in computer vision starting from maybe Alex net or something in the beginning of of CNN's and it was a few years from that to face recognition becoming a commodity technology right on everyone's phone and in NLP you would say yeah you advert and attention is all you need that was what 2017-18 and this again like four year yeah four years you have you have so it's six years to chat GPT right so five years whatever so I mean I think the reason these things have taken three to five years is there happening at human speed like that's how long it takes people to type in papers and go to conferences and talk and for people to refocus their career from one thing to another right it's not it didn't have to take a number of years but right and I think that's kind of getting at where my question was coming from uh and that is you know will the will the Breakthrough be known as the Breakthrough when it actually occurs or will it be kind of this continuous kind of building on I mean look look like a breakthrough it may not have looked like a breakthrough to the average person it certainly looked like a breakthrough to to everyone in in in in the AI field though right so I mean I think it I think at very least it would be like that so I mean if you for sake of discussion assume open assume my own project gets there I mean suppose suppose that you have a system that's chat GPT like but can actually can actually reason and and understands you know the relation between what what it says and empirical reality as in terms of what it sees through a camera or sees in the database and suppose suppose that you do have a bunch of little guys running around in mind scrap Minecraft and they really are communicating back and forth in their own language and they look like a bunch of primitive tribes people right I mean so I would say when we get there that will look like a breakthrough to 99 of people in in in in the AI field and they'll be like whoa like there's a there's a huge leap here right now in indeed the average person may not be able to tell whether that's an epocal breakthrough or merely a breakthrough on the level of chat GPT or alphago or something right so I'm in on the other hand what would be the lag between that and and like an avatar that appeared on everyone's phone that clearly had superhuman level general intelligence I mean it it might be a nine month or a year lag by my best guess I don't think it's a it's a 10-year leg certainly if this thing can help you develop the next thing then things start to accelerate yeah and there's so once you get to a human level AGS system like that yeah I mean you already have systems that can do simple Python Programming right so you can it's not hard to see if you had something with a measure of general intelligence it's going to be able to learn to update its own source code I mean that now that's still that's still a slow sort of experiment though right because even if even if it could upload its own source code it's still going to run experiments you can upload it can modify its code but then it has to deploy that across a lot of machines just to run that experiment it has to see if it if it if it worked or not right so I mean there's still a certain pace of work so I mean if it if it redesigns our open Cog pattern matching chip I mean that that that's great it's still going to go through fpga and and and testing and I mean it's still gonna do a limited production run and then do a large production run so I mean I mean there's you're still not having like five seconds from the initial breakthrough to the to the singularity but it on the other hand you're certainly looking at a lag of months to years not not not decades right which now what's happening during those two years if you take that as a as a causing random period of time what's happening in the world during those two years is going to be very weird and and messy right because it I mean presumably we already had chat gbt level systems obsoleting people's jobs so what what you're going to see is Universal basic income get rolled out throughout the developed world as people realize quickly that agis are taking most jobs you're not going to see anyone give Ubi in the Central African Republic because that's not what people what people like to do in those countries don't have their own budget for it so you're you can just see like a incredible exacerbation globally of the diverts between haves and have-nots right with the the developed world is all leaning back playing VR video games living on Ubi where while agis and robots do 95 of the work while the developing world is 95 subsistence farming with no more jobs being Outsourcing them so what what level of terrorist activity you start to see in that setting I'll I'll leave it to you to figure out what what what level of attempts to use proto-agi for defense or offense on the part of various countries I mean you can see you can see the potential for a very weird few years while this transition is is happening right and I mean I wish I wish I thought that was going to be a beautiful smooth transition but like right now we have major countries going around randomly and uselessly blowing up other other people in the world right I mean I mean it's we we can't we can't even deal with ourselves without without a transition to human level AGI so at what point do you get to the optimism that you described earlier after the AGI is a couple times human level intelligence because then then the world is not being being run by us like hypertrophied monkeys anymore right we we we we we have help by uh by systems that are smarter than us and hopefully more compassionate and well balanced as well uh I mean we're still the the monkeys and so therefore the system is controlling us by force presumably I don't think it would have to no I mean I think once you've abolished scarcity we're in a different domain right but she says scarcity is not abolished for everyone right not initially but imagine the super AGI created the direct solurian molecular assembler and just literally airdrop them everywhere in the world that then solar powered right got it so free energy free food resources don't matter asymptotically approaching free or whatever yeah I mean I mean right right now right now we could air drop smartphones to everyone in Africa if if we felt like it right I mean a billion smartphones we could manufacture we're not quite doing that but smartphone penetration is larger and larger right so I mean if if you really did develop a molecular assembly you could Mass produce them in a huge Factory you could airdrop them from drone drones everywhere people could feed in matter it's solar powered they can they can 3D print whatever physical object they want right I mean nothing in the laws of physics that that prevents that from from happening we don't know how to build it now could an artificial engineer would twice the intelligence of a smart human do that most probably we've gone we've gone a bit a bit uh afield from machine learning per se no all of this raises really interesting and important questions um so uh I appreciate you indulging them no I mean I I love I love I love to think about this stuff I mean I've most of my day I spend working on nitty-gritty computer science stuff rather than rather than thinking about the the broader future but I'm thinking about the broader future my whole life right and I mean what's what's cool is even the rational part of my brain feels we could be only years off from from seeing these science fictional things come come to pass right where it's a my confidence interval width for the Advent of AGI was much larger 20 years ago the the than it is now awesome well Ben thanks so much for taking the time to chat and share a bit about what you're up to and thinking about sure no it's great great stuff to talk about thanks for your own time too thank youall right everyone welcome to another episode of the twimmel AI podcast I'm your host Sam cherrington and today I'm joined by Ben gertzel Ben is the CEO of singularitynet before we get going be sure to take a moment to hit that subscribe button wherever you're listening to Today's Show Ben welcome to the podcast hey thanks it's a pleasure to be here it's great to have you on the show I'm looking forward to digging into uh our conversation we'll be touching on a number of topics including your work around artificial general intelligence and uh top of mind for many of us the intersection between that and large language models uh before we do I'd love to have you share a little bit about your background and how you came to the field sure thing so I I have a PhD in math from the late 1980s but I've been interested in Ai and AGI and related topics since really the early 70s when I first encountered it in the you know Star Trek in the space 1999 and and and and so forth and uh you know we passed through 1999 and it didn't get to human level AI yet but it seems Seems we're close now right so I mean after after 10 years in Academia teaching math computer science and cognitive science I've been in the software industry building AI systems since the late 90s both research toured artificial general intelligence real thinking machines and applied AI across quite a variety of vertical markets from Finance to biology to language language processing and Robotics I did this software and AI behind the Sofia humanoid robot who became the first first robot citizen so I've been you know playing around with AI of various sorts for a long time but now is by far the most uh most exciting time to to be doing it keeps getting better and better it is a super exciting time to be in the field um you mentioned that you know we're getting close uh you know I'd love to have you Riff on that a little bit and in particular you know what does it mean to get close to AGI how do we know uh when we're getting close to AGI we you know talk about tools like the Turing tests um but that doesn't seem to be a sufficient Benchmark for measuring artificial general intelligence how do you think about defining AGI and how close we get to it yeah first of all I think having a rigorous definition of AGI is not that important any more than biologists need a rigorous definition of life to you know to work on some synthetic biology or analyze viruses and whatnot there is a strong mathematical theory of of AGI you could look at the book Universal AI by Marcus huder who's now at Google deepmind but one of the lessons of this theory is humans are not that General in the scope of all possible General intelligences like if if you think about full-on AGI is the ability to let's say achieve any computable reward function in any computable environment as one overly simplistic definition which Marcus puts forward I mean we're very far from that right like I'm I'm a fairly smart human being I cannot optimize an arbitrary computable reward function like I can barely run the maze in two Dimensions let alone 755 Dimensions right so I'm in what we mean in practice by AGI is having a decent amount of ability to generalize and to extrapolate and creatively leap Beyond one's programming and one's training like roughly as much ability to creatively lead Beyond one's background as as human beings have right and that's sort of pragmatically what we mean by it and I I think we're clearly not there yet with any of the AI systems that we've created so far on the other hand progress in that direction is very very interesting at the moment right and the the up the opposite of AGI I think of as narrow AI which are AIS that do sort of one specific thing that you've configured them to do which could be could be drive a car answer questions based on on a certain knowledge based play a board game and so forth and uh it's not entirely clear even to the greatest experts in the world which of the things that humans do can be solved really well by a narrow AI approach versus which ones or what we'd call AGI hard and really need something with human level general intelligence I mean that's a something on which people have made the wrong judgment over and over again through the the decades that AI field has been around do you differentiate AGI and sentience that that's come up uh you know comes up uh often in that context most recently the whole thing around Google's Lambda llm and um how do you think about the distinction between those ideas well I think AGI is at least a relatively well-defined quantity I mean you have Marcus hooder and that whole theory of AGI is maximizing computable rewards in computable environments and you have Weaver David weinbaum's theory of open-ended intelligence where general intelligence is about a complex system maintaining its boundaries and then seeking to transcend itself and Achieve new functions I mean at least there's not an agreement on what it is exactly but there are you know formal not formal mathematical approaches to it I'd say sentience is Meaningful but remains a bit fuzzier like what's the relation between sentience and sapience and intelligence and and Consciousness that's really more fully in the domain of of philosophy I would say so I mean on on the question of machine consciousness there's a lot of different philosophies out there I tend to be pen psychists and you don't believe that this ballpoint pen has its own species of of Consciousness which is not as complex or dynamic or richly structured as as human consciousness and my own feeling is when you get an AGI system carrying out human-like cognition and intelligence is basically going to have human-like conscious experience but then to nail down what that means and argue for it scientifically remains somewhat of a vexed question right now a concept like sentience or sapience it's a weird blend of Consciousness and intelligence which isn't isn't all that well defined now when it was one that was claimed that Google Lambda was sentient what was meant wasn't really just it was giving smart answers what was meant was it seemed to be responding holistically and emotionally to what was going on like when when you asked when you ask the questions that made it nervous when you asked questions that were sort of maybe hitting some filter or something and it wasn't sure if it was allowed to answer then it would display sort of textual patterns like hemming and hawing and so on that were consistent with with nervousness right so it it appeared to the guy interacting with it to be giving some indication of a holistic emotional response to the sort of tenor of the overall dialogue it it was and then you've seen similar phenomena like that in in people's chats with Microsoft's like Bing chatbot with bang on chat GPT right it's not not just in it's saying smart things is that seems to be responding emotionally as a whole system to the overall like tenor in context that it that it's it's involved in and that's that's interesting it's kind of a slippery and not terribly well-defined thing at at the moment it's not really something that drives my my research at the moment but it's it's interesting you know and thinking about getting closer to general intelligence uh to what degree has the does the recent progress with large language models play into that is that uh yeah do you see that as evidence of getting closer or is it ancillary too I think somewhere in between the two so I I I I don't think that current large language models display a heck of a lot of of general intelligence on the other hand there's certainly evidence that we've come a long way in the AI field generally and I think they can be used in various ways to accelerate progress toward general intelligence and I will talk a bit a bit about that so I mean if you look at a system like chat GPT or Google Lambda and so forth these achieve what seems like a lot of generality relative to the human scale of understanding but they achieve it basically by having a very very general training data set where like the training data says the whole damn web right so if everything in the web is in your training data set you achieve a certain generality just by doing fancy nearest neighbor matching against the whole web because almost anything that people throw at the system has some near matches somewhere on the web so it can throw back stuff which is minor variations and combinations once in its training data set so it's it's achieving what's a lot of generality relative to any one human without a heck of a lot of abilities to generalize right and that's that's a weird thing for us to wrap our brains around because we don't we don't have a great intuition from what it means to have the whole web at the fingertips of your of your memory I mean just as we don't have a good intuition for the difference between a sextilian or a septillion or something right I mean this so there's some some things we can do mathematically and on the computer that bollocks are are intuition now on the other side I'd say some of the results with few shot learning on llms do demonstrate an intriguing ability to generalize but yet there are severe limitations there right like that you don't do these few shot learning techniques on top of llms don't deal well with negation let alone like nested negations or nested negations and quantifier bindings or something like you find in a in a in a math room for a scientific hypothesis so that there is there is some ability to generalize which you can see in few shot learning on llms it is it is real progress It's also has fairly strict limitations right so I could I could see why some people would think these llms are on the path to ATI I mean there's a there's a sort of naive dumbass reason for thinking that which is why they can answer a lot of questions and then there's a there's a subtle reason for thinking that which is I mean few shot learning which is just learning in modeling the activation pattern in the network without updating the weights is able to make some leaps beyond what was in in the training dead even if not really really big leaps and that that's an interesting phenomenon that's not fully understood right you know what I I don't think that Transformer neural Nets in the end represent knowledge in this sort of way that you need for human level AGI I mean I think even though these are deep networks in the sense of having multiple layers I think the knowledge representation is far too shallow and and surface level and and will not be able to make leaps of innovation Beyond training data the way that the way that that people can so as one as one sort of illustrative example you know if you're using a model like music LM to generate music you can imagine training a music LM model on all music up to 1900 or something and then no music from after 1900 then give it prompts asking it to put you know western church music together with a West African poly rhythms What's It Gonna spit out it's not going to invent Jazz right I mean which in a way comes from putting together West African polyrhythms with western church music and its chordal structure I mean it it's gonna it's gonna at best Give You Mozart with a West African beat or something but it's it's not representing these things deeply deeply enough to make him really major creative leap like the invention of jazz or the invention of modern poetry or something or or invention of quantum mechanics right so there's clearly a limit of creativity and generalization the human mind can do and these models just aren't representing things at an abstract enough way to do it now from from a practical standpoint it might be that 95 of human jobs don't need this sort of radical creativity I mean not many people invent jazz or quantum mechanics and in their spare time right so I mean it could be one lesson of this sort of AI is almost all the things that people do for a living just require sort of shallow repermutation of stuff that was that was done already and then you can obsolete almost all human jobs without making an AGI breakthrough just by fancy K nearest neighbor matching against against everything that's that's been done been done already I mean but that that could be huge huge economically and in a humanitarian sense without yet making the leap to something with human level ability at generalization and Imagination so I mean I'm I think there there could be multiple approaches to really get to AGI I mean you could you could simulate the human brain in a much more thorough level including recurrent neural networks and glial cells and you know charge diffusion through the extracellular Matrix all sorts of stuff in the brain they're not in informal neural net models I'm more interested personally right now in hybrid systems that put together neural net symbolic logic systems and evolutionary Learning Systems so sort of neurosymbolic evolutionaries evolutionary systems and I think the symbolic aspect has a capability for abstraction we don't see in current neural Nets I think evolutionary algorithms have a capability for creativity we don't see in current neural Nets and hybridizing these together within an appropriate sort of synthetic architecture could be could be interesting but if if you do take that sort of approach it could be that llms are a powerful accelerated right so I mean one thing I'm working on now is using systems like open AI codecs or salesforces Cogen using llms trained on code to make models that translate English Senses Into predicate logic and term logic and other forms of logic so if if you can basically use llms to translate the whole web into structured logical statements feed these in to a logical fear improver and a neural symbolic AI system I mean then then you may have something with the same scope of knowledge as an llm but much less susceptible to hallucination and and because it can do inductive deduct and abductive reasoning and can combine different different pieces of knowledge together right so I mean that's one example of how llms can be used as a tool to accelerate a somewhat different approach to to AGI and of course if my team in Singularity net and the open Cog project is thinking in that direction there's going to be other teens those other teams also doing that in in their own ways not just thinking how do we double triple quadruple down on gbt type models but thinking how do we use the power that's here to help accelerate other other approaches approaches to to AGI so in that in that sense they may be significant on the path to AGI even though what they're doing now isn't exactly AGI in the sense of the ability to generalize you kind of pose this um you know this kind of limitation of llms with regard to creativity uh in a sense to put it you know high level uh you know their inability to create Jazz based on their training data uh have there been attempts to formalize to formalize that notion the the limitations of llms with regard to generalization well the limitations with regard to creativity are particularly hard hard to formalize I mean that of course there are attempts to mathematically quantify what's what's a creative leak but that's I mean that's not that's not that refined an area of of study right and uh limitations of llms that at generalization I'm I'm not aware of any really sort of Knocked Down rigorous study of this there's certainly a lot of papers giving case studies and and examples showing many many cases where llms fail to fail to generalize in in pretty simple ways I mean so there was there was a recent paper they got a lot of attention on social media and mainstream media on like large language models can do theory of mind right with theory of mind is the phenomenon and cognitive science where like I have a sense of what of what you know like so I'm I may I may know what you can see even if even if it's different the the the the the than what I could the what I can see right like I know I know that right now I see while recording this podcast I see your head in a little box on my laptop and I see a robot behind the laptop but I know that you cannot see that robot which is behind my laptop you just see me right um this theory of Mind originates in human babies during during the first year of Life at various points during their maturation so you can give a large language model various puzzles regarding theory of mind like you know two people are in in in in a room together playing with the toy Bob and Jill then Bob walks out the door into the next room and and shuts the door and scratches his head Ken Jill who was in the room with him beforehand you know because you see him scratching his head right and there was a paper published showing that that large language models are are able to solve some problems regarding theory of mind then when you investigate further you see it does really well at solving problems on theory of mind that we're in Psychology papers before because there's there's so many write-ups about about the exact solution to that problem but it can solve some puzzles that don't seem good in any exact psychology paper before but if you deviate too far from the theory of mind puzzles that have been discussed in Psychology literature that it will screw up right so it's it's a bit subtle like it's not just spinning out only the examples that that you fed to it it is abstracting to some degree but it's not abstracting to the level of a two-year-old child you can deal with examples that are are more Divergent from the cases if they've seen in in the past so how the how to rigorously measure these limitations is is interesting and I'm sure a bunch of academics are are working on it from my own standpoint as a researcher I can see it's limited from playing with it I'm more interested in working on on building AGI systems and in rigorously documenting the the shortcomings of systems with qualitatively obvious shortcomings uh that that's totally fair I think where the question comes from is you know you hear uh I hear casual dismissal of llms is interesting they're you know uh blurry jpegs they just are a poor copy of things that already exist but there's you know an obviously creative element to what they're doing they are you know no one put to that the uh you know the the poem that it created about your mom you know flying a dragon didn't exist anywhere on the internet um there's there's a degree of creativity there uh in your example where you are kind of combining uh previous types of music maybe that's an llm that you know yeah to which some Randomness is injected and it's you know it is creating more things so how do we like formalize these ideas to I'm using all them for music generation and it is it is interesting and I haven't thought about formalizing the limitations that much but I can say like you can take I don't have a model as good as music LM yet but I have some models similar to that trained on on smaller amounts of data and I mean you could you can say like you know play me uh passage of a song with a reggae beat but with a rhythm guitar section that sounds a bit like Master of Puppets from from Metallica and the female vocalist and it will do that right and that's really cool on the other hand it sounds like how a bunch of fairly uninspired Studio musicians who are very talented would do that like it's not it's not it never it doesn't sound amazing right it it but it's remarkably competent that putting the pieces together this is my own qualitative evaluation as as a musician though now what one thing that interests me there is could we improve the creativity of output from music LM type models by using like beam search and making up some mathematical measure of what's interesting like you use Mutual information a relative entropy use some information theoretic surprising this measure and then specifically try to milk a continuation out of Music LM which is a valid continuation with reasonably high probability but also has a high level of of surprisingness value right so I would I would like to milk as surprising things I can out of these models because for for the band I play keyboard in I would like to be able to play a riff and then have an AI model respond and play vacuum riff that like incorporates what I played but with an interesting Twist on it right now what it will do it can play back what you played but it will like homogenize it and and make it sound kind of boring and typical for the for the genre which is not it's still an amazing capability right but it's not it's not it's not what I really want as a musician trying to make interesting music and I think I think you'd see the same thing music I happen to have played with a lot but you I mean you see the same thing in every domain like I I talked to a friend of mine is the editor of cointelegraph which is a crypto magazine and then online crypto scene and they evaluated using chat GPT and other systems to write articles and coin Telegraph and they finally decided like the these are just really boring articles I mean it's it's more like if you got middle school students to write to write articles for your for for your magazine I mean it's competent it starts with a summary tells you the thing it it gives you another summary at the end it raises a few key points so I mean at first blush like wow it's amazing that amazing that an AI can do this on on on on the other hand it's also cleared the articles are more boring than the average article in cointelegraph which is is a decent online Journal but is is not like Atlantic Monthly either right like that that's not the absolute the absolute highest end so I would say in that in that case it's not a mathematical measure but in in that case I mean you could you could take the music that comes out of a music LM type model or the the blockchain articles that come out of chat GPT I mean you could submit those articles to be evaluated by human magazine editors or or by the by the folks who audition students for the Berkeley School of Music or something and it's pretty clear what you would find if they would find these things are competent but mediocre right I mean I mean that's I think that experiment is underway with a certain science fiction magazine yeah yeah I mean Cena various other outlets have grown articles created by by chat GPT and it makes sense because like if you're doing a product review I mean that's it's a boring kind of article right I mean no it doesn't have need that much much drama to it but then much simpler language generation algorithms have been used for a long time to like summarize sports sports scores and and weather reports so what we're doing now is a leap beyond that but it's not it's not a leaked to the level of the average journalist let alone to Mark Twain or James Joyce I mean if you want to position it and how to how to measure that in a mathematical way as opposed to a sort of human and qualitative way is it's I mean it's also an interesting question so you deposit that our path to AGI is likely some hybrid of uh neural approaches symbolic approaches I think there's multiple possible paths to AGI I'm really interested in Neuroscience I've spent some of my life doing computational Neuroscience so I mean I mean I think there's a there's a path to AGI where you really do a non-linear Dynamic simulation of of of of of the brain right I mean you've got amazing chaotic Dynamics in the brain you've got EEG waves that are generated by you know glia and neurons and weird Dynamics in the water Mega molecules in the brain like there's amazing stuff happening in the brain I think we we need a revolution in brain Imaging before we can measure enough about the brain to to really do that right but on the other end people aren't trying very hard right like no no one has tried to take say an ishakevich neuron like the best the best non-linear dynamical model of the neuron the best models of glia and astrocytes model all the you know 350 major networks known in the human brain and trying to put together a real large-scale brain stimulation doing something intelligent attached to it to a robot's body right like no that's a doable thing it can be done for no more money than you know Google Microsoft Facebook Ted centers something spend on on AI r d but I mean no one is no one is doing that I'd hoped Henry Markham would do that with the human brain project but that wound up sort of fragmenting and going in different directions so I mean I think I think that's really interesting it's not the approach I'm I'm now taking I I think that uh sort of Deep mathematical Fusion of methods from probabilistic programming neural net learning evolutionary program learning and probabilistic logic I think I think this can probably get us there faster so I mean the Casual way to look at that is you're making a hybrid system where you're taking it deep neural net for large-scale pattern recognition and synthesis you're taking a symbolic logic engine for for inductive abductive deductive reasoning you're taking an evolutionary programming engine to create create new things and you sort of glomming them together I mean then in reality we're not building like a modular system with a neural evolutionary and symbolic box I mean we've actually done a whole bunch of of mathematics to put to put neural probabilistic and evolutionary systems in a in a common mathematical framework using higher order probability distributions and the homotopia type Theory and and uh you know intuitionistic logic and the whole connection so a whole bunch of math Voodoo that we can't go into in the time we have here but I mean we're trying we're really trying to find the mathematical level that shows that neural symbolic and evolutionary are all just different aspects of the same underlying methods for probabilistic search of of program space right and so I mean key there are mathematical results like Curry Howard correspondences that show there's an equivalence between logic systems programs and categorial abstractions so so really when you're doing a logical theorem proof you're also you're also learning a program and you're also learning an ontology that can be represented in category three like these these AI paradigms that historically have seemed very different like symbolic versus neural or learning versus reasoning actually if you dig in the math behind them there's morphisms between them all but all kinds of different ways different ways of looking at at the same thing so that's a yeah on the surface it's a hybridization of methods for different AI paradigms but we're really trying to make a common math framework that leads leads to to to all of them and then trying to figure out how to deploy trying to figure out how to deploy this at Large Scale which is a very interesting software challenge right or even small scale how uh how close have you gotten to uh a concrete demonstration of this kind of approach well so we we had a system called open Cog that we launched in 2008 with some code going back to 2001 and we were doing various experiments on this sort of AI I mean we published a bunch of Journal papers and we we use this on the back end of commercial AI systems in Financial prediction and in functional genomics and and and so forth but that was more these are more narrow AI applications using a framework that was being developed with it with AGI in mind so what what we're doing now we're basically rebuilding the whole infrastructure of opencog systems we have a new a new framework called open open Cog Hyperion and we're we're trying to build a framework where we can do this sort of AI at at the large scale because what we found with the old version of opencog is to really attempt ATI like things with the system we needed quite large knowledge bases and our system was was just too slow so I mean the the underlying architecture we have this large Knowledge Graph or more properly uh a metagraph which is a more General graph like object that than the graph and we have this big knowledge metagraph living in Ram across multiple machines and then distributed across across multiple multiple machines using using uh distributive processing technology and you need to get metagraphs with many billions of nodes in them and then your neural evolutionary and logical AI algorithms they're actually little programs represented as nodes and links in that same metagraph so you have this big like self-modifying self-reprogramming knowledge metagraph we implemented that in opencog and what we found is to try to do really serious things with it it just is like 10 000 times slower than tensorflow or something it's terrible so I mean it's a very high level of abstraction but I was I was reminded how in the 90s you know in the 90s I was teaching neural Nets in the university and you were doing recurrent back propagation you had a neural net with like 40 nodes it would take three hours to train right and so I mean so you could do a few things now now obviously things are scaled up tremendously because we have gpus and all these libraries about bonding out of them right so I think with the new programming language we built in opencog hyper on which is Meta Meta type talk and the new distributed Knowledge Graph that we built are our hope is to be able to scale up all of our all of our earlier experiments and so we've uh it's been a bit painful to take a year out of doing AI r d to build scalable infrastructure but I mean looking at looking at how much good scalable infrastructure did for deep neural Nets I mean it seems like an interesting thing to do and I mean we're we're also working with a company called simulai out of Florida on the dedicated AGI board which has a you know a chip for hyper Vector manipulations a chip I designed for large-scale graphs pattern matching and a deep learning chip and the CPU like wired together very tightly on on the same board and the and if if that goes well then simulized AGI boards can do for this sort of hybrid AI approach basically what gpus did for the Roman approach meaning fairly modest variations and algorithms that have been the literature before suddenly you deployed them at greater scale and they started they started doing doing magical seeming stuff right so if if all goes according to this planned it's probably early next year by the time we start to see really amazing results put out of this I mean it's not like five or five or ten five or ten years or something now that's that's much sooner than where I thought you were were going I thought you were gonna need Quantum Computing to materialize or something like that I would love to see Quantum Computing materialize but I I think that the new chip I mentioned just is three years out just because we're now in the software simulation phase you need to go to FPA fpga we need to go to the hardware but I mean we already have the distributed Knowledge Graph running across many different machines we have a limited version of the of our inter preter working and we're we're working on using uh row calculus which is an advanced form of process calculus so the next step neon Pi calculus to allow it to use multiple processes on this on a single machine because we you know we need to go beyond match reduce like mapreduce is good for something that's matrix multiplication based but we what we're doing isn't just mapping I mean if you're into functional programming you need you need not just fold and unfold you need like chronomorphisms and future morphisms you you need a whole bunch of of ways of holistic processing data structures Beyond mapping so if you look in everything everything that Amazon Google Facebook and so on has done is pretty much based on using mapreduce or it's variations to to take neural net architectures and then run them across massive multi-gpu server Farms right so so what we need is some more sophisticated math to take take algorithms for large-scale graph manipulation and efficiently efficiently run them across multi-gpu server farms and I think that's what we're working on mostly this year I think once that's done we can we can start getting interesting things and I mean that probably there's a few different approaches we're taking in parallel so one approach is just take llms extract all the knowledge from an llm into a distributed logical knowledge base do some symbolic logical entrance on there and see if you then get something llm like but that can compare its other instances now with its other instances in the past and that can reason about whether it's various thoughts and statements are consistent logically coherent with each other right and that's that's not AGI but that that would be you know Elon Musk is thrown around the term truth GPT but all he really wants to do is not make politically correct filters on on chat GPT right if you if you take everything out of an llm put it into a distributed logical knowledge base and do logical inference on that that that's really the truth truth GPT right so I mean I mean that's it that's that's one thing you can do the other interesting thing to do we're looking at Virtual Worlds so we've been playing with Minecraft and we've been looking at various metaverse environments and look looking at using using the opencog hyperun system to control societies of little agents buzzing around buzzing around in in the virtual world and the the experiment I really want to do there is get a community of Agents buzzing around in Minecraft or some other virtual world but we're starting with Minecraft make them do Collective activities together so they have to work together to build stuff to achieve their goals I want to make them invent their own language for communicating with each other like you you might remember there was some total news report maybe three three years ago like Facebook chat box have invented their own language which is great like I had I had software programs invent their own formal language 30 years ago and no one no one wrote me up in the news about it right have a bunch of little agents running around in Minecraft and they're they're making actual noises to communicate with each other and they're collaborating on on you know building structures to to be able to get somewhere they want to to get get rewards and they need to communicate and coordinate to achieve their goals can you get them to sort of invent invent their own language in in that world to to achieve to achieve their goals so I think that's a really interesting experiment I'd rather do it with robots but and I mean we are working with robots with Sophia and Grace in the Hanson robots so robots robots are still a pain it's quite straightforward to do this in the in in a virtual world at the moment so I think the direction of making the real truth GPT is really interesting it's a little more commercial it could set the work could set the world on fire right on the other hand I think making a bunch of little agents in a virtual world that cooperate socially to achieve goals together and invent their own sort of new communication paradigms to do it I think that's more likely to lead to fundamental AGI breakthrough but the beauty of it is you can connect these two things through the same knowledge base right like like a I mean you and I learned our own separate things but we can't wire our brains together we have to communicate by making noises and drawing pictures but I mean if you have if you have a real truth GPT then you have a community of Minecraft agents to learn learn to build stuff and develop their own language I mean you can you can develop machine translation between the Minecraft agents language and English then you can you can you know hybridize these two open Cog hyper on based AGI systems together if you want to right so I mean so those there's a lot of interesting potential directions you can you can you can go here and then the the other ingredient is with with Singularity net we've built a deployment fabric where we can roll out systems like this on the decentralized infrastructure so you could have like 100 server Prime sitting in different countries and pieces of the network underlying this running in all these different places and the and these all these all coordinate together I mean of course without any one Central controller because using using blockchain for decentralized coordination I mean that that that doesn't intrinsically help you make the AGI smarter but it it means that your AI system doesn't necessarily end up monolithic and nothing to run on on the huge server Farm because we're starting with this wildly distributed decentralized infrastructure from the get-go you mentioned Sophia in there uh can you talk a little bit about what that system uh is really intended to demonstrate it it has received a lot of criticism for um I don't know one article says it is is it a show robot um is a parlor trick kind of thing of course it's a sure of course it's a show robot but it's not just a parlor trick I mean I mean so Sophia is a character created by David Hansen and it's a obviously a hardware design and a sort of squishy wear design in the face developed by David Hanson who's a he's an amazing robot assistant and creative artist and the software behind Sophia has actually changed over and over again since 2015 when Sophia was was was first created right so I'm in the I think initially Hanson robotics put a sort of role-based chat bot behind it then I came on board hessnerabox was their Chief scientist for a while we then shifted to using a combination of neural chat box with with open Cog for some some some reasoning behind it so now now behind Sophia Grace Desmond and the other handsome robots I mean you have you have a hybrid you have a hybrid dialogue engine I mean we have we have open Cog in there which is doing some reasoning to generate some some responses you have a rule-based chat system in there the Hanson robotics created the Hanson AI system that responds to some basic knowledge about the robot and who it is I mean you have you have a gpt3 behind there with a cusp we don't use chat gbt but we use a custom set of prompts that were crafted for for gpt3 and when So when you say something to the robot then there's a decision process it checks what the different sub-dialog engines behind the scenes have to say it picks which response makes the most sense and responds with that then there's some subtly behind the scenes because whichever sub dialogue system responded of course that response has to be known to all the other sub dialogue systems so you have multiple dialogue systems with some notion of of shared State behind behind the scenes so I mean it's a obviously a year and a half ago chat GB G3 wasn't an ingredient there and we've also transitioned from the old version of open Cog to using the new version opencug hyper on though so it's really I mean I'm like a human being like we learned but we got the same brain year on year right I mean Sophia has had radical brain upgrades year on year as as the AI field has has developed I mean I think David Hansen is a brilliant sort of theatrical Mastermind right like I mean he worked at he worked I mean he worked into he worked at Disney for a while early in the in his career and he he has been both trying to advance Ai and Robotics and he's been trying to sort of show the world what with a kind loving friendly emotional intelligent robot really look like so certainly some people along the way have assumed that Sophia was much smarter and more human than she really is but I would say that's not a specific problem to Hanson robotics like right now many people think chat GPT is much smarter than it than it really is like that's that's uh that's that's how people roll and right there is the responsibility then to tell people what's actually going on but what I found with Hanson robotics is even when you tell people what's actually going on they don't care and don't believe you they're just like no she really loves me I don't care what you say right I mean and that's I mean that I see that like I have a two-year-old daughter she believes her stuffed animal really loves her right I mean that's a that's what that's what that's what people want that's what people want to do uh for better and For Worse yeah on the topic of responsibility how do you kind of Reason through the the ethics of AGI some critics say it's you know we shouldn't be doing it uh there's uh you know certainly a lot of conversation around AI safety and I mean what what humans should or shouldn't what humans should or shouldn't do is a vexed question I mean it's not it's not clear humans now are happier than we were in the Stone Age so I mean I don't know if I don't know if morally we should have developed Agriculture and civilizations and and machines it depends depends on your risk tolerance it depends on how much you value you know happiness and contentment with your everyday life versus the joy of of do of doing new exciting things right I mean clearly curly Factory workers in the industrial revolution were a lot less happy than Kalahari Bushmen right so I mean I mean that's so I think for better and worse Humanity keeps willing to do high risk novel things which is probably tied in with why we're not Apes anymore right I mean that's sort of what what people do and I mean developing AI is certainly along those lines it's it's high risk High reward it's bringing us into New Frontiers that we don't that we don't fully understand and I think there there is non-trivial existential risk with developing AGI but you don't necessarily see it as overwhelming existential risk I mean some some folks see only a negative AGI outcome that's very irrational I think if we really wanted to be maximal rationalists we have to say we have no idea what the is going to happen right I mean I I think the bottom line is the confidence interval is is really really wide because if once you have something 10 times as smart as a human there's certainly no overwhelming reason to think it's going to be nasty but I mean there's not an overwhelming rational reason to think it's going to be it's going to be nice either like we just we just don't know what's going to happen any more than the mouse could predict you know whether tencent or Badoo is going to become dominant in the Chinese in in the internet Market as a mouse that doesn't understand what's what's going on I mean I think intuitively and sort of spiritually if you would like I I have a strong optimistic sense about the the future of AGR like in in my heart I feel like it's going to be for the good for for for Humanity and I can see that some people have a similar opposite intuitive gut sense like they have a strong gut sensitive it's going to come out come out badly but I don't I don't think that's really a fully rational thing that that that that's more a matter of our our heart feeling about it because when you when you rationally plot it out I mean okay you're gonna build quantum computers you're going to build femto computers we don't have a theory of quantum gravity we don't know all the things that femto computer can do what will the what will the Super AGI discover about that like I mean and we we fundamentally fundamentally don't don't know right I mean my my gut feeling is you know if we raise up a baby AGI with love and compassion and we have it doing education and medicine and Healthcare and Science and a bunch of bunch of good things as that baby AGI goes into adulthood it's going to overall feel well disposed toward human beings and the resources that human beings use are trivial compared to the amount of energy in the solar system let alone let alone the the universe right I mean there I see no reason to assume an early stage superhuman AGI is going to be a psychopathic megalomaniac that needs to use every Quantum of mass energy for its own its own you know self-gratification I mean even even in that description you're kind of anthropomorphizing that the AGI and the relationship between us and the AGI um is is there a way to kind of separate those there is but it's hard to talk about right I mean I think I think I antrimorifies Less in my mind than I do in verbalizations because we just don't have a language for talking about these things I mean yeah anthropomorphize you would say a super AGI could view humans like we view the squirrels in the in the in Yellowstone Park or something right like I mean we we want we want to preserve them yeah I was responding mostly to the the idea of raising the AGI what does that even mean I mean in the current way of doing things I mean I mean it means a lot because we have Sophia Grace and Desdemona robot and we talk to them and we we enter we interact with them right I mean so we we do I do have I mean in my life I have these humanoid robots that I am actually talking to and and they're seeing me and they're learning from the they're learning from the interaction with me and I can see if I put these systems to work selling Amway door-to-door versus if I put them to work in an elder care facility or a kindergarten they're totally learning different things about how they interact with people I mean the one case they're objectifying people like I'm using this person the only thing I care about is to get them to sign them online and get their money on the on the other hand I mean they fundamentally want the elderly person in the Elder Care Facility or they want the kid in in the kindergarten to be to be fulfilled and happy and that they're trying to establish what uh Martin boo bear the philosopher would have called like an eye vowel relationship right and you don't need that as an Amway salesman you get them to sign on the dotted line then you move on to the next house right you you you you do you do need that to be effective in in Elder Care or in early childhood education so I mean I think we and there's a there's a anywhere there's a concrete level here actually yeah yeah one question that comes up for me just hearing you describe this and maybe it kind of goes back a bit to one of my first questions around like sentience versus AGI um which you kind of clearly delineated it's not clear to me that AGI uh necessitates agency are those two more more closely linked that I'm that I'm thinking right now I think they are but it's a subtle point I think for I think for cognitive architectures that even vaguely resemble human cognitive architectures those two are very tightly bound together general intelligence and and agency but I think if you were exploring cognitive architecture space more broadly and looking at like radically non-human Minds it's it's not it's not as as clear like you if you think about like I mean the mathematical fear improver that could prove brother and brother are more and more General theorems than any possible logic system how could you make that really really really generally intelligent without a self model or a model of self another or agency it's it's somewhat of an of an open question whether agency would emerge at some abstract level there but I think in in the human mind or in the human-like cognitive architecture which is I think how we're going to make the AGI breakthrough just because it's what we understand better than other more alien forms of cognitive architecture there I think they're very tightly combined because the formation of Concepts in the human mind is very closely Tied by analogy to the formation of the self-concept and it's I mean it it's by learning how we distinguish ourselves from the environment and our self relates to others I mean that's how we learn to relate a concept to other Concepts and you you can see this in a lot of psychology results like people who are what are called thin boundary meaning they're very influenced by what happens to them they're very emotionally sensitive will tend to develop Concepts that are more flexible and thin boundaries and we'll adapt more more readily to new information whereas people who are very thick boundaries like they they distinguish themselves from their environment very rigidly they're not much influenced by what other people say or do these people tend to develop very thick boundary Concepts and they have a hard time adapting their Concepts to what what happens around them right which can be good or bad depending on what you're doing but you can see from that example like the way we form Concepts in general and the way we form our self-concepts are very closely tied together and this is a big theme in developmental cognitive Neuroscience so I think for for people these are very closely tied together and they are for the open Cog systems that we're exploring now in my own AGI r d but whether they're of necessity tied together I I think if he took any AI system which is controlling an embodied agent in the physical world together with other similar embodied agents in that same physical world right I think which is what humans are doing it's what mobile robots would be doing is what virtual characters would do in a game world I think that setting naturally leads itself to agency being closely linked with Concept formation but of course that's not the only thing that an intelligent mind could could do in the end right it's just what humans and and the robots we build we build are doing and that I mean here as with the question of existential risk you're faced with like whoa we're going into like a a very large Uncharted domain that we that we we barely understand that don't have the theoretical or intuitive tools to to Grapple with and all we can do is take it step by step right so as I'm looking at it now my goal my main goal as a researcher is make an intelligence with roughly human level roughly human-like general intelligence that's about as smart as you and me and then once you get there then that system is going to help you go on go go on to the next level and plus once we get there we're going to learn so much that our current Ai and cognitive theory we'll probably start to seem seem like classical physics relative to quantum gravity or something right so I mean I think there's there's a limited as fun as it is there's a limited extent to which we can we can think uh five steps ahead in this sort of domain yeah related to uh I guess related to to risk you know one of the things that you've spoken out about is um maybe the politics of AGI you know in particular with regard to kind of control of of AGI and the hands of a few large companies or large government agencies yeah this is going to be it's going to be a big deal and I've put a lot of work into building the infrastructure to correctly deal with the Advent of AGI once it happens which is a sort of a sort of ballsy thing to do because I mean it's of course the notion that my team will be the ones to achieve AGI first well I have a fair level of confidence in that not an absolute confidence I I understand the most most people aren't necessarily go going to buy that because they they don't see everything I do in terms of the underpinnings of the systems we're building but if you do believe that like if if you believe we're say three five six years away from making an AGI breakthrough with open Cog hyper on running on Singularity net right then then what happens once you get there right I mean that that's it you could say Okay post Singularity once you have an AGI that's ten times as smart as people okay none of us can figure out what's going to happen but can you think through the sort of end game of the the pre-singularity era right I mean and that then so what happens once you've demonstrated you have a true AGI system that can really think like people and everyone everyone sees that it's not and it's not just a narrow AI right then I think what could happen very quickly is some jack booted thugs come to come to your house and force you to hand over the keys to the thing right I mean I I think I I think this clearly is gonna everyone will see this is the biggest event in the history of the human species and this uh I mean this could lead to all sorts of techno Thriller level scenarios and what I would like to see happen is for the first AGI to be rolled out in the manner comparable to the internet or Linux right like you want to be rolled out across every country at once you want to be rolled out everywhere in the world you want not just the source code to be open you want the knowledge bases that are that are learned to be open you want the code to be running on thousands of different machines everywhere so that there's no one person you can shoot and stop the singularity from happening and and there's no way for any any one government to take the thing over and try to use it to gain the gain a you know a military or financial advantage you really want it to be distributed everywhere and there the infrastructure we built in Singularity net and the new net another Allied sort of blockchain based AI platforms this this and it enable enables that I mean it also it also enables us to have a decentralized infrastructure for for narrow AIS along the way which which can also be interesting but it will it will demonstrate its value to the greatest level if whoever makes the big break through the AGI chooses to roll it out on the decentralized infrastructure rather than on the big tech companies server Prime which is operating in close alliance with the the government of some particular Nation mm-hmm so do you necessarily think that uh that AGI will be the result of uh a big breakthrough I'm trying to think about this question and the degree to which it makes sense I mean if you think about the way AI as a field has evolved like you know there have been kind of these you know there continue to be these Stepping Stones like uh you know but there's also chat GPD which is kind of breakthrough-ish or at least it appeared to us as a breakthrough I don't think that was the Breakthrough I think I think the Breakthrough was burnt made within within Google brain yeah once you have the the paper attention is all you need was a real breakthrough right right and so but on top of that in order to have the thing that broke out uh which kind of defines a breakthrough you you a bunch of steps like rlhf and other things well sure I mean I mean that that's like you know what was the Breakthrough the the atom bomb or the development of nuclear physics before that right I mean so you could pinpoint it however you want but I think if we look at how the AI field is developed in recent years you had this sort of three-year Burris of amazing advances in computer vision starting from maybe Alex net or something in the beginning of of CNN's and it was a few years from that to face recognition becoming a commodity technology right on everyone's phone and in NLP you would say yeah you advert and attention is all you need that was what 2017-18 and this again like four year yeah four years you have you have so it's six years to chat GPT right so five years whatever so I mean I think the reason these things have taken three to five years is there happening at human speed like that's how long it takes people to type in papers and go to conferences and talk and for people to refocus their career from one thing to another right it's not it didn't have to take a number of years but right and I think that's kind of getting at where my question was coming from uh and that is you know will the will the Breakthrough be known as the Breakthrough when it actually occurs or will it be kind of this continuous kind of building on I mean look look like a breakthrough it may not have looked like a breakthrough to the average person it certainly looked like a breakthrough to to everyone in in in in the AI field though right so I mean I think it I think at very least it would be like that so I mean if you for sake of discussion assume open assume my own project gets there I mean suppose suppose that you have a system that's chat GPT like but can actually can actually reason and and understands you know the relation between what what it says and empirical reality as in terms of what it sees through a camera or sees in the database and suppose suppose that you do have a bunch of little guys running around in mind scrap Minecraft and they really are communicating back and forth in their own language and they look like a bunch of primitive tribes people right I mean so I would say when we get there that will look like a breakthrough to 99 of people in in in in the AI field and they'll be like whoa like there's a there's a huge leap here right now in indeed the average person may not be able to tell whether that's an epocal breakthrough or merely a breakthrough on the level of chat GPT or alphago or something right so I'm in on the other hand what would be the lag between that and and like an avatar that appeared on everyone's phone that clearly had superhuman level general intelligence I mean it it might be a nine month or a year lag by my best guess I don't think it's a it's a 10-year leg certainly if this thing can help you develop the next thing then things start to accelerate yeah and there's so once you get to a human level AGS system like that yeah I mean you already have systems that can do simple Python Programming right so you can it's not hard to see if you had something with a measure of general intelligence it's going to be able to learn to update its own source code I mean that now that's still that's still a slow sort of experiment though right because even if even if it could upload its own source code it's still going to run experiments you can upload it can modify its code but then it has to deploy that across a lot of machines just to run that experiment it has to see if it if it if it worked or not right so I mean there's still a certain pace of work so I mean if it if it redesigns our open Cog pattern matching chip I mean that that that's great it's still going to go through fpga and and and testing and I mean it's still gonna do a limited production run and then do a large production run so I mean I mean there's you're still not having like five seconds from the initial breakthrough to the to the singularity but it on the other hand you're certainly looking at a lag of months to years not not not decades right which now what's happening during those two years if you take that as a as a causing random period of time what's happening in the world during those two years is going to be very weird and and messy right because it I mean presumably we already had chat gbt level systems obsoleting people's jobs so what what you're going to see is Universal basic income get rolled out throughout the developed world as people realize quickly that agis are taking most jobs you're not going to see anyone give Ubi in the Central African Republic because that's not what people what people like to do in those countries don't have their own budget for it so you're you can just see like a incredible exacerbation globally of the diverts between haves and have-nots right with the the developed world is all leaning back playing VR video games living on Ubi where while agis and robots do 95 of the work while the developing world is 95 subsistence farming with no more jobs being Outsourcing them so what what level of terrorist activity you start to see in that setting I'll I'll leave it to you to figure out what what what level of attempts to use proto-agi for defense or offense on the part of various countries I mean you can see you can see the potential for a very weird few years while this transition is is happening right and I mean I wish I wish I thought that was going to be a beautiful smooth transition but like right now we have major countries going around randomly and uselessly blowing up other other people in the world right I mean I mean it's we we can't we can't even deal with ourselves without without a transition to human level AGI so at what point do you get to the optimism that you described earlier after the AGI is a couple times human level intelligence because then then the world is not being being run by us like hypertrophied monkeys anymore right we we we we we have help by uh by systems that are smarter than us and hopefully more compassionate and well balanced as well uh I mean we're still the the monkeys and so therefore the system is controlling us by force presumably I don't think it would have to no I mean I think once you've abolished scarcity we're in a different domain right but she says scarcity is not abolished for everyone right not initially but imagine the super AGI created the direct solurian molecular assembler and just literally airdrop them everywhere in the world that then solar powered right got it so free energy free food resources don't matter asymptotically approaching free or whatever yeah I mean I mean right right now right now we could air drop smartphones to everyone in Africa if if we felt like it right I mean a billion smartphones we could manufacture we're not quite doing that but smartphone penetration is larger and larger right so I mean if if you really did develop a molecular assembly you could Mass produce them in a huge Factory you could airdrop them from drone drones everywhere people could feed in matter it's solar powered they can they can 3D print whatever physical object they want right I mean nothing in the laws of physics that that prevents that from from happening we don't know how to build it now could an artificial engineer would twice the intelligence of a smart human do that most probably we've gone we've gone a bit a bit uh afield from machine learning per se no all of this raises really interesting and important questions um so uh I appreciate you indulging them no I mean I I love I love I love to think about this stuff I mean I've most of my day I spend working on nitty-gritty computer science stuff rather than rather than thinking about the the broader future but I'm thinking about the broader future my whole life right and I mean what's what's cool is even the rational part of my brain feels we could be only years off from from seeing these science fictional things come come to pass right where it's a my confidence interval width for the Advent of AGI was much larger 20 years ago the the than it is now awesome well Ben thanks so much for taking the time to chat and share a bit about what you're up to and thinking about sure no it's great great stuff to talk about thanks for your own time too thank you\n"