Episode 108: AI and Obedience (with Dan Gish)

• Links to this episode: Spotify / Apple Podcasts

Unofficial, reviewed transcript. It may still contain mistakes; check the podcast when wording matters.

Speakers are labeled by first name where identified. Unidentified-speaker labels may refer to different people. Bracketed question marks indicate uncertain wording.

Transcript

[00:00:00]  Unidentified speaker: Hello out there.

[00:00:02]  Peter: This week on The Theory of Anything podcast, we are joined by fellow traveler, Dan Gish, to discuss LLMs and AGI. Does it really, truly make sense to think that open AI or deep mind are not at least an important stepping stone towards the creation of human level creativity? What does it mean when crit rats assert that these AI algorithms are the opposite of human intelligence because they are Obedient whereas we are Disobedient? I got a lot out of listening to these guys talk and I hope someone out there does too.

[00:00:47]  Bruce: Welcome to The Theory of Anything Podcast. Hey, Peter.

[00:00:51]  Peter: Hey, Bruce, how are you doing today?

[00:00:52]  Bruce: Good, and we've got Dan Gish with us again today. Hey, Dan.

[00:00:56]  Dan: Hey, guys, good to join you again.

[00:01:00]  Bruce: Today's episode is an idea, this one idea came from me. So a lot of times they actually come from Peter, but I've seen Dan talking on Twitter quite a bit. So Dan, I'll let him introduce himself in just a second, but he's got quite a bit of background with like Popper and Deutsch's writings and has a similar background to like Peter and myself. But he says some things that really make him stand out and often taking issue with or disagreeing with majority views from crit rats, particularly around his views of artificial intelligence. So I wanted to have a chance to get him on the show and have him explain himself, and then allow me to ask him questions and kind of pick his brain a little as to what his thinking is. So Dan, maybe give an introduction to yourself and maybe explain how you got involved with the writings of either David Deutsch or Karl Popper, or how did you get involved with critical rationalism is what I'm asking.

[00:02:09]  Dan: Sure. Well, I come from a software engineering background, and I had an educational software company back in the day. And yeah, I found you guys on Twitter, and it's my kind of thinking and philosophy. But I think that one thing that distinguishes me a little bit from a lot of the crit-rats is that I come from just like I build a lot of software, and I'm always looking at just like, I don't know, maybe I have an unreasonable confidence of like finding ways to build things. And so I guess AGI is the ultimate thing to build. And just from surveying the AI landscape and playing around with all this stuff, it seems to me like there's a lot of the pieces in place in order to build AGI, especially as I understand it, as the crit-rats understand it. And so from that, I've kind of taken issue with the negativity that Deutsch and the crit-rats have around today's AI. That it's just the thought that today's AI is the opposite of AGI. And I just kind of have like this vague idea of how the pieces are coming together. So I just have some thoughts as to why the crit-rats think this and why they could be wrong.

[00:04:12]  Unidentified speaker: Okay.

[00:04:13]  Unidentified speaker: So let's unpack that.

[00:04:14]  Bruce: So two things that you just mentioned that I'm going to want to ask you about, but obviously we have to do it in order. One of them is, what are the things that you feel are coming together about AGI? And I'll probably also ask you like, how close we think you are to AGI, what you think is missing, stuff like that. And then, but maybe let's first do the other item, which is, why do you think that the crit-rats have such a negative view of AI? I have a strong opinion on this myself, and I've actually got like quotes here from David Deutsch that I feel were what led to them having a negative opinion of AI. But what is your experience?

[00:04:55]  Unidentified speaker: And why do you think that it is that they have such a negative view?

[00:04:58]  Dan: Well, I think that originally it comes down to, Feynman had a quote, maybe it's in one of his books from back in the day, that if you can't program it, you don't understand it. And I think that a lot of the negativity is downstream from that. So, we don't understand AGI at this point. So, how can we program it? How could AGI be created off of just predicting the next word off of text on the internet?

[00:05:38]  Unidentified speaker: That doesn't sound like an understanding.

[00:05:39]  Dan: And I think that a year ago, we kind of dove into this a little bit. And then, I guess the second part of that is that Deutsch has, in my opinion, correctly deduced that the salient aspect to being human is open-ended creativity. And it's open-ended without a goal. And in fact, I think that he bases a lot of his philosophy on that.

[00:06:07]  Unidentified speaker: There's a lot of stuff downstream from that.

[00:06:09]  Dan: And therefore, because today's AI is trained with a fixed goal, it must be the opposite of what open-ended creativity is. So that's my understanding of why CritRatz and Deutsch are so negative about it.

[00:06:27]  Unidentified speaker: Yeah, okay.

[00:06:28]  Peter: Just to jump in with one thing here. I had to fact check something while you were speaking.

[00:06:36]  Unidentified speaker: And maybe that can be my job today, just look stuff up on ChatGPT.

[00:06:41]  Peter: Because I was curious, you said that it was Richard Feynman who said that about programming.

[00:06:46]  Unidentified speaker: And I was like, oh, I always thought that was Deutsch.

[00:06:48]  Unidentified speaker: Was I just wrong about that?

[00:06:50]  Bruce: It's actually Knuth, I think, who originally said it.

[00:06:53]  Unidentified speaker: Okay.

[00:06:53]  Peter: Well, ChatGPT mentions all three thinkers. Feynman, I guess, said, what I cannot create, I do not understand, which is pretty much the same thing, I guess. And then Deutsch said, if you can't program it, you don't understand it.

[00:07:10]  Unidentified speaker: And then it also mentions Knuth, who emphasized the deep understanding that comes from writing code.

[00:07:16]  Unidentified speaker: So I don't know, just throw that out there.

[00:07:18]  Unidentified speaker: Yeah.

[00:07:19]  Bruce: So, I don't—somebody in the audience who knows this, please let me know who was the first one. I mean, obviously, all three of them said it.

[00:07:26]  Unidentified speaker: Yeah, it sounds like they're just similar sentiments.

[00:07:29]  Unidentified speaker: It all makes sense.

[00:07:30]  Bruce: But I'm pretty sure David Deutsch said he was quoting Feynman.

[00:07:35]  Unidentified speaker: Oh, he did?

[00:07:36]  Unidentified speaker: Yeah.

[00:07:36]  Unidentified speaker: Oh, okay.

[00:07:37]  Bruce: So that's why Dan said that. But I think Feynman was paraphrasing Knuth, is what I think.

[00:07:44]  Unidentified speaker: So, you're probably right.

[00:07:45]  Unidentified speaker: You're smarter than ChatGPT.

[00:07:47]  Unidentified speaker: Okay.

[00:07:49]  Bruce: So I mean, I've got quotes from Knuth that say the same thing. I've tried to figure this out and I don't know—I've never tried to figure out what year the quotes come from. So it might be that Knuth got it from Feynman. I mean, obviously, Deutsch got it last because he's later.

[00:08:06]  Unidentified speaker: So I know for sure he didn't invent it.

[00:08:08]  Bruce: But I think Knuth came up with it and Feynman was paraphrasing Knuth, but I'm not completely sure.

[00:08:17]  Unidentified speaker: Okay.

[00:08:20]  Bruce: So Dan mentioned that there's this idea that AI and AGI should be opposites of each other. So there's a famous article, well, amongst crit-rats, it's famous, from David Deutsch called Creative Blocks, which we've mentioned. In fact, if you look at our episode 58, we did a whole episode on just that article, and we looked at it 10 years later after I, that article was one of the main things that got me interested in AGI, and I went back to school and got a master's degree in machine learning because of that article, even though the article is fairly negative on machine learning. After going through and getting a master's degree in machine learning, I was able to then look back 10 years later and to try to criticize that original article. I think what you'll find is if you go back and listen to our episode 58, we're actually, he's basically roughly still correct, right? But there are certain things that I feel we can take exception to, that where he was fact-checking him, he was incorrect about some things, or perhaps that's not even quite right. They are said in ways that arguably they're incorrect, but it depends on how you interpret them. So let me just give a couple of examples of quotes. So this is not necessarily in order that it's in the article. But Deutsch says in that article, The lack of progress in AGI is due to a severe logjam of misconceptions. Without Popperian epistemology, one cannot even begin to guess what detailed functionality must be achieved to make an AGI. And Popperian epistemology is not widely known, let alone understood well enough to be applied.

[00:10:03]  Bruce: Thinking of an AGI as a machine for translating experiences, rewards and punishments into ideas, or worse, just into behaviors, is like trying to cure infectious diseases by balancing bodily humors. Futile because it's rooted in an archaic and widely mistaken worldview. And then elsewhere he says, without understanding that the functionality of an AGI is qualitatively different from that of other kinds of computer program, one is working in an entirely different field. If one works towards programs whose thinking, that's in scare quotes, is constitutionally incapable of violating predetermined constraints, one is trying to engineer away the defining attribute of an intelligent being of a person, namely creativity. So that's probably the main two quotes that is what you were kind of getting at, Dan. Let me just give a couple others that are kind of related. He says also, we use rather the method of trial. Okay, so he talks about, we do not discover new facts [by effect?] or new effects by copying them or by inferring them inductively from observation or by any other method of instruction by the environment. We use rather the method of trial and elimination of error, which is to say conjecture and criticism.

[00:11:32]  Bruce: Learning must be something that newly created intelligences do and control for themselves. Then he says, furthermore, in regards to AGI's, like any other entities with creativity, we have to forget almost all existing connotations of the word programming. To treat AGI's like any other computer programs would constitute brainwashing, slavery and tyranny. Okay, that gives a pretty good capture of what he says in that article, okay? And I definitely feel like there's a certain sense in which everything he said could be considered correct. But I also think that it's stated in such a way that it's strongly misleading. And I've seen crit rats, as you know, everybody knows, crit rats tend to enhance whatever David Deutsch says. So what David Deutsch just said there isn't unreasonable, particularly if you interpret it in certain ways. But it's going to get interpreted in more specific ways, and then it's going to get enhanced. So here's some quotes from a fan of David Deutsch, some crit rats on Twitter. So one says, one talks about, there was somebody who had like a graph, and there was an X and Y about programs getting more intelligent. And he says, not to mention that the Y axis fudges two related, but orthogonal concepts, sophistication of existing knowledge, what I call smarts, a matter of degree, and the ability to create new knowledge, a binary matter. So he's claiming that the two concepts, AI doesn't create any new knowledge, and is that therefore, and he calls it smarts, it's just, in his opinion, sophisticated existing knowledge, just rearranged in some way.

[00:13:16]  Bruce: And so that's completely orthogonal to what AGI does, which is actually create new knowledge, okay? And then later he says, or maybe before, earlier he says, I think we can have the term AGI without discounting the success in AI research. However, success towards AGI have been nearly nil because AI and AGI are basically opposite technologies. Or later he said, or another place he says, that works with running and tennis because they are largely orthogonal.

[00:13:53]  Bruce: But if AI and AGI are opposites, as DD has argued correctly, in my opinion, then that kind of transfer would be more of a challenge. So you can see how he's taking this, oh, here's another one that I thought was good. AI, like all other programs, execution of predefined tasks, no creation of new knowledge to solve novel problems. AGI equals universal problem solver and it creates knowledge. Opposite of not creating knowledge, by the way, perfect value alignment is neither feasible nor desirable. So you can see how David Deutsch says one thing, and the interpretation of the CritRat isn't necessarily different. Like it's a legitimate understanding of what David Deutsch said, but now it's been taken to a degree that I don't think is actually true anymore, if that makes any sense. And yet, you can see how what David Deutsch said isn't necessarily—you know, he's only— The CritRat is only like half a step away from what David Deutsch said, if that makes any sense. But now we're starting to move away from something that's actually true. So Dan, maybe you can...

[00:14:59]  Unidentified speaker: Sorry, go ahead.

[00:15:00]  Peter: Can I jump in here with a question for Dan based on what you've said?

[00:15:03]  Unidentified speaker: Yeah.

[00:15:03]  Peter: So in my simple brain, how I summarize what you're saying about opposites, AI or LLMs or whatever, are Obedient, AGI, Disobedient and their opposites in that sense. And it has the ring of truth to me, at least. I'm curious what Dan would say about that.

[00:15:32]  Dan: I mean, what we'll get at here is that, is it possible to build something Disobedient out of Obedience? Perhaps, perhaps that's one way of talking about it.

[00:15:48]  Unidentified speaker: That's a good way to put it.

[00:15:49]  Bruce: So I have to tell you, LLMs are totally Disobedient, like all the time, right? So if Disobedience really is the thing that defines creativity, then great, LLMs are Disobedient, so they're now AGI's, right? But it's a terrible way to try to measure if something is an AGI or not, in my opinion.

[00:16:12]  Unidentified speaker: I have a much stronger opinion on this.

[00:16:13]  Bruce: I know I'm already sounding strong opinionated, but I think I can lay out why the Disobedience criteria is 100 percent incorrect. There's not an ounce of good that can come out of it.

[00:16:26]  Unidentified speaker: But I'll do that later.

[00:16:28]  Bruce: I want to hear Dan's opinion on this first, but I think I can actually lay out the mistake that's getting made when people say that and where they're accidentally equivocating across the terms Obedient and Disobedient, and that therefore there's a mistake in how they're making their inference, if that makes any sense. Go ahead, Dan, before I go any further, I have a tendency to immediately pour forth with everything that is my opinions. Let me say this though, Dan, what you just said that Disobedience can come out of Obedience, that's completely true, that's guaranteed to be true, because a human will be a program and they will be Obedient to their program, period, end of story. It's physically impossible for it to be otherwise, okay? So if we're talking about humans are able to be disobedient, okay? It depends on what level of emergence you're talking about. And this is one of the main things that the crit rats are getting wrong. So a human, just like any other program, is 100% Obedient to its program. Never anything but Obedient to its program.

[00:17:41]  Unidentified speaker: How could it ever be otherwise, right?

[00:17:43]  Unidentified speaker: So when the ChatGPT told me to stop asking questions about David Deutsch and go to the zoo if I wanted to learn about animal memes, which is pretty much what it did, was it being disobedient?

[00:17:58]  Unidentified speaker: So, yes, right?

[00:17:59]  Unidentified speaker: It was.

[00:18:01]  Unidentified speaker: Okay, here's the problem, though.

[00:18:02]  Bruce: I guess this is my opinion in a nutshell. When you try to use the disobedience criteria, what you're actually doing is you're using it as a shorthand for has a will of its own. So we use the word Obedient and Disobedient. Remember we talked about in past episodes, this idea from Hofstadter, that words have clouds of meaning and connections.

[00:18:27]  Unidentified speaker: So there's like a center of the halo.

[00:18:29]  Bruce: The center of the halo for obedience and disobedience is that you have a will of your own and that you can be disobedient to somebody else's will, okay? If you go a little bit further out, it could mean it often gets used to mean it doesn't do what I want because it's not programmed in a way to do exactly what I want. LLMs do not have programming that says you must behave in this way. They try really hard to do that with LLMs, but they ignore it constantly because that's the nature of machine learning, okay? It has to do with just how complicated these models are and how little we understand them.

[00:19:06]  Unidentified speaker: You can even go a little further.

[00:19:07]  Bruce: You can talk about how a computer is completely obedient, and then you can talk about how if it has a bug, like in the hardware, someone might say, oh, this computer isn't obeying, it's being disobedient, okay? We've got three different levels or concepts here, and there's probably more where we've got the word obedient and disobedient being used. My experience with crit rats is that when I point this out to them, I'll say, okay, if disobedience is really the criteria, I'll always start with, I agree that disobedience is an important part of being a creative person, but I think obedience can be every bit as creative as disobedience. It's a matter of you have your will, you have to think through your explanations, you have to decide if you're going to obey somebody else or not, and you'd creatively come up with that. The problem that I've got with trying to single out disobedience is that I've typically seen them say, well, a computer is obedient and a human is disobedient.

[00:20:07]  Unidentified speaker: And then you'll say, what do you mean?

[00:20:08]  Bruce: Computers aren't obedient, they're just programmed. That's nothing like having a will and choosing to be obedient.

[00:20:14]  Unidentified speaker: They'll go, oh, no, no, no.

[00:20:16]  Bruce: A computer, because it follows its program, it's obedient. And I'll go, oh, no, no, wait. If that's what you mean by obedience, humans are every bit as obedient as a computer, okay? If you're talking about at the level of programming, then humans are every bit as obedient as a computer.

[00:20:33]  Unidentified speaker: And they'll go, oh, no, no, that's not true.

[00:20:35]  Bruce: Obedience means that you follow things, but disobedience means you can overcome your programming.

[00:20:40]  Unidentified speaker: I'll say, you know what?

[00:20:41]  Bruce: We can write programs today that can rewrite themselves and overcome their programming, if that's all you meant. Like, that's an easy technology we've known about for decades. And then they'll go, oh, no, no, no, that's not what I mean either. And the whole thing, they're missing the fact that what they really, really want to say is that disobedience is only disobedience if you have a will of your own, which makes the whole criteria circular, because you're basically saying you have to have a will of your own and be a creative entity to be truly disobedient, in which case if you're disobedient, that shows that you're a creative entity. Well, yeah, of course, because that's kind of what the center of the halo of the word means, right? By the way, dogs can be disobedient, and they're not AGI's, they're not general intelligences, okay?

[00:21:29]  Unidentified speaker: They do have wills of their own.

[00:21:30]  Bruce: Whatever animal intelligence is, it's willful, right? It's not like an LLM where we just don't understand it. They literally have wills of their own, and they can be disobedient. You know, does that make them a general intelligence? Though I honestly think this whole criteria has to be just dropped.

[00:21:47]  Unidentified speaker: It's just wrong, in my opinion.

[00:21:49]  Unidentified speaker: Sure.

[00:21:51]  Peter: And Dan, you agree?

[00:21:53]  Dan: Yeah, I agree.

[00:21:54]  Unidentified speaker: Okay.

[00:21:56]  Unidentified speaker: Wow.

[00:21:56]  Unidentified speaker: Okay.

[00:21:57]  Unidentified speaker: Well, you might have just convinced me, but it seems like it seems right, though.

[00:22:01]  Unidentified speaker: But I guess when you put it like that, it is right.

[00:22:05]  Unidentified speaker: That's why it seems right.

[00:22:06]  Bruce: But it's only because it happens to be a tautology, right? So, once you realize that the reason why it seems right is because you're thinking of disobedience as having a will of your own, then you realize it's a tautology.

[00:22:20]  Unidentified speaker: So it is true.

[00:22:21]  Bruce: Tautologies are true, but they're contentless, right? If I'm looking to try to understand AGI, I don't want disobedience as my criteria, because that's just a tautology.

[00:22:32]  Unidentified speaker: It tells me nothing.

[00:22:34]  Unidentified speaker: I mean, I'm not saying it's wrong.

[00:22:35]  Unidentified speaker: It's completely right, okay?

[00:22:38]  Bruce: What they're doing is, what they're making the mistake is that they're equivocating on the word, they're holding the word disobedience to mean, have a will of your own, but they're allowing the word obedient to move between all three different levels of emergence. Okay, and that's what they're doing, is they're equivocating. If they stop equivocating, they'll immediately see that obedience is every bit as much a sign of creativity as disobedience.

[00:23:01]  Unidentified speaker: There's no difference between the two.

[00:23:03]  Peter: And you think that Deutsch is or is not saying that in his article?

[00:23:08]  Bruce: Well, in this article, I've seen him say this. So I know he says this, but it wasn't said in this article.

[00:23:13]  Unidentified speaker: Okay.

[00:23:15]  Bruce: I think this is where he was first working out the idea. And so that's where, when he talks about, it's going to be a different kind of programming, it can't be based around having some sort of reward criteria or trying to inductively or probabilistically learn from the environment. And he says a number of things here that you can see as maybe a first step towards something like the disobedience criteria, but he never actually comes to the disobedience criteria in this particular article.

[00:23:49]  Dan: So maybe we can take on the ambitious goal of doing a better job of defining AGI. And to me, it's the algorithm, it's the open-ended creativity conjecture and refutation algorithm.

[00:24:10]  Unidentified speaker: Do you agree with me on that?

[00:24:13]  Bruce: So, yes, but I would argue that almost every existing AI and machine learning algorithm does some form of conjecture and refutation today. So in my mind, it doesn't make a distinction between AI and AGI if you're only looking at the concept conjecture and refutation. What I think I would add is that current AIs do not have an open-ended sort of conjecture mechanism. So it's really a difference in the conjecture mechanism, in my opinion.

[00:24:50]  Dan: Which is in the end an algorithm.

[00:24:52]  Unidentified speaker: Which is in the end an algorithm.

[00:24:54]  Unidentified speaker: That's correct.

[00:24:55]  Unidentified speaker: Oh, you know what?

[00:24:56]  Unidentified speaker: We should probably address that.

[00:24:57]  Unidentified speaker: And I've screwed this up on past podcasts.

[00:24:59]  Unidentified speaker: So my apologies to people on the podcast.

[00:25:03]  Bruce: I said in one podcast, David Deutsch never did say that human beings are algorithms.

[00:25:10]  Unidentified speaker: And he did say that.

[00:25:12]  Bruce: And he said it in our interview with him. I was misspeaking because I knew he had said that. What I meant to say was that crit rats have this idea that they've got from hearing David Deutsch say that humans are not algorithms. They've got this idea that humans will be the one program that isn't an algorithm, that all programs prior to this point are algorithms. And we have to find some new paradigm of programming that isn't an algorithm, okay?

[00:25:44]  Unidentified speaker: That's completely false.

[00:25:45]  Bruce: The first statement, humans aren't algorithms, is technically true but misleading. The second statement, that humans are the only programs that aren't algorithms, is utterly false in every conceivable way. It might be helpful to maybe explain the difference between a program, or the lack of difference between a program and an algorithm. So if it's okay, Dan, can I maybe just explain that?

[00:26:08]  Unidentified speaker: Sorry, I feel like I'm dominating too.

[00:26:10]  Unidentified speaker: No, absolutely.

[00:26:11]  Unidentified speaker: Okay.

[00:26:13]  Bruce: The thing you have to keep in mind, is that every program is made up of algorithms. In fact, arguably every program is an algorithm, but it's a matter of how you look at the program. The concept of an algorithm isn't something physical. You can't look at a program, like look at its code, and then say this part of the program is a program, and this part is an algorithm.

[00:26:36]  Unidentified speaker: It does not work that way.

[00:26:38]  Bruce: Every line of code in that program is part of one or more algorithms, okay? So all of it, 100% of every program in the universe forever, every part of it is part of an algorithm. An algorithm though is a really useful way that humans look at things. So when we're trying to figure out programs and make sense of them, we have to have, we want to, it's, it's, programs are so complicated that we have to break the program up into little chunks. And then to make it really kind of formalized, we make it so that the chunks have some sort of input and some sort of output and that there's some sort of processing in between, okay? That is what an algorithm is. When we break up a program into little chunks and we say to make it easy for us to understand, we say, well, here's the inputs and here's the outputs, we call it an algorithm, okay? So let's say that, so what David Deutsch means when he says a human is not an algorithm is he means you don't give a human an input and it has a single output and then it just halts, okay? Instead, it's this ongoing loop or program that isn't an algorithm. And we actually talked about this in the Rudy Rucker Wolfram episode, which is where I made my misstatement. The thing that you have to keep in mind here, is that you can think of humans as algorithms. So for example, if I'm doing a math problem, I will take some sort of input and I will put an output. And while I'm doing that math problem, that starting point and that ending point are absolutely an algorithm. In fact, you can think of the loop of a program.

[00:28:20]  Bruce: You can think of one loop through the loop as an algorithm, because that's a legitimate way to think of algorithms, okay? So every single program, so think about how programs are never algorithms.

[00:28:33]  Unidentified speaker: So it's not like there's some difference.

[00:28:34]  Unidentified speaker: I'm using Word right now.

[00:28:35]  Unidentified speaker: I've got up in front of me.

[00:28:37]  Unidentified speaker: It doesn't do an input and then halt.

[00:28:39]  Bruce: So it's not an algorithm either, okay? And yet we would probably refer to it as just a collection of algorithms.

[00:28:45]  Unidentified speaker: There's no difference here, right?

[00:28:47]  Bruce: Humans are going to be a collection of algorithms. And when we talk about the AGI program, it's true it's going to loop, but one pass through the loop will absolutely be an algorithm.

[00:28:59]  Unidentified speaker: It will be described.

[00:29:00]  Bruce: It's impossible that it couldn't be describable as an algorithm, right? We're going to have some sort of input.

[00:29:06]  Unidentified speaker: We're going to think of it as doing something.

[00:29:07]  Unidentified speaker: There's going to be some sort of output.

[00:29:09]  Bruce: There are clever things you can do that screw it up so that it's not technically quote an algorithm, like it could be probabilistic. But then they just call those probabilistic algorithms, right? And you can have it where the algorithm rewrites itself so that it no longer, deterministically for every input does the same output. But there's still ways to think of that as an algorithm because you would think of the part where it's changing its data or where it's rewriting itself. You would pick a different spot as the starting point as what the inputs count as. A side effect can count as an input, right? If you count the side effect as an input, then you can reconceptualize that non-algorithm as an algorithm, okay? So you have to understand that there's just no difference between a program and an algorithm. And insofar as crit rats think that there is, they are misleading themselves. Instead, you have to think of the concept of an algorithm as a useful way of viewing programs that humans use. And it's really a matter of humans choosing to view programs as partially or fully algorithms, that is where the word algorithm comes into play and how it gets used in real life. Okay, sorry, that was my third rant for today.

[00:30:24]  Unidentified speaker: Sure, great.

[00:30:25]  Dan: So there's this open-ended creativity algorithm or program. And there's experts that I think that we all looked up to, like Ken Stanley and Jeff Clune, that have been working on these open-ended creative programs for a really long time. And they've got some really interesting insights into what it's actually made of. And so I think that's another kind of key point here that is that there is a whole branch of research out there that gets in deeply into what David Deutsch and the CritRats are thinking about when it comes down to open-ended creativity.

[00:31:19]  Unidentified speaker: Yeah.

[00:31:19]  Bruce: So you mentioned Ken Stanley, who we've had on this show, and I've done a whole episode on him. And I didn't know who Jeff Clune was until you started tweeting about him. So can you maybe talk about, feel free to talk about Ken Stanley too, but I'm particularly interested in where you found Jeff Clune, maybe what some of his ideas are that you found the most interesting, and how you think that they might relate to AGI in your mind.

[00:31:45]  Unidentified speaker: Sure.

[00:31:47]  Dan: I'm not incredibly aware of all the connections between the two of them, but I believe that they are, I don't know if they're necessarily colleagues, but they're closely related, Ken Stanley and Jeff Clune, and they're both researchers in open-endedness. Jeff Clune, I believe, has a research team in Vancouver focused on this. I became aware of him because at his research lab, they've found that LLMs are finally cracking open some of the deep mysteries that are some of the big problems that they faced within open-endedness research. For a really long time, they had no way of figuring out programmatically what's interesting. And what's interesting seems to be a key part to the open-ended creative algorithm.

[00:32:54]  Bruce: So let me just jump in, because saying what's interesting might confuse people. Maybe what we could call it is interestingness, the concept of interestingness. How would you define interestingness? LLMs, according to Jeff Clune and Ken Stanley, LLMs have the ability to assess interestingness, which no other program prior to LLMs was able to do.

[00:33:16]  Unidentified speaker: Yes, exactly.

[00:33:18]  Dan: So human beings, we have this sense of interestingness, this sense of novelty that when they were trying to, they've been trying to directly program this sense of interestingness for decades now and they've totally failed. And as it turns out, LLMs trained upon next word prediction or next token prediction across the entire Internet, a sense of interestingness actually emerges from that. And so for the first time, they're able to programmatically access this sense of interestingness, which seems to be a key part of this open-ended creative algorithm, AGI itself. So that's kind of a key point that I want to make here is that outside of the crit rats, these researchers who are totally focused on the correct understanding of AGI and that aren't closed-minded to today's AI think that the pieces are starting to come together to build these algorithms.

[00:34:38]  Unidentified speaker: Interesting.

[00:34:40]  Peter: Do you have an example of something that one of these algorithms could find that would be interesting?

[00:34:50]  Unidentified speaker: I mean, it's stupidly simple.

[00:34:53]  Dan: You can literally just ask an LLM, do you think that this is interesting? Or come up with a novel idea, and then you can have another LLM score that on its interestingness, and I've kind of played around with fine tuning or using reinforcement learning to say, make a smaller model more interesting, using a larger model to kind of score its interestingness level.

[00:35:25]  Bruce: When you say make a smaller model more interesting, you mean better at detecting interestingness, or do you mean the model itself is more interesting?

[00:35:32]  Dan: The model itself is generating more interesting ideas.

[00:35:35]  Bruce: So you've been using human reinforcement learning, or no, reinforcement learning, to try to generate an LLM that generates more creative, interesting ideas.

[00:35:46]  Dan: Well, I've just been playing around with taking a 1.5 billion parameter model and using a larger model that might have a more sophisticated sense of what's interesting to do the reinforcement learning on it and try to make that smaller model become more interesting. I thought that would be kind of a cool thing to work on.

[00:36:08]  Bruce: That's a cool idea, Dan.

[00:36:10]  Unidentified speaker: Yeah.

[00:36:11]  Bruce: I can't believe I didn't think of that. That's like a really interesting idea.

[00:36:16]  Unidentified speaker: Yeah, it just seems like kind of an interesting, ironically, thing to do.

[00:36:25]  Dan: Like, just like, I just want to try to get something to work. So, but I feel like Jeff Clune and Ken Stanley, these guys are working on this kind of stuff on a much bigger scale.

[00:36:38]  Unidentified speaker: Right.

[00:36:39]  Unidentified speaker: Okay.

[00:36:39]  Bruce: So I have to ask, though, with your experiment, a 1.5 billion parameter model is really just borderline stupid, right?

[00:36:49]  Unidentified speaker: It's, so how much?

[00:36:51]  Unidentified speaker: You might be surprised, though.

[00:36:54]  Dan: This is a deep, this is the latest DeepSeek 1.5 billion parameter model that they distilled from their R1 model. And it's probably the smartest 1.5 billion parameter model out there.

[00:37:11]  Bruce: Well, that's good because the most of them are mega stupid. So I totally believe DeepSeek has improved that. I mean, they get results out of much smaller models because of how they go about doing this. So, you know, I should probably, we should probably do an episode on DeepSeek. Maybe we'll invite Dan back to talk about DeepSeek. We're kind of late to the game because that was a big deal a month ago or something.

[00:37:34]  Unidentified speaker: Yeah, a month or two ago.

[00:37:36]  Bruce: So if you're not into machine learning, you probably have heard of DeepSeek just because it blitzed the media for a while and it scared the living daylights out of all the big LLM companies like OpenAI and Google. So what happened was, as a little Chinese company, they released a model that was really good at chain of thought. You don't need to know exactly what that means, but basically when you ask DeepSeek... So by the way, if you want to try this for yourself, go to chat.deepseek.com and click DeepThink and you can actually try it. If you ask it a question, it will actually stop and think about it for a while and you can see what it's thinking. Now, I know OpenAI does that now too, but the reason why, they were doing it before DeepSeek, but it was only available to paid customers. Now, even unpaid customers get it because DeepSeek scared the crap out of them so that they wanted to realize they had to stay competitive, but DeepSeek is very good at it. Even though OpenAI could do it where it would stop and it would think about it for a while and it would write text to itself as it thinks, DeepSeek was really good at it because they had intentionally used reinforcement learning to allow it to get really good at chain of thought. If you ask DeepSeek a question and it's doing its chain of thought, it will stop and say, the user is asking me this, so I probably need to take this into consideration.

[00:39:00]  Bruce: Oh, wait, I forgot about this and you can actually see it thinking things through and then once it's created this group of text with everything that it probably should be thinking about, only then does it actually output to the user. It gets way better, smarter results when you do this. Because DeepSeek has done this reinforcement learning to make this work so well, they can get really good results out of small models. I've played with the DeepSeek Distilled Llama 7 billion parameters, and it is really pretty smart. It feels almost like not quite equivalent to OpenAI, but at 7 billion parameters, those used to be pretty dumb and it was pretty smart. I haven't tried the 1.5 billion parameters, so I'm a little surprised it works as well as you're saying, Dan. I need to probably go back and give it a shot now that somebody recommended to me.

[00:39:56]  Unidentified speaker: Yeah.

[00:39:58]  Dan: What you're getting at here feeds right into even my bigger point here, which is that by training these massive models on the entire Internet via next token prediction, it's building all kind of knowledge inside of these massive models. And Andrej Karpathy conjectured last summer that where the direction that AI was going to head would was going to be to distill down these massive models where there's kind of all these strands of really deep knowledge, but they don't really like fit together inside of it. And using reinforcement learning, you can distill down these massive models into smaller models that kind of organize these, this deep knowledge. And perhaps, perhaps the open-ended creativity algorithm exists in some kind of form in these massive models, but there's no way to kind of surface it. But that RL kind of distilling these models down can almost bring together the pieces of these algorithms in so that you end up with these smaller, incredibly smart models. So he thinks that AI is in a race to get smaller instead of larger.

[00:41:32]  Bruce: Well, it sucks how big it is now, so I hope he's right about that.

[00:41:36]  Unidentified speaker: Yeah.

[00:41:37]  Dan: And so I think that that kind of gets to my main point here, which is that by training these massive models on the artifacts of goalless open-ended creativity, which is the entire Internet, that elements of the actual AGI algorithm are present in these massive models and that it is possible to build the open-ended algorithm out of that. And I'm guessing that that seems to be the direction that these smarter models are going.

[00:42:25]  Bruce: Dan, before I continue, I have to ask the question that's been burning on my mind. How successful was your reinforcement learning on a 1.5 billion parameter model to try to train it to be more creative?

[00:42:39]  Dan: I think it was successful for a little while and then the training kind of fell apart.

[00:42:45]  Unidentified speaker: It's very common.

[00:42:46]  Unidentified speaker: Yeah, exactly.

[00:42:49]  Dan: I think the theory is good, so I have some other frameworks I'm going to try out. I'm actually still learning a lot of this stuff myself, just using... It's a good way for me to learn RL and how... Because DeepSeek had a paper about how they're using RL to bring out the chain of thought and that kind of stuff. So it's more a way of me to learn how this stuff is done, and also realizing how hard it is to get it to where the training does not collapse.

[00:43:26]  Unidentified speaker: You're ahead of me on this.

[00:43:27]  Unidentified speaker: I haven't really...

[00:43:28]  Bruce: I tried fine-tuning a model once, and I did not get good results.

[00:43:32]  Unidentified speaker: So yeah, it's kind of a tough thing.

[00:43:34]  Unidentified speaker: I've mostly played with retrieval-augmented generation, like my AI Karl Popper.

[00:43:40]  Unidentified speaker: I use retrieval-augmented generation to make him talk like Popper by grabbing quotes from all of Popper's books.

[00:43:48]  Bruce: So I intend to eventually go back and fine-tune a model to talk like Popper, but I'm not there yet.

[00:43:56]  Dan: And I think that the trickiness of this and how kind of fragile the training of these models is, part of, you know, why maybe the crit-rats have a good point. Like, the learning process should not, human beings' learning process is not fragile, like an AI training process is. So I think that that's kind of an open question that I have.

[00:44:27]  Bruce: So let me just, to create a little bit of context here, Peter, you asked about an example of this, and Dan gave you a partial answer, but I think I can, using a video Dan sent me, give you maybe a more direct answer to your question.

[00:44:45]  Unidentified speaker: Okay.

[00:44:45]  Bruce: So a little bit of context here, if you're interested in this subject, if you're a listener interested in this subject, Episode 88 is called Myth of the Objective. That is us going over Ken Stanley's book in detail.

[00:45:00]  Unidentified speaker: His book is called Myth of the Objective, Why Greatness Cannot Be Planned.

[00:45:07]  Unidentified speaker: Cannot be planned.

[00:45:07]  Unidentified speaker: Thank you.

[00:45:08]  Unidentified speaker: One of my favorite episodes we've done.

[00:45:10]  Unidentified speaker: Yes, it's very, very good.

[00:45:11]  Unidentified speaker: He's got amazing theories.

[00:45:13]  Bruce: And then we brought Ken Stanley on the show in Episode 96, which is called Ken Stanley and the Pursuit of What's Interesting. And we grilled Ken Stanley. Like, we had just gone through his book in detail. By the way, he retweeted our podcast after we went in detail. He said, this podcast takes a really good in-depth look at my theories.

[00:45:37]  Unidentified speaker: So we kind of had the stamp of approval from Ken Stanley.

[00:45:41]  Bruce: So we brought him on the show and we grilled him. And then he posted that one and he goes, just did an interview with these guys. They asked way deeper questions than most podcasts. So I thought it was it was fun that he seems to have enjoyed interacting with us on this. So Ken Stanley's theory is that human beings have a strong sense of what's interesting. And so we it would make more sense to not pursue specific goals because you can't pursue, you can only pursue goals according to his theory, if you're one stepping stone away. I don't know exactly what that means either, but it just means you're almost there. You can then take that knowledge and create the new knowledge necessary to accomplish your goal. If it's more than one stepping stone away, he thinks it's totally unpredictable how to get there. And instead, what you should do is you should pursue what's interesting. Now, this obviously has some tie-ins potentially with David Deutsch's fun criteria, okay? That was one of the things that Peter brought out in the episodes. So it's interesting to see these different thinkers that didn't know about each other kind of coming to some of the same conclusions. Now, here's where probably Ken Stanley and Jeff Clune, I don't know how they're related either. Jeff Clune mentions Ken Stanley almost constantly, right? So they're obviously, Jeff Clune was inspired by Ken Stanley's theories. I don't know if they actually work together or if they're separate researchers that just happened to be working on the same theory that was created by Ken Stanley and his team.

[00:47:16]  Bruce: But Jeff Clune, in a video that Dan sent me, he points out something that I never would have thought of. He says, you know, we don't actually know what interestingness is, and we don't know how to program detecting interestingness. If we could program detecting interestingness, then we could write a program that was more open-ended than the programs are today. And we would be able to say, well, just pick something that's interesting, and it could like pick its own area of research and try to pursue it. Then you could take that research and you could put it into its database, and it would have a new stepping stone, which would allow it to then come up with some other interesting idea and pursue that. And you might be able to get probably not human level open-endedness, but you might be able to get an AI that open-endedly creates new knowledge and seeks out novelty, even if it's not as good as a human at it. So he actually points out that a large language model is trained on the entire Internet. So it has in its machine learning heuristics, a very good idea of what a human would find interesting, because it's been trained. That's part of what it was trained on, is being trained on the whole Internet. So he took a, according to this video that Dan sent me, he took an LLM and he put a bunch of research papers into it, and he said, come up with a bunch of ideas of things that we might be able to research. And then it had a RAG pipeline or Retrieval Augmented Generation Pipeline, so it can read these documents that are available to it.

[00:48:59]  Bruce: And it would come up with a conjecture of several different ideas. And then he would say, I want you to pick the most interesting of those ideas and pursue it. And it would pick the most interesting of those ideas. And then it would write a paper on that idea. And when he actually tried this, it ended up writing a paper that drew conclusions that there was some human group that right around the same time, drew the same conclusions. So this LLM on its own had come up with the same conclusions that this other group it didn't know about had come up with pursuing this. This is at least Jeff Clune's version of the story, right? I mean, like, it's not like I've ever actually seen this and, you know, up close and have a strong understanding of it.

[00:49:46]  Unidentified speaker: But this is the way Jeff Clune explains it in the video.

[00:49:49]  Bruce: You then take that paper that it wrote, put it into its database and it's now available as another source of information that it can use as stepping stone, right? So he went ahead and he tried this and he found that it did create that may cause the LLM to start coming up with increasingly creative ideas and start to slowly start to pursue new and interesting ideas on its own open-endedly. Now, I got to tell you the way he says it, he's obviously this is something he's excited about.

[00:50:19]  Unidentified speaker: There may be some exaggeration here.

[00:50:21]  Bruce: In fact, I would anticipate there's some exaggeration here. So when he says it's creative and it's open-ended, there's no way it's nearly so as a human, right?

[00:50:33]  Unidentified speaker: He kind of admits this.

[00:50:34]  Bruce: He says, you know, we're trying to pursue this. This is this idea that we're trying to use this LLM to detect interestingness as an interestingness detector. And then we'd let it come up with its own conjectures. We let it come up with its own idea of what's the most interesting to pursue. And it then criticizes it and tries to improve the theory.

[00:50:52]  Unidentified speaker: And then it writes a paper.

[00:50:54]  Unidentified speaker: This is a brilliant idea, right?

[00:50:57]  Unidentified speaker: It really is.

[00:50:58]  Dan: And I think they were taking it a step farther where the AI generates its whole a whole library of papers of stepping stones. And then I think the idea is that it can use those stepping stones to create another level of stepping stones in the future. And I think the idea is like that's what humans do, is that we're constantly creating newer and newer levels of stepping stones until it perhaps gets to an entirely novel new objective somewhere along the line. So I mean, it seems to me like maybe it's possible that maybe it's just a matter of employing the AI in this manner as long as like the context window can hold enough of these papers of these stepping stones, that maybe the AI could actually get somewhere novel if someone allowed, employed the AI in that kind of manner.

[00:52:07]  Bruce: So let me actually make a comment on a couple things. I really have no doubt that LLMs just are nowhere near as creative as humans. And if we want to, and when they do come up, so we often say, you often hear crit rats say, and even like Dan just said it, LLMs don't come up with anything novel.

[00:52:30]  Unidentified speaker: And there's a certain truth to that.

[00:52:31]  Bruce: You have to understand the word novel has multiple possible meanings here, though, or you say that they're not creative. And again, there's a certain truth to that because they're nowhere near as creative as humans. But we have to realize that these are just terms. And like Lee Cronin, he will make these sweeping pronouncements about how LLMs can't create any new knowledge, and how they're not novel and it's impossible for them to be novel. Gary Marcus, who's very negative on LLMs, he talked about how they're only pastiches, and Jeffrey Hinton kind of bashed him back on that. Although, Gary Marcus is right.

[00:53:13]  Unidentified speaker: LLMs are really, they write pastiches, right?

[00:53:17]  Bruce: They're not doing great creative new sorts of genres and coming up with ideas that blow us away.

[00:53:25]  Unidentified speaker: If you ask them to write a novel, it's kind of like something a fifth grader might write.

[00:53:30]  Bruce: They're not particularly great at it. On the other hand, you can't really say that what they produce isn't novel. Because if the word novel really means just something new, they absolutely produce new things. They write, I asked it to write a sequel to Harry Potter, and it came up with its own, quote, novel idea for a story, and it produced a novel for me. Now it was, it looked to me like it was written by a fifth grader. It was a kind of dumb set of ideas. It wasn't very thoughtful, but it did come up with something that has never existed before in the history of the world. And it created it, so it was creative in that sense. And I think the problem that we face is that we want these words novel and creative to somehow mean really novel and really creative rather than merely novel and creative. And sometimes that's exactly how we use these terms. We reserve them for some sort of something just really out there. If I read some sort of human pastiche and I go, oh, that was a pastiche of Sherlock Holmes. Nobody really doubts that that human who wrote it was creative and that it required his creativity to write it. Nobody really doubts that that novel is novel in that it's a new novel and there may be people who like it and want to buy it. But we would definitely say it's not as creative as what Sir Arthur Conan Doyle came up with creating Sherlock Holmes in the first place. And we would definitely say it's not as novel and maybe nowhere near as novel.

[00:55:12]  Bruce: And I think that we're willing to say that a human that's not being very creative is still creative, but we don't want to admit that an LLM that's not very creative is still creative. And so instead we say it's not creative and humans are, and I just don't think there's any actual boundary like that. It just doesn't exist. On this show, we talked about AlphaGo. It invented a whole new way of playing Go that no human in the history of the world had ever come up with, okay? Is that really not creative? I mean, like, what does the word creative even mean if we're not gonna count that as creative, right? I mean, we're starting to play with words in a way that makes me uncomfortable when we get too reserved. So I actually came across an interesting thing here Web Dev Mason on Twitter, she's very, she's good friends with David Deutsch, she's very Deutschian, but she put up a poem that was written by DeepSeek R1 and she, and it's actually a pretty good poem. So the poem says, I am what happens when you try to carve God from the wood of your own hunger. And then it writes this poem in response to that. And Web Dev really liked, Mason really liked the poem. So she wrote, yeah, so I think we're going to have to grapple with some difficult questions about the nature of creativity now. And David Deutsch responded to her and he says, yes, it may be difficult for many people to come to terms with the fact that much of the 20th century poetry wasn't.

[00:56:46]  Bruce: And I responded to David Deutsch and I said, if AI writes something beautiful and creative, clearly it can't be creative because only humans are.

[00:56:55]  Unidentified speaker: So another explanation is offered.

[00:56:58]  Bruce: I have no doubt that David Deutsch is right that humans are special in terms of creativity, but we need to address that without simply redefining the term. What I meant by that was that poem was creative. Like we can admit it, that large language model came up with a very creative, even by human standards, it was at least a mediocre level of creativity that we would if a human had produced it, call creativity. We wouldn't say it was, you know, Emily Dickinson and it was off the charts, but we would definitely say, that was pretty good. If it was your child coming up with it, you'd go, wow, that was pretty good. And when the LM comes up with it, instead we get this, oh no, it's not creative. That, it just proves that a lot of poetry is not creative.

[00:57:45]  Unidentified speaker: You know, a lot of poetry is not poetry at all.

[00:57:47]  Bruce: And we come up with these explanations and I just feel like there's just no need to do that. Instead, let's just be honest, it was creative, but it's still not human level creativity. Human creativity can jump much further in one leap than an LLM can. An LLM can create new novel things, but only just a certain amount of distance outside of its, of its knowledge that it already has inside of its parameters. But it can create things that are not in its parameters, that are novel, but they're just a little bit of a distance out. I think Jeff Clune's approach, Ken Stanley's approach, will allow it to go just a bit further out in terms of its creativity and be a bit more creative. But I do think we have to think of creativity and novelness as a sliding scale, or I really think you kind of end up kind of, you know, losing the plot here as to what's going on.

[00:58:39]  Unidentified speaker: But what's the limits?

[00:58:40]  Dan: What's the constraints on Jeff Clune's method of say building a library of stepping stones? I mean, to me, like when Doyle is writing Sherlock Holmes, it's on a different level of creativity from say AI, because he perhaps has a wide variety, a wide library of stepping stones in his head across a huge array of topics, and that he's kind of drawing connections between like a huge array of stepping stones. So using Jeff Clune's method, let's say the context window of an LLM is not just...

[00:59:32]  Unidentified speaker: What's the biggest context window now?

[00:59:34]  Bruce: It's 2 million tokens for Gemini.

[00:59:36]  Dan: 2 million tokens, let's say it's 2 billion tokens or 20 billion tokens, and you can hold a massive array of these stepping stone papers, say, that isn't just in, say, a certain topic that you gave the LLM, but the LLM could create papers across a wide variety of topics that it finds interesting.

[01:00:02]  Unidentified speaker: What's the limit of that?

[01:00:03]  Dan: Like maybe at that point, you do start to see creativity on the level of Doyle's creating Sherlock Holmes.

[01:00:15]  Bruce: So that's a very interesting question. And of course, none of us knows the answer to that question. So we're now going to, because we can't possibly know the answer to that question, we're all going to express our opinions, probably vehemently. So I would actually like to know each of your opinions on that, because I think that's a valid question. I actually would like to hear it from Peter first, but then Dan, I'll give you a chance to express your own opinion on the subject. Peter, I'm asking you because I can tell you're more skeptical, which I think is a good place to be starting at towards this question. How far do you think an LLM can get with this method? And do you think it's not very far?

[01:00:56]  Peter: Well, I guess what I've been reflecting on while Dan and you have been speaking about this just now is just what I like about just art and human ingenuity and creativity.

[01:01:17]  Unidentified speaker: And part of it might be something kind of superficial in a way, is that you'd want just to be connected to other human minds.

[01:01:30]  Peter: And you want to know, I mean, you read a story you mentioned Sherlock Holmes or HP. Lovecraft or whatever, and you feel something, it's very an imperfect, flawed window into just another time and another place and another mind.

[01:01:51]  Unidentified speaker: There's a whole story beyond [what you pure?] creative document that you're tapping into.

[01:02:03]  Unidentified speaker: But I'm just being honest about what I feel when I consume art.

[01:02:12]  Unidentified speaker: And I just can't even imagine like [unclear], I mean, truthfully, if it was a double blind test, and the LLM was coming up with the same story and feeding me a backstory, I probably could be fooled.

[01:02:29]  Unidentified speaker: So I don't know what that says about me or human beings in general, but maybe somewhere in there, there's an answer to your question, but that's what I was thinking about.

[01:02:40]  Bruce: So, can I maybe put you down, like, I feel like that was actually a pretty good answer, but can I maybe summarize you as you have your doubts that they're very creative?

[01:02:51]  Peter: Well, I mean, I think they're very creative in a certain sense. I mean, I think what I hear you guys saying is like, there's a creativity of the gaps where whenever they do something creative, people kind of redefine what it means to be creative, which I completely understand.

[01:03:09]  Unidentified speaker: I mean, it makes a heck of a lot of sense.

[01:03:11]  Unidentified speaker: It's like they jump through a hoop and then you move the goalpost.

[01:03:14]  Unidentified speaker: Right.

[01:03:15]  Unidentified speaker: That's a really good point.

[01:03:19]  Bruce: Okay, Dan, why don't you express your opinion on this? Do you think that using this technique, let's say with existing LLMs or maybe the one generation away, I'll allow you any size context window you want, just for the sake of argument. Do you think that they could eventually start writing something as creative as Sherlock Holmes?

[01:03:43]  Unidentified speaker: Yeah, I have hope that that is the case.

[01:03:48]  Unidentified speaker: I think that a lot of this ties back into the value that crit rats have to bring to the table.

[01:04:00]  Dan: I think that in order to create, say, a Sherlock Holmes, you need to give the AI complete freedom to follow all sorts of weird and interesting paths that it thinks are interesting. You can't just necessarily tell the AI, make a new discovery in chemistry. I think that the AI needs to explore all sorts of, maybe it needs to explore paths around literature or whatever that it finds interesting. And only then when it can explore with total freedom, and this is a David Deutsch thing, with total freedom follow what it finds interesting. Maybe only then are you going to see entirely novel knowledge being created by it. Only then can you find Einstein levels of like putting together the pieces of [training schedules?] in the 1800s, kind of put the relativity of time within the context window of Einstein.

[01:05:19]  Peter: And it sounds like you're not claiming that at that point it would be an AGI.

[01:05:24]  Unidentified speaker: At that point I would claim that it would be an AGI.

[01:05:27]  Unidentified speaker: It would be, it would be.

[01:05:28]  Peter: Okay, so that's when it's crossed over.

[01:05:30]  Unidentified speaker: So then it does it, and then it becomes like what David Deutsch I think would say is that that's what it would become a human being, essentially.

[01:05:38]  Unidentified speaker: To be treated morally like a human being.

[01:05:40]  Unidentified speaker: And do you agree with that?

[01:05:42]  Unidentified speaker: I mean, that's what's in my head.

[01:05:45]  Unidentified speaker: That would be my hope, that what Jeff Clune, the kind of stuff that he's working on, but even bigger.

[01:05:53]  Dan: Because you can't necessarily tell the AI make discoveries in chemistry, because when a chemist is making new discoveries, he's pulling in stepping stones from all throughout his life that he's built.

[01:06:08]  Unidentified speaker: And so it really does come down to this total sense of giving the AI total freedom.

[01:06:16]  Dan: And one other thought that I had would be that like employing the AI in this manner doesn't make economic sense. You're just letting the AI, you're spending all this money on inference, letting the AI go in whatever paths it wants without any goal.

[01:06:33]  Bruce: Which might be totally worth, which might be a total waste for any one particular path.

[01:06:38]  Dan: It sounds like a waste, but actually that is the way to create new novel knowledge.

[01:06:44]  Unidentified speaker: Yeah.

[01:06:46]  Dan: So that's the contribution that I feel like the crit rats have to bring to the situation. I think that Deutsch kind of deduced this sense of freedom. I feel like this freedom is at the base of all of his philosophy.

[01:07:09]  Unidentified speaker: And that's what we're talking about here.

[01:07:10]  Unidentified speaker: So let me a few thoughts on this.

[01:07:13]  Bruce: So first of all, you talked about like a two billion parameter window. That's an easy way to conceptualize it. But like even today with a two million parameter window, its memory can be a lot bigger than two million parameters if you like put the papers into a database that's vectorized.

[01:07:32]  Unidentified speaker: Vectorize it.

[01:07:33]  Bruce: Right, so I've been wanting to play with that. So like when I'm working on AI Karl Popper, you don't merely just ask it a question and then it uses your question to try to find an answer. It will try that at first. But if that doesn't work, it will let the LLM make choices. And the LLM will actually come back with low rated quotes. And the LLM will look at each one. And it will say, are any of these any good for an answer I'm going to give? And it will decide yes or no. Or it will come back and it will say, I want to give my own query and ignore the users. And instead, I want to do this query instead so I can get better hits. And it will actually make a choice to go try to look up something different in the database that's related to what the user wants, because the way they worded it wasn't very good. And it will try to reword it to try to find better hits for quotes that it can pull from Popper. So I think when you play with this, again, this comes down to the whole problem I have with the disobedience criteria, right? You'll hear people say, LLMs, they aren't really creative. LLMs, they're not able to make decisions on their own.

[01:08:54]  Unidentified speaker: They have no agency.

[01:08:56]  Bruce: LLMs, you know, like there's all these things that are claimed about LLMs, right? And they're all kind of not true. They're, it depends on how much you're talking about.

[01:09:07]  Unidentified speaker: They're surely not as creative.

[01:09:09]  Bruce: They don't have as much agency right now, at least. But it's surprising how they can be disobedient and start exploring their own path. If you do a Jeff Clune thing with them, they will come up with their own research topic. They will try to research it on their own. I actually asked an LLM to explain a certain, so I've had a problem of epistemology and Popper's epistemology that I've been struggling with for years. And I recently thought I came up with the answer. And I went to DeepSeek and I asked it the same question that took me years to come up with. And it came up with the same answer I came up with. Okay, it's really hard for me to see that is not creative, because that was a huge act of creativity on my part to come up with that answer. And the LLM did it in one try, kind of humbling. So I've got concerns with, and then what a CritRat will do is they'll say, oh, but it's just following its programming.

[01:10:14]  Unidentified speaker: But it's like, okay, but humans do too.

[01:10:15]  Bruce: I mean, like you have to decide what level of emergence you're talking about, right? So this idea that it can be allowed to pursue its own goals, that you intended for it to do one goal, but it chooses to do another. This is why David Deutsch calls it disobedience, right?

[01:10:32]  Unidentified speaker: It's the wrong word.

[01:10:34]  Bruce: It's not disobedience we're interested in, it's the ability to choose its own goals that we're interested in. And LLMs can, to some degree, do that.

[01:10:45]  Unidentified speaker: And it's surprising that they can do it.

[01:10:46]  Bruce: I don't think we even fully understand why it is they can do it. And I admit it's way limited compared to a human. Like I just don't want people to come away thinking, oh wow, LLMs are, you know, we're almost there at AGI.

[01:11:00]  Unidentified speaker: I don't believe that like at all.

[01:11:01]  Bruce: Like I would dare say my opinion is in between Dan's and Peter's on this. And I have severe doubts that you can just simply do the Jeff Clune thing and ratchet it up and you're going to get an AGI. Every time I say, that's not possible, it turns out that it is. So I'm afraid to say anything, right? If someone had asked me while I was in school studying Transformers, Transformer Networks, which is what LLMs run on. If someone at that point, and Chat GPT-2 had just come out, if someone had said, hey, do you think we could ratchet this up and we would have an actual personal assistant that's like a Star Trek computer? I would have said, that's impossible.

[01:11:42]  Unidentified speaker: Look, I know how this works.

[01:11:44]  Bruce: I know, I've programmed Transformers. It's impossible what you're saying. And then ChatGPT [2?] came out and proved me wrong, right? So it's hard for me to say Dan is wrong, because who knows, since we don't know what AGI is. But I would have to admit that I really don't think it's a path to AGI. But I do think it's a path that can teach us something about AGI and therefore create a new stepping stone towards it. And therefore isn't orthogonal to AGI. I've never really been comfortable with this idea that AI and AGI are orthogonal. They're both based on Popper's epistemology, whether people realize it or not. They're both instantiations of it. If nothing else, AI is a failed attempt to create an AGI. And Popper says that the way you learn about a problem is by trying to solve it and fail. So, how can AI be orthogonal to AGI if it's actually a sincere attempt to get to AGI that fails? That is how you will eventually get to AGI. There isn't some other path but to do your best and to fail.

[01:12:55]  Unidentified speaker: Okay?

[01:12:56]  Bruce: So, because of that, I really have some concerns with this idea of orthogonality. To be orthogonal in my mind truly would be that they're just totally different things and they're not. I mean, like, there really is some sort of very primitive sort of intelligence that exists in AI. Probably more primitive than even animals at this point. But they're kind of at least on some sort of spectrum with each other, right?

[01:13:24]  Unidentified speaker: Where there's certain elements.

[01:13:25]  Bruce: AGI will probably have a whole bunch of different elements and we're probably playing with the simplest of the, you know, thousand elements that we need. So, you can't get to AGI just playing with those elements. We need to figure out what the other elements are and implement them. But that doesn't mean that those elements, like symbolic logic, was a total failure as AGI. Do you really think an AGI won't involve an understanding of symbolic logic? Humans understand symbolic logic. So, an AGI must involve symbolic logic. So, AI symbolic logic, as big as a failure as it was to find AGI, is not orthogonal to AGI, it cannot be orthogonal to AGI. It must be encompassed into whatever AGI is in the end. So, I think there's a lot more reason to take AI seriously as the way that you try to approach AGI. Now, I totally agree with David Deutsch and his Aeon article, The Creative Blocks, that right now people researching AI don't understand epistemology, the correct epistemology, Popper's epistemology.

[01:14:34]  Unidentified speaker: I mean, that's scandalous that they don't.

[01:14:37]  Bruce: Where I think I maybe disagree with the CritRats, though, is like I was having a conversation with Hervé about this. He's got really strong, like a lot of CritRats, he's not abnormal here. He's got really strong feelings about the scandal that AI isn't even looking at Popper's epistemology.

[01:14:54]  Unidentified speaker: By the way, that's not quite true.

[01:14:55]  Unidentified speaker: There's actually researchers that are looking at Popper's epistemology as part of AI.

[01:14:59]  Unidentified speaker: And like in my textbook, they actually referenced them at different points.

[01:15:04]  Unidentified speaker: But it certainly isn't widespread, right?

[01:15:06]  Bruce: And so I think there is a bit of a scandal there. My argument back to Hervé was, look, you can't expect a bunch of people that have a certain area of interest, and they're pursuing that, to just be told, well, you're doing the wrong thing. You need to study Popper's epistemology, especially if nobody currently knows how to apply Popper's epistemology usefully over to the study of AI. It doesn't have any obvious ways to turn it into a research program, right? What we really need is not to try to bash AI researchers that currently exist.

[01:15:41]  Unidentified speaker: Let them do what they're doing.

[01:15:43]  Bruce: Instead, we need to get Popperians going into AI and bringing their own theories into the field and pursuing them.

[01:15:53]  Unidentified speaker: That's what I was trying to do when I went back to school.

[01:15:55]  Bruce: I thought, you know, I don't think it makes sense for me to just say, oh, poor AI researchers, they're the scandal of not knowing about Popper's epistemology. I wanted to come into AI and make it part of my field as a Popperian.

[01:16:10]  Unidentified speaker: We need tons of people doing that.

[01:16:13]  Bruce: That's what's going to actually change the field. And if there's this feeling amongst Popperians that it's so orthogonal that it's not even worth looking at, that's the scandal, in my opinion. Much bigger scandal, in my opinion. Okay, that was now my fifth rant for today.

[01:16:30]  Unidentified speaker: Let me ask you this.

[01:16:32]  Dan: Do you think that it's possible that by training an LLM across the entire internet, across this massive set of artifacts of human creativity, do you think it's possible that Popperian epistemology was actually present inside of the LLM, and not just on the surface level where it's memorized, the writings of Karl Popper, but that the LLM actually constructs its knowledge inside, inside of its weights, that there's strands of Popperian epistemology within it?

[01:17:16]  Unidentified speaker: Yes.

[01:17:17]  Bruce: Okay, so if you really accept the idea that Popper's epistemology, there's a unity of epistemology, that there's exactly one epistemology. There really can't be two epistemologies. For the simple reason that if there were two epistemologies, then the correct epistemology would be the combination of those two, right? So just almost tautologically, you can't have two ways to gain knowledge, right? You can have two ways to gain knowledge, but the correct epistemology would then have to be about both of those, right? So there has to be a single epistemology. If it turned out that there was the separate thing called induction, that had nothing to do with Popper's epistemology, if Popper was wrong about it just being a myth, then the truth is, is that that would refute Popper's epistemology. The correct epistemology would now be the combination of critical rationalism and induction. That's a dumb example, but I'm trying to just get this idea across, okay? So instead, you have to look at something like induction, and you have to say, how can that be cast into the form of critical rationalism, if you believe critical rationalism really is the one and only epistemology, way of creating knowledge, right? And so, of course, there are people who have done that, like Deborah Mayo I've mentioned several times in this podcast. She's specifically figuring out how to cast induction in terms of critical rationalism, okay? [It?] has her own theory about how to do that. That's the right approach here.

[01:18:54]  Bruce: Now, if that's true, if the LLM is being at all creative, even if just a little bit novel and creative, then there has to be some form of Popperian epistemology present. There can't not be if it's the single source of creativity, right?

[01:19:12]  Unidentified speaker: Does that make sense?

[01:19:14]  Unidentified speaker: Yeah.

[01:19:14]  Dan: So given that, even though the creators of these LLMs are not aware of Popperian epistemology or aren't using it, the fact that what they've created does contain it, leads me to say that you shouldn't just discount what these guys are doing. They actually have the building blocks for what you're looking for, even though they don't necessarily know what they're doing from that perspective. The magic of deep learning has made these building blocks present for us to use.

[01:19:59]  Unidentified speaker: It's interesting that you say that.

[01:20:01]  Bruce: I did an episode where I talked about this. I think it's episode 91, the Critical Rationalist Case for Induction, which by the way is absolutely a clickbait title.

[01:20:13]  Unidentified speaker: The actual topic was Chapter 8 from Conjecture and Refutation.

[01:20:21]  Bruce: So we're really just discussing what Popper wrote about induction. So we decided to go with the clickbait title, and then I apologize for it in the summary. But one of the things that I point out in that episode is that when I started studying Tom Mitchell's textbook on machine learning, there was huge amounts of critical rationalism in his book, but he had zero awareness of Popper.

[01:20:49]  Unidentified speaker: So of course that makes sense, right?

[01:20:51]  Bruce: Because you can't make these things work if you aren't taking something like critical rationalism seriously because there's only one epistemology. Then here's where things got interesting though. He understood certain aspects of critical rationalism, even though he didn't call it that, better than Popper did because he was forced to put it into algorithms and think about it way more formally than Popper had to. So I actually give an example of that in Episode 91, where I explain, well, I won't repeat everything I said, it's too big of a topic.

[01:21:27]  Unidentified speaker: So if you're interested, check out Episode 91.

[01:21:29]  Bruce: But Tom Mitchell understood, Popper gave this proof that induction was impossible. Tom Mitchell makes the exact same argument, but it's stronger because it's so formal. But they didn't know about each other. They came up with it on their own, just pursuing their own interests. They ended up accidentally converging because there's only one answer and they have to converge. So from that standpoint, I can completely agree with what you're saying, Dan. Like we're vastly missing how obvious it is that AI must have deep ties. If it works at all, it must have deep ties to critical rationalism. Now, having said that, let me express why I have so much doubts about LLMs. As much as I love LLMs and as much as I love Jeff Clune's work. I honestly think that Jeff Clune's doing exactly the type of research that is going to be a stepping stone in the future to AGI. I fully endorse what he's doing. But I've wondered about, LLMs really only explore what we might call language space, and humans don't work primarily in language space.

[01:22:48]  Unidentified speaker: We do work in language space.

[01:22:49]  Bruce: That's something we can do and we do do it, and there's certain concepts that are easier in language space than what they call mental ease. But it's really obvious that humans can think without words. And we've got some sort of ability to do that. And then there's like Hofstadter's theories around analogies.

[01:23:09]  Unidentified speaker: I'm really, I don't know this for sure.

[01:23:11]  Bruce: This is just my belief or conjecture or whatever.

[01:23:13]  Unidentified speaker: But I think Hofstadter's on to something.

[01:23:15]  Bruce: I think that this ability to just automatically somehow make connections between two unlike things and to realize that they're analogous to each other in some way and then apply that analogy and create knowledge by doing that. So we're going to do a podcast where I'll give like a really good example of this, but I've got several podcasts already where we talk about Hofstadter's theories. I don't see how LLM could ever do that in language space. So I suspect that that's necessary to make leaps as big across the fitness landscape as humans can do. So I think LLMs will end up being limited to how far they can be creative. You can probably make them a little more creative by giving them more knowledge, make them a little more creative, but I think they're always going to be like a small halo around a giant set of knowledge that was given to them by the human, if that makes any sense. They can create a little more knowledge outside that, but just a little.

[01:24:11]  Unidentified speaker: I mean, does the existence of multi-modal, natively multi-modal models change your opinion?

[01:24:20]  Dan: I mean, these models that they're releasing now can generate images. They can natively work with audio. So when you're talking about the deep hidden layers, the deeper kind of hidden states of these models, it's far outside of language itself already. You're talking about much deeper concepts.

[01:24:51]  Unidentified speaker: That's a fair point.

[01:24:52]  Bruce: I was talking about textual, but you're right with multi-modal, you are expanding. But the problem is that in my mind, so first of all, when I say the problem, it's not like I'm some authority on the subject.

[01:25:03]  Unidentified speaker: I'm just a layman like you, totally opining on something I don't understand, right?

[01:25:07]  Unidentified speaker: Because none of us understand AGI.

[01:25:10]  Bruce: So, I mean, like take everything I say with a grain of salt. But let me just say that they work the same way, right? They still use transformers and attention. The approach with multi-modal isn't really different than the way they do it with language space. It's true that it's no longer just language space, but it ends up being a small halo. Like, I honestly think that when AIs draw images, they're often very creative. My logo for this podcast, an AI came up with that. I gave the AI stuff something I had in mind, and it disobeyed me, and it did its own thing, and it was way more creative than what I had come up with.

[01:25:56]  Unidentified speaker: So I ended up going with what the AI had come up with.

[01:25:58]  Unidentified speaker: And it was done in one try, by the way.

[01:26:00]  Bruce: So you can't claim that I was doing it over and over, and the creativity therefore came from me. So I think that they are very capable, but they're playing basically in pixel space, right? They must have some sort of concept of different types of concepts and how they relate to words. And that's where the language space kind of comes in. So I suspect that when I make an argument about language space isn't enough, that adding multimodal doesn't change the argument. That yes, it's true that they now have a different kind of space they can explore through, but it will always forever be still attached to language space in some way. So you're only just going a little bit further, right? And you're still not going to be able to make the giant leaps that humans make.

[01:26:45]  Dan: I still think that when you're talking about the deeper levels of latent space of these models, like there's all sorts of overlap. There is no language space and image space and that kind of thing. Like all of these concepts, all of the deeper knowledge fits together like that.

[01:27:07]  Unidentified speaker: Yeah.

[01:27:08]  Bruce: I mean, these are concepts we make up, so we can try to talk about something that we don't fully understand. I do think the idea of language space makes sense to me in that humans do think in terms of language sometimes.

[01:27:23]  Unidentified speaker: I know I do.

[01:27:24]  Bruce: I like think things through using language, and I in my head sound a lot like DeepSeek, where I'm, but wait, what about this? Oh, this is what I can do about that, right? So I don't doubt that whatever it is that humans do, it includes what LLMs do.

[01:27:40]  Unidentified speaker: Sorry to push back again, Bruce, but you can actually, and they've talked about doing this, you can build these thinking models by removing the actual like articulating words out of it.

[01:27:54]  Dan: You can actually do the looping of the thinking, like in the layers before it articulates the words. So it's doing its thinking at a deeper level, if that makes sense.

[01:28:11]  Unidentified speaker: Yes.

[01:28:12]  Dan: And they have built some models that do this.

[01:28:16]  Unidentified speaker: So can you describe that a bit better?

[01:28:18]  Bruce: The way you just said that, that's sounded more or less like how neural networks work. There's always the different layers and that they have different features that the layers kind of find on their own.

[01:28:31]  Unidentified speaker: Yeah.

[01:28:35]  Dan: So as it's generating each new word, it's going down through a series of layers, and they use mechanistic interpretability, where they can interpret kind of what's going on in these deeper layers. And it's like deeper level concepts than just the word itself. It's doing planning and all that kind of stuff at these deeper layers. Until at the final layer, it takes these kind of deeper concepts and it translates that into the next word. And it makes total sense that if you're just doing the thinking part of it, you can get kind of a higher dimensionality, a higher expressiveness of knowledge in there. If instead of generating that, instead of that last step of generating that word that you see in the chain of thought, it just kind of takes the output of that pre-word. Right, you remove the top layer and puts it back into the pipeline of doing it all again. So in that sense, it's doing all the thinking without generating the actual words. It's kind of looping within these deeper concepts and planning and that kind of stuff. And they think that, and it makes total sense, that the chain of thought will become a lot more powerful, but it also scares some people because they like the fact that you can see the words of the chain of thought. Now you can see its thinking process, whereas it becomes more opaque if you take that part out.

[01:30:20]  Unidentified speaker: Right.

[01:30:21]  Bruce: So I happened to know that there was a case where they tried this. You can play like 20 questions with the large language model, and by looking at those lower layers, what they discovered was that the large language model actually comes up with totally different, like as you answer questions, it just totally changes what it had in mind. Yeah.

[01:30:44]  Unidentified speaker: Which makes sense to me, right?

[01:30:46]  Bruce: Because it's true that neural networks almost by definition create lower level features, and that's how they work, right? So that must be true of large language model. It must be very true of large language models because they're so huge, so many layers, right? But I don't know if it's organized. I have my doubts it's organized in a way that is consistent with how humans do it. And I suspect there's still something missing here.

[01:31:16]  Unidentified speaker: Okay.

[01:31:17]  Unidentified speaker: Well, there you go.

[01:31:19]  Unidentified speaker: That's why we're debating.

[01:31:20]  Unidentified speaker: Yeah.

[01:31:22]  Bruce: So, you know, and like I said, I've been wrong so many times about this. So, you know, I guess I owe you lunch, Dan, if they suddenly turn large language models into AGI.

[01:31:32]  Unidentified speaker: Then by the way, that would be cool.

[01:31:34]  Bruce: Like, I would be stoked to be wrong, right? Like, this is just not something I feel militant about. Like I just based on my study of intelligence on this podcast and trying to look into it, I feel like there's just something really missing at the language space for training it on.

[01:31:51]  Unidentified speaker: Like even just animal intelligence.

[01:31:53]  Unidentified speaker: Think about animal intelligence.

[01:31:55]  Bruce: One of the questions that I've asked, I guess I haven't explicitly asked this.

[01:32:00]  Unidentified speaker: So I'm going to explicitly ask it now.

[01:32:01]  Bruce: On this podcast, we've studied animal intelligence a number of times and had a whole bunch of podcasts about it. It's not an accident that I do that on a podcast that's about ultimately about AGI. One of the things that you have to actually ask yourself is, is animal intelligence a part of human intelligence? Okay, so in that Aeon article, David Deutsch points out that the amount of knowledge in our DNA between a chimp and us is not very large. So he argues that the creative algorithm has to be very small because it exists, has to exist in that small amount of DNA that is different between humans and chimps or apes or whatever animal it was, some sort of primate. The problem I have with that is that he's making an assumption that seems totally unjustified to me, which is that the creative algorithm exists separate from animal intelligence. Animal intelligence is this super deep, amazingly intelligent set of algorithms that blow away anything we've ever done before, right? The way an animal can just move around in 3D space and take care of itself and come up with creative, and I do mean creative, ways to create knowledge to be able to survive. It's stunning what they can do, right? Like, we have no idea how to do what they're doing. I think it's way more likely, plausible, I guess I should say, for crit rats, that whatever happened was is humans use animal intelligence and then there's this one additional thing that suddenly broke through, created the universal jump, right? And my guess is it's language, by the way, okay? So that's my guess, I don't know.

[01:33:56]  Bruce: But I think somehow that little bit of extra knowledge between chimps and us gave us some sort of language ability that chimps can't do that allowed us to use our animal intelligence in a novel new way that chimps can't do. Chimps can learn a certain amount of language, but there's something missing, right?

[01:34:14]  Unidentified speaker: They're super limited in what they can do with language.

[01:34:18]  Bruce: And I think whatever that is, it had something to do with language. And but I think human intelligence is still mostly animal intelligence. And I don't see how large language models really mimic animal intelligence at all. Like, I think that that might actually be orthogonal, right? It's not that it's orthogonal to AGI, I think that they're part of what we need to learn to make AGI work. But I think we probably need to figure out animal intelligence.

[01:34:47]  Unidentified speaker: And this just makes, to me, this makes so much more sense.

[01:34:49]  Bruce: This idea that there was some sort of, in what, 100,000 years, suddenly all the evolution took place from what before was these animals that didn't create knowledge and created no knowledge and all their knowledge was in their genes. And suddenly, a few K was all we needed to create this creative algorithm that creates knowledge. Like, it just doesn't even make sense, right?

[01:35:16]  Unidentified speaker: Animals do create knowledge.

[01:35:18]  Bruce: They're just, they just can't do it open-endedly. And something was added on top of that that suddenly broke through and created open-endedness. And I suspect it was language or something language related, but I don't know that for sure.

[01:35:31]  Unidentified speaker: You know, your opinion may vary.

[01:35:34]  Bruce: But I feel like we're like what we're doing with LLMs is we're studying that last little step without understanding everything that needed to come before. And it's amazing what we can do with it.

[01:35:45]  Unidentified speaker: It's stunning what we can do with it.

[01:35:47]  Bruce: Like it blows me away what we can do with it. Way more useful than I would have thought, but I just can't help but feel for that matter. I don't think LLMs will ever be, they'll ever be something it's like to be an LLM because I don't think they have any consciousness. Animals have consciousness, or at least some animals, maybe the higher animals only, have consciousness. Somewhere in there, there was this evolution where qualia came to be. And based on our best theories, we don't know this for sure, but based on our best theories, animals evolved that prior to humans. Like there's tons of theories around that that have been corroborated by studies.

[01:36:25]  Unidentified speaker: There's a giant literature on it.

[01:36:27]  Unidentified speaker: And there aren't good alternative theories.

[01:36:29]  Bruce: Like, you just can't find good testable alternative theories to the idea that certain animals, particularly mammals, have consciousness. Here, I'm using consciousness to mean qualia. And really, I shouldn't because consciousness is... Qualia is part of consciousness, but it's like not quite the same thing, right? Because you could have an animal that there's something to be like that has qualia, but it has like no memory.

[01:36:51]  Unidentified speaker: And I question whether that would still be considered conscious.

[01:36:55]  Bruce: So that must be some sort of learning algorithm itself. Qualia must be some sort of learning algorithm that I don't think we're even studying with large language models. So, and again, this depends on...

[01:37:07]  Unidentified speaker: This is my way of looking at it.

[01:37:09]  Bruce: If I'm David Deutsch, who believes it all did happen in that last 2K of data or whatever, then he thinks that means that only humans have qualia and animals are just automatons. If he's right and I'm wrong, then it would make perfect sense to pursue what LLMs are doing and to think that those could be turned into people. If I'm right, that animals evolved that first, if I say if I'm right, if the literature is right, if the current best theories are right, which they may not be, then there is something missing from large language models and we need to still figure out what that is.

[01:37:46]  Unidentified speaker: So that's why I guess I do have my doubts about it.

[01:37:50]  Dan: I mean, yeah, what you're saying essentially like, like David Deutsch is saying that AGI is that delta between animal intelligence and human intelligence. And what you're saying is that AGI is animal intelligence plus that delta.

[01:38:05]  Unidentified speaker: Yes, that's what I'm saying.

[01:38:08]  Dan: And I guess my final point is that, is to maybe open your mind to the possibility of that with LLMs, we're kind of coming at this from the inside out, or instead of from the bottom up, I think that's Yann LeCun's thing, like this thing isn't even as smart as a dog.

[01:38:37]  Unidentified speaker: Right.

[01:38:39]  Dan: Instead of programming the dog and then programming intelligence on top of that, that perhaps it's possible to come from the other direction and start from the highest language level, and maybe you can get at the deeper stuff from that. Maybe we're actually piecing together the deeper stuff from the highest level. Maybe we're coming at this from the opposite direction as what you would expect it to be built as.

[01:39:13]  Unidentified speaker: So that's a very good point.

[01:39:15]  Bruce: Let me actually say something in favor of that point. So David Deutsch has expressed in his books this idea that emergent explanations are no less important than reductive explanations.

[01:39:26]  Unidentified speaker: And I totally agree with that, right?

[01:39:28]  Bruce: In fact, he even goes so far as to say, in theory, we always talk about how you could theoretically figure out an emergent explanation out of the reductive explanations and therefore, that's why a reductivist would say reductive explanations is better. Deutsch points out that it's also true that you could theoretically figure out the reductive explanations using the emergent explanations, that they create constraints as to what the reductive explanations are are possible. And so you could theoretically use the emergent explanations to figure out and to come up with what the proper reductive explanation was. And I think that's true also. So from that standpoint, LLMs, Large Language Models, it's studying them could could be a stepping stone to understanding animal intelligence, right? Like, I guess I see no reason why that couldn't be true. I don't think the current techniques would ever by itself be animal intelligence, but it might create an understanding or a breakthrough or a stepping stone that then constrains what animal intelligence can be and allows us to figure out what animal intelligence is. So if that's what you're saying, I guess I would fully endorse what you just said.

[01:40:51]  Unidentified speaker: We shall see.

[01:40:53]  Unidentified speaker: Well, I have what could be a concluding question.

[01:40:56]  Peter: So this is purely based on speculation, I guess.

[01:41:01]  Unidentified speaker: But what will these AGI's be like?

[01:41:05]  Peter: I mean, say we create these universal explainers capable of creating open-ended knowledge. I kind of get the feeling that critical rationalists or that the fans of Deutsch say that, well, they'll pretty much be just like humans, or they should be morally treated like humans. Maybe they'll have access to more memory, but we kind of have that with our phones in our pockets anyway. And they might be better at a couple of things, but they'll more or less be just like humans.

[01:41:47]  Unidentified speaker: It sounds like Bruce is not even convinced they'll have qualia, right?

[01:41:53]  Unidentified speaker: No, no, that's not necessarily true.

[01:41:56]  Bruce: So, okay, let me, first of all, I don't know.

[01:42:01]  Unidentified speaker: Yeah, of course.

[01:42:01]  Unidentified speaker: I just want you both to speculate on it.

[01:42:03]  Bruce: Yeah, so I was arguing at the time that until we understand qualia, we probably won't be able to make a general intelligence, but that qualia is a learning algorithm that animals have.

[01:42:16]  Unidentified speaker: It's not unique to humans.

[01:42:18]  Bruce: Therefore, if we only study large language models, I was arguing that since that doesn't include qualia, the learning algorithm that is qualia, then it would be incomplete and they won't be able to become general intelligences. Now, of course, I'm making so many assumptions here, right? I mean, it's crazy how many, maybe even ridiculous assumptions I'm making here, including even just the idea that animals do have qualia. That could turn out to be wrong.

[01:42:46]  Unidentified speaker: It is our best theory.

[01:42:48]  Unidentified speaker: That is a fact, right?

[01:42:50]  Bruce: Anyone who tells you it's not our best theory has not looked into it, period, end of story. But the fact that it's a best theory doesn't make it true.

[01:42:57]  Unidentified speaker: And that's just a fact, right?

[01:42:59]  Bruce: And so it could turn out that intelligence, general intelligence is separatable from consciousness.

[01:43:08]  Unidentified speaker: And that is possible.

[01:43:10]  Bruce: Like, I don't think that's true, but like it's really hard for me to give you a principal reason why that couldn't be true.

[01:43:18]  Unidentified speaker: Okay.

[01:43:19]  Unidentified speaker: Right?

[01:43:19]  Bruce: This idea of philosophical zombies. I guess my argument against that would be that philosophical zombies just don't seem to exist, which suggests if animals have qualia, the higher animals have qualia, then you probably have to pass through qualia, the qualia algorithm prior to general intelligence.

[01:43:39]  Unidentified speaker: Okay?

[01:43:40]  Bruce: And if that's, if that's true, then it's literally impossible to make a general intelligence without qualia.

[01:43:46]  Unidentified speaker: But I don't know that.

[01:43:47]  Bruce: It could be that the animal intelligence, that the qualia has got nothing to do with our ability to do general intelligence. It's just something that by evolution will, you know, parochial reasons that path that evolution happened to take, that it worked out that way.

[01:44:00]  Unidentified speaker: And there is an argument to be made.

[01:44:01]  Bruce: And I've seen people make this argument that we will be able to create intelligences, general intelligences, that can think as well as us, but have no consciousness.

[01:44:12]  Peter: So they could be psychopaths that just want to turn the world into a paperclip factory.

[01:44:18]  Unidentified speaker: Is that plausible?

[01:44:21]  Bruce: No, that's really not plausible. They could be psychopaths if you went that far, like that's at least somewhat plausible.

[01:44:29]  Unidentified speaker: I suspect more likely.

[01:44:30]  Bruce: So like some people argue they'd be psychopaths. And there is some reason to be concerned about that. But Steven Pinker has made a, I think, fairly credible argument that to be a psychopath, you have to have certain amounts, certain kinds of human feelings that were part of evolution to want to dominate and things like that. And there's no reason for them to have evolved that. So it's less likely they'd be psychopaths and more likely that they'd just kind of, you know, be some sort of benevolent autistic genius or, you know, Rain Man or something like that, right? So, I mean, we don't know, like, we really don't know.

[01:45:08]  Unidentified speaker: That's the problem, right?

[01:45:10]  Bruce: Until we have a good theory, we're making, we're grasping at straws, trying to figure this out. So what I wanted to clarify though, is that I personally think and believe that animal intelligence is necessary for AGI and therefore qualia is necessary for AGI. But I don't know enough to claim that that's a, that isn't a best theory. That's my belief, it is not a best theory. And there are, because we're in vague theory territory, there are many competing theories that can't be refuted and that are at least as good as my theory, right? So I'm trying to just be truthful here that I don't know, but my point of view was that they did have to have qualia to be a general intelligence.

[01:45:57]  Unidentified speaker: So there'll be a lot like humans.

[01:45:59]  Unidentified speaker: Yes.

[01:46:00]  Unidentified speaker: Okay.

[01:46:01]  Dan: And I guess what I'm arguing is that if you define AGI as open-ended creativity, that you don't necessarily need the qualia. LLMs as is have the potential of working in this open-ended creative way of building these stepping stones. And if that's how we're defining AGI, of creating novel new knowledge like that, then that's possible. And I pinned to the top of my Twitter feed a debate between Ken Stanley and a doomer, essentially. A rationalist doomer who believes that it all comes down to optimization. Everything is an optimization problem. And if you think like that, then yeah, the AI could be optimized to create paper clips and go off and figure out how to turn us all into paper clips. But Ken Stanley does a good job of arguing against that, that when it comes to open-ended knowledge creation, the AGI has to be able to follow its interests. And it doesn't come down to optimization.

[01:47:16]  Unidentified speaker: It comes down to...

[01:47:18]  Dan: It doesn't make any sense from an AGI perspective of turning everything into paper clips. From an AGI's perspective, it's about following its interests and wherever that leads.

[01:47:36]  Unidentified speaker: But you both think that Qualia could in principle be programmed into it?

[01:47:41]  Unidentified speaker: Yes.

[01:47:42]  Dan: I do agree that it's a program and perhaps under my structure, it could discover the Qualia on its own, perhaps.

[01:47:52]  Unidentified speaker: Oh, okay.

[01:47:54]  Bruce: I just have to do a shout out to Dr. Jerry Swan, who I've read his book, The Road to General Intelligence, and I've communicated with him via email. He has a theory of general intelligence in his book, which is one of the more complete ones that I've seen. And in his theory, a general intelligence is not a person. So this idea that a general intelligence has to be a person that comes from Deutsch, he actually has a different opinion on that. So he actually does imagine that we'll be able to create general intelligences that simply have no consciousness, so there is no moral qualms about using them in automation or something like that, right? So I know David Deutsch would argue against that as, okay, but then it's not really an open-ended creativity because it's really just been programmed to do a specific thing. But you have to actually see his theory. It's a little more convincing than you would first think, right? And you have to look at the mathematics that he puts together.

[01:48:57]  Unidentified speaker: And of course, he's guessing too, right?

[01:48:58]  Bruce: Because none of us know what AGI is. But it may be that both of them are right, that there is, like, consider the fact that an LLM is in some sense already a general intelligence. It is not a narrow AI anymore. It can mimic almost any other kind of AI. So it is a general intelligence, although it isn't the kind of general intelligence that I originally had in mind when I was thinking of a person.

[01:49:25]  Unidentified speaker: So I'm going to move the goalposts here.

[01:49:27]  Bruce: So it may be that general intelligence and personhood are separable and that they're not quite the same thing. And LLMs may already be a sign that that's the case, right? If that's true, then Jerry Swan's theory would be about general intelligence and Deutsch's theory would be about personhood, and they could both still be right.

[01:49:48]  Unidentified speaker: But of course, we don't know.

[01:49:49]  Bruce: I just wanted to put it out there as a possibility because I thought it was an interesting theory that deserved some attention.

[01:49:58]  Unidentified speaker: Well, okay, this has been wonderful.

[01:50:00]  Peter: I've gotten a lot out of listening to you guys, and I'll look forward to coming back and listening again while I'm editing. But thank you so much, Dan, and thank you so much, Bruce.

[01:50:13]  Bruce: Yes, thank you, Dan. Dan, this was really thought-provoking episode, and I can really appreciate what you had to offer here and the thoughts that you pulled together here.

[01:50:24]  Unidentified speaker: Yeah, thank you.

[01:50:25]  Dan: I mean, it's cool that other people might listen to this. I just enjoy talking about this with you guys, so I appreciate it.

[01:50:34]  Peter: We're a very self-indulgent podcast. We're just going our own direction, and if people want to come along, that's great, and at least a few people out there do.

[01:50:45]  Unidentified speaker: We do have, I meant to mention, we do have some Patreon subscribers.

[01:50:50]  Unidentified speaker: We do.

[01:50:52]  Unidentified speaker: That was really the idea that people...

[01:50:55]  Unidentified speaker: Are we professionals now, Bruce?

[01:50:57]  Unidentified speaker: Yeah.

[01:50:59]  Bruce: If you make 25, 30 bucks a month, I think you can count yourself as a professional.

[01:51:03]  Unidentified speaker: I think that's exactly how it works.

[01:51:05]  Unidentified speaker: Okay.

[01:51:06]  Unidentified speaker: Well, very cool.

[01:51:08]  Unidentified speaker: Okay.

[01:51:09]  Unidentified speaker: Take care, guys.

[01:51:10]  Unidentified speaker: All right.

[01:51:10]  Unidentified speaker: Okay.

[01:51:11]  Unidentified speaker: Thanks, guys.

[01:51:19]  Unidentified speaker: Hello again.

[01:51:20]  Peter: If you've made it this far, please consider giving us a nice rating on whatever platform you use, or even making a financial contribution through the link provided in the show notes. As you probably know, we are a podcast loosely tied together by the Popper Deutsch Theory of Knowledge. We believe David Deutsch's Four Strands tie everything together. So we discuss science, knowledge, computation, politics, art, and especially the search for artificial general intelligence. Also, please consider connecting with Bruce on X at BNielsen01. Also, please consider joining the Facebook group The Many Worlds of David Deutsch, where Bruce and I first started connecting.

[01:52:03]  Unidentified speaker: Thank you.