Episode 96: Kenneth Stanley on the Pursuit of What’s Interesting

• Links to this episode: Spotify / Apple Podcasts

Unofficial, reviewed transcript. It may still contain mistakes; check the podcast when wording matters.

Speakers are labeled by first name where identified. Unidentified-speaker labels may refer to different people. Bracketed question marks indicate uncertain wording.

Transcript

[00:00:00]  Peter: Hello out there. This week on The Theory of Anything podcast, we interview AI researcher Kenneth Stanley about what he calls the myth of the objective. In his book on this, he asserts that in all the complicated areas of life, from machine learning to education and more, we should strive to follow what is interesting rather than a predefined objective. He is such a calm, reasonable guy that it might at first be easy to overlook just how radical this idea is that goes against so many known knowns. What I hear him saying is that on a core level, whether in natural selection, machine learning algorithms, or human brains, what really matters is the search for novelty. This is how we solve problems, how knowledge increases, and maybe how we grow as humans. Out of all the out there ideas we have looked at on this show, from Many Worlds to Tipler's Omega Point, this may actually be one of the most out there, but also strangely sensible too. Perhaps an idea that is even likely to influence our lives in a positive way as we wrestle with his thesis.

[00:01:25]  Bruce: Welcome to The Theory of Anything Podcast. We have Peter here, and we have with us Kenneth Stanley today. Hey, Kenneth, how are you doing?

[00:01:34]  Kenneth: Great, great, I'm very glad to be here. Thanks for having me.

[00:01:37]  Bruce: He is the author of the book, Why Greatness Cannot Be Planned, The Myth of the Objective. We did an episode on his book. Actually, we've done multiple episodes on his ideas, but we did one specifically on his book several episodes ago. And then he also featured prominently in our episode on the problem of open-endedness. I had several people I was quoting, but he was one of them that I kind of went over his ideas. So we wanted to bring him onto the show, and I wanted to have a chance to ask him all the burning questions that I'd been wanting to ask him ever since coming across his articles and his books. So Ken has graciously agreed to come on. And so we're going to probably jump right in to maybe start with an introduction for him and kind of what his ideas are. So Ken, can you maybe just... We have on this podcast already gone over your ideas in a great deal of detail, but there's no guarantee that the person listening to this particular podcast has listened to those podcasts. Can we maybe get a quick summary of like your main thesis, your main ideas and how you came up with them and things like that, and just kind of give us a good introduction to your theories?

[00:02:57]  Unidentified speaker: Okay.

[00:02:58]  Kenneth: And yeah, happy to do that. And I think we're talking about mainly the ideas in my book.

[00:03:04]  Unidentified speaker: Right?

[00:03:05]  Kenneth: Not like all ideas I've had, but...

[00:03:08]  Bruce: But if you have related ideas, I'd be actually interested in those. I'm familiar with the ones from your book, but yeah, just...

[00:03:15]  Kenneth: Okay. We'll start out with the book. So I think that's really where a lot of the topics we like to go, which is based on an idea that setting objectives can actually have a really bad effect on pursuing, especially innovation and creativity. But just discovery in general, it can be really undermined by setting an objective. And it's such a general point, because we set objectives all the time. It's a normal thing for us to do that it needs a lot of kind of explanation, like how we would come to a conclusion like that. But if it's true, clearly it has really profound implications, because setting objectives is so ubiquitous across the culture, across institutions, as individuals, like everywhere is about setting objectives. And so it was a really long road to getting to this kind of viewpoint, which went mostly through artificial intelligence, because I'm an artificial intelligence researcher. And most of my career was spent doing that. And so really I was focusing on algorithms, algorithms for learning, learning algorithms. Some of them were evolutionary in nature. And experiments in that area revealed that there were some funny things going on when algorithms that usually would try to be optimized towards some objective would actually perform worse than algorithms that didn't have an explicit objective. And I mean, we could get into the details if we want, but these kinds of experimental comparisons started to reveal this principle.

[00:04:47]  Kenneth: And if it doesn't make sense, because usually it doesn't make sense because it's counterintuitive, the reason that this would happen is because objectives are often deceptive, which means that it appears that you're approaching the correct direction to get to your objective, but you're actually not. You're being deceived, and this is common in complex spaces with hard problems. I should note that that's why this is counterintuitive, is because in less complex spaces they tend not to be deceptive, and therefore objectives do work. So, many of us have experience with setting objectives and succeeding, and because of that we intuitively believe in objectives and sort of across everything. And that actually isn't the case, because once you get into really high dimensional spaces and problems, you start to face the problem that those spaces are complex and deceptive. And so, because of that, like the objective can cause you to hit a dead end. But the alternative, which sounds to most people at first, like I'm advocating being random, isn't that's not what I'm advocating. Randomness is not a good algorithm either. So, optimization has its limits. Optimization means basically optimizing towards an objective, but randomization also has big limits. It's not like a great formula for achieving something you want to achieve. Rather, you would do something that's sophisticated in a different way than we're used to.

[00:06:03]  Kenneth: So, what we're used to is optimization means following a gradient towards an objective, but in terms of the alternative, it means following a different gradient, but it's still an information-rich gradient, something that's informed by information, but it would be something like novelty or interestingness. So, I don't know where I'm going, and that's okay, but I know that where I'm going seems interesting to me. And really, the difference is that when you have an objective, you're comparing where you are to where you want to be. But when you're pursuing something like novelty or interestingness, you're comparing where you are to where you've been in the past. So either way, you're making decisions based on a comparison, which is informed by information, but it's a different kind of information. And the information about the past is actually really interesting and rich, because actually we know more about the past than we know about the future. And so in terms of creative pursuits, innovation, discovery, in other words, how to achieve blue sky objectives that are far beyond our understanding right now, or the way I usually put it, more than one stepping stone away, it requires a degree of exploration to achieve those things. And then setting an objective can actually be antithetical to eventually achieving those objectives paradoxically, which is why I call it the objective paradox often.

[00:07:15]  Kenneth: And so you need sometimes to just back up, forget about worrying about where we're going, actually just drop the idea we're pursuing an objective, and instead collect interesting stepping stones and hope someday that they will lead to some of the things that once we thought of as objectives.

[00:07:31]  Peter: Can I just emphasize before Bruce gets going one thing you said there for our audience? So we're not just talking about something in machine learning, right? This is business, this is art, this is just our personal lives perhaps, this is just like a basic epistemological statement about life that you're making. Is that fair to say?

[00:07:56]  Kenneth: Yeah, that's a good point to bring into this for clarity and also because it's interesting. It's interesting that what I'm saying, which absolutely I totally agree is not just about algorithms, it's very much about life, it's very much about society and how we run it, it's about how we run our institutions. Because the same principles that apply algorithmically also apply to all of those other spheres of life. As a company, in Silicon Valley, you hear about things like OKRs or Objectives and Key Results. It's basically applying these algorithmic principles in an institutional setting because we have an absolute die-hard belief in the iron-clad reliability of objectives, which is something interesting to get into. But what happened with this research, I think is a very unique story in the field of artificial intelligence, where something that was originally really just algorithmic observation seeped out or you could say leaked out into social critique. I mean, I've actually never heard of this. I don't know of any other story like this. And it's not just social critique. I mean, it has an emotional valence for a lot of people. Like a lot of people feel like it changes how they feel to understand that they don't need to justify all their endeavors through objectives. And I've had people that I've encountered like talking about this for the first time, that in certain contexts have been near tears. And this is like to me an astounding thing to think about that people could be almost crying about something that is really just a dry result from algorithmic research.

[00:09:25]  Kenneth: And I don't think there's any other story like that in the history of the field.

[00:09:30]  Unidentified speaker: Thank you.

[00:09:31]  Bruce: So, you know, I, in general, I definitely agree with your thesis here, but being the good critical rationalist that I am, I immediately tried to think of the strongest counter examples that I could think of. So one that came to mind was I'd read one of the biographies on Elon Musk and how and why he built SpaceX. So here we have a guy that if the biographies to be believed, which I suspect it is, he actually built SpaceX because he has a grand objective or goal to get humans to Mars, so that when Earth gets wiped out some day, that humanity doesn't get wiped out with it. So here's this guy who's started a company, it's been successful, building rockets. The reason why he's doing this is because he has this objective, or maybe I'm abusing the word objective here, that's one of my questions for you, to try to get to Mars. And it seems like he's somewhat successful, right? Like it hasn't really necessarily been deceptive for him because he has improved rockets and we're still not to Mars, but surely we've made strides in the right direction to where that may someday come to be because of the technologies that he has created along the way. Is this an encounter example to the myth of the objective, or is this not really an objective and not really what you mean? Certainly, you could word it like it's some sort of ambitious objective, and that's how he sometimes words it.

[00:11:04]  Unidentified speaker: Yeah.

[00:11:05]  Kenneth: This is a fun game to play that I inadvertently created by writing this book, which is to find counter examples and say, can I show that this isn't always true? There's a lot of nuance though to analyzing these real-world cases. You really have to pick it apart to understand what it actually means. At one level, what you have to think about is that in this particular case, for example, we actually don't have the resolution of the question yet, so we can't really say it is or isn't a counter example yet. Like you say, surely we've made progress, but you have to remember though, the whole point of deception and the objective paradox is that it always looks like you're making progress when you're being deceived. So even the smartest person will think that they're making progress because that's the definition of deception.

[00:11:56]  Unidentified speaker: It's just a truism.

[00:11:57]  Kenneth: Now, I'm not saying he's being deceived. We don't have evidence of that yet because maybe he is making progress. So this story is not finished yet. So the jury remains out and it could turn into something that you might argue or you might have more reason to argue as a counter example if he really does get us to Mars. Now, one possibility here is that really I could argue that, well, actually just analyzing this particular case, it's a good example of somebody who's actually a victim of the exact objective paradox we're talking about. This is too ambitious. It's too many stepping stones away and he's making a mistake. So we're not going to go to Mars, at least not on his stepping stones that he's laid for us. I can't prove that though since, like I said, the jury remains out on this particular example, but it's very possible. But maybe a more nuanced view of this is like, what usually is instrumental in analyzing a particular case is this kind of subtle distinction that I make often when I talk about this, which is that there is a moment where it's okay to set an objective. Like I conceded early in our conversation, sometimes objectives work. And I said, the way I couched it back a little bit of a while ago is saying that when objectives work, usually it's because they're relatively modest. But there's another kind of situation which actually causes objectives that were once extremely ambitious to become more modest, which is when you get to a point where you're close enough, that something that used to be far away is no longer far away. And I usually articulate that.

[00:13:33]  Kenneth: I usually call it one stepping stone away. I think about it like the approach to an objective is basically walking across a sequence of stepping stones. Like first you have to invent this and then this and then this. And obviously a lot of these stepping stones have been laid, like the invention of rockets, for example, precedes Elon Musk but is probably relevant to getting to Mars. And so we've traversed some stepping stones, but there's some stepping stones left to go. And I, the way, and this is just a conceptual point because it's not formal, but basically what I'm saying is that when you're one stepping stone away, suddenly something that was inconceivable as an objective snaps into possibility. Because now all the stepping stones are laid. So it becomes like what you might call like, I'll put in quotes, modest. It's still not necessarily modest. I mean, and by the way, the things I call modest can still be a lot of work. It's like something like getting in shape or something. It's like getting in shape might actually be hard. It might take years of work. It's not an easy thing to do, but we do know how to do it. Like we know what the steps are to getting in shape. So it doesn't mean, I don't mean to demean things by calling them modest. I just mean that they're not subject to this objective paradox. And so like the real question with visionaries, or the people that we usually think of as visionaries like Elon Musk, somebody with this like amazing achievement that they're trying to achieve, is are we one stepping stone away? That's what I would ask.

[00:14:54]  Kenneth: And usually I think a really good visionary, or what I would call a genuine visionary, because there's a lot of false visionaries. A genuine visionary is somebody who realizes for the first time that we're one stepping stone away, or you could say the first person to realize that we're one stepping stone away. It's like you're keeping an eye on the accumulation of stepping stones in the entire world. Of course, you're probably only keeping an eye on ones relevant to your interests, but you're keeping an eye on those. And you notice before anyone else that something has snapped into possibility that wasn't possible before. And this is very different from someone when we're really far away from something, many stepping stones away, who just creates some crazy, far-off objective just to get people hyped up and excited, which has almost no chance of being achieved. That's more the way that visionaries are depicted in mythology and fiction and things. It's like somebody who sees way across the horizon, beyond anything we can see. I don't believe those people exist.

[00:15:53]  Unidentified speaker: That's basically impossible.

[00:15:54]  Kenneth: Obviously, you may come up with some one exception, because this is not a theoretical point. This is a conceptual point. So possible there have been exceptions in history where something highly unlikely happened. But in general, you won't find cases like that because it's basically impossible to see multiple stepping stones into the future. So the real instrumental question in this particular case is, did he see that we're one stepping stone away before other people, that we actually now do have the technology to get to Mars? It doesn't mean there's not some work that needs to be done. But in effect, the work that needs to be done is pretty evident from the start. And so it's just a matter of doing the work that needs to be done. And I still am not saying that is where we are, but it's possible and then there would be no contradiction. Because that's consistent with what I'm saying, when you're one stepping stone away. Another, I just want to put in one third point about this, because it's a very complicated thing to say there's a counter example. There's also the possibility here that that isn't actually an objective in his mind. The biography, which I haven't read, may have a different connotation. It's not clear, but it's very hard to really know someone's mind. Sometimes it's hard to know your own mind. But you can see that there's a lot of possible fruits to this labor other than getting to Mars. Intuitively, he may understand that there's an opening here for a kind of an industry or in industry that hasn't existed before.

[00:17:18]  Kenneth: That's like a different thing than getting to Mars. Like the space industry, because of changes at NASA and various other kinds of political issues, has changed, and he just stepped into the vacuum. It was a very astute move. It actually moves towards a different objective, which is still ambitious, but also really interesting, which is to dominate this future industry that's going to be very important. He's in the right place at the right time. Interesting that he can do this, and that he may even have used this narrative about Mars just as a motivation. It's not actually an objective, but just as a motivator, it's sort of an advertisement to get people to rally around the real objective, which is more realistic, which is basically to dominate the space industry. You can see certain interesting stepping stones, like you can see SpaceX to Starlink or something. That's a more conceivable short-term thing, which leads to a satellite domination of communication networks, which is pretty considerable and amazing. It's hard to say what really was the genuine fundamental motivation here, so that I can critique it and say, did he defy the objective paradox or not? I think it's a very multi-faceted and complicated thing, and every case like this, you can have this back and forth and argue about. But maybe if you really want to get to counter examples, you would go to things that actually have happened. We got to the moon, for example. What's the explanation of that? Would I concede that? Or we built planes that could fly, like somebody tried and they succeeded.

[00:18:45]  Kenneth: And generally, I would always say the same thing, that you're focusing only on the last step in a long eternal chain. And almost every step in the chain until that final step had absolutely nothing to do with that achievement. Like all the prior inventions that were essential to getting to space, to getting an airplane, to any kind of amazing thing, to building a computer. Like the vacuum tubes that lead to the computer, the people working on them were not thinking about computers. If they were, they wouldn't have been working on vacuum tubes, and we'd have no vacuum tubes and no computers. And so I think I would present the hard assertion that there's no invention ever in history that doesn't have this property, that the stepping stones were conceived at some point in the past without understanding the final product that they were leading to.

[00:19:33]  Unidentified speaker: All right.

[00:19:33]  Unidentified speaker: Thank you.

[00:19:34]  Bruce: Let me now use one that's a little closer to home as a possible counter example. I loved your article about open-endedness as the last grand challenge that you've never heard of. You were one of the authors on that article. What's the difference between an objective and a grand challenge? So what kind of what I have in mind here is that obviously this has motivated you and the people who wrote the article and maybe members of your team, this idea that there's this problem of open-endedness to solve. That is maybe arguably an ambitious objective because of course you have no idea what the actual stepping stones are towards it. So and yet it did motivate you to go out and move on to do novelty search, which we're going to talk about in a second and trying to find a stepping stone towards that goal. So what's the difference then between an objective and a grand challenge like this in your mind? In what place does a grand challenge have if it's too much like an ambitious goal?

[00:20:34]  Kenneth: Yeah, this is another really good question that gets into a lot of subtleties, which are important. People often wonder, how do you operate with no objectives at all? What does that mean? Are you allowed to be an AI researcher since that's sort of an implicit objective or an open-endedness researcher like when you're posing it now? The way that I would think about this is that there are two ways to be dedicated to a cause, but it's really subtle. The way that almost everybody always thinks of is you dedicated to the cause as your objective. Like my objective is AGI, something like that, or my objective is to create strong open-endedness, like I could say that. And then my theory would suggest that that's actually an exercise in futility. But more, it's not so much that it's an exercise in futility. It should be more, a little bit more nuanced than that. It's basically that by creating an objective like that, I'm actually blinding myself to a lot of interesting things that I could be doing that might even be related to that objective, but only because they don't seem to actually be moving me closer to it. And so it has a sort of deleterious effect on my ability to innovate and think out of the box, just by the virtue of setting that objective, and that's the downside of it. And so then, what's the second way then that I can be dedicated to the cause? Well, the second way to be dedicated to the cause is basically, the way I think of it is that I am interested in the stepping stone itself, not where it's going. And so another way I think of stepping stones is playgrounds.

[00:22:11]  Kenneth: Like, playgrounds are places that when you go into, there's all these things you could do. And it's kind of like overwhelming. Like, I'm thinking from the perspective of like a five-year-old. Like, you go to the playground, suddenly there's all these things that you could explore. Like, I climb up that thing over there, slide down that thing. So like, I could swing over there. Like, what's that rope do? I don't even know what it's for, but it's something I can explore. And you go into a playground, it means you've opened up new stepping stones. Like, from the stepping stone of the playground, now you will discover new things to explore, which will lead to other types of things that you can discover. And in that sense, like, I think of AI or open-endedness as a playground. If you think about the article, like, is what I'm advocating, is that this is a playground where nobody's playing and that's a mistake. Like, there's got to be tons of interesting stepping stones in that world, the playground of open-endedness. It's different than saying we're going to achieve open-endedness as an objective, which I haven't and wouldn't claim to have achieved. But certainly by hanging out in that playground, I've uncovered a number of really interesting stepping stones, may some someday lead to actually solving the things, so to speak. I mean, it's possible someday through some circuitous route, but I don't think of myself as trying to optimize along that gradient. I'm just exploring that playground. And so I can be an AI researcher because I'm interested in AI.

[00:23:26]  Kenneth: It's not because I'm pursuing it.

[00:23:27]  Unidentified speaker: It's a subtle thing.

[00:23:28]  Kenneth: What's the difference between pursuing a goal and just being interested in a subject? But I have to do that to be consistent with my theory. It's like I have to think in hindsight, like why did I get into AI if I'm against having an objective?

[00:23:40]  Unidentified speaker: Like that's true.

[00:23:40]  Kenneth: It sounds like a contradiction. But I think the resolution is it's just an interest. I'm interested in the area. And so I'll do algorithms that I don't necessarily know or have an opinion about whether they're advancing us towards AGI or not. But I don't care because the algorithms are intrinsically interesting in their own right. And that's philosophically very different from like how the field works. Because generally the field works by comparing any new algorithm to the current algorithms to see if you're making progress towards the goal, which is AGI, which is usually represented through a bunch of proxy benchmarks. And so I just don't care about that stuff because I think that I'm independently confident in my own ability to assess whether something is interesting, irregardless of a bunch of probably deceptive benchmarks.

[00:24:24]  Unidentified speaker: All right.

[00:24:24]  Peter: Something that I've been interested in lately is the way that all these, practically every technology you can think of has been predicted by science fiction authors at some point. I mean, whether it's the internet or the tablet or just everything. So maybe this is the most extreme example where these guys are kind of in the playground playing with ideas and concepts, but they're not setting objectives. An objective is when you're trying to take specific steps to walk towards something in a more algorithmic way. Is that a fair way to put it?

[00:24:59]  Unidentified speaker: Yeah.

[00:24:59]  Kenneth: I mean, I think science fiction writers are just talking about things that are interesting. They're not actually achieving the things that they're describing. I mean, it's like they could talk about time travel, but it doesn't mean it's been achieved. It also doesn't mean that they're helping us achieve it. I don't think fiction about time travel has advanced the cause of time travel in a significant way. So it's just kind of an aperture into the playground, just showing you there's a place there. I mean, there's many centuries people were thinking of flying machines. It's kind of an obvious cool thing before the Wright brothers. But it didn't cause the flying machines to work at all. I mean, Da Vinci even tried to build flying machine without success. So we had to wait until the stepping stones were laid to be able to build things like that. So yeah, I don't think it fundamentally changes things, but it's true they play a useful role in sort of helping us to refine our interestingness detector. So it's not like an irrelevant part of the cultural cooperation that leads eventually to exploring these things.

[00:26:00]  Unidentified speaker: It's very important.

[00:26:02]  Bruce: I'm going to actually skip forward a little of my notes here to the two interestingness, because we've kind of brought that up several times. What is interestingness? I mean, obviously, that's very hard to define, right? And it seems very subjective. But like, when you say interestingness, kind of help me understand what it is you have in mind with that.

[00:26:28]  Kenneth: Yeah, so I often raise interestingness because to me, it's the compass that's the alternative to the objective. Like the objective is used as a compass. We're trying to figure out where is our North Star. We look to the objective to figure out what direction to be going. That's what leads to all these benchmarks and measures of progress and so forth. So if you're not following that compass, what's your compass? Well, the interestingness compass is the alternative compass. And it's true, though, the reason that we generally, as a society, don't engage with that explicitly in serious discourse is because we think it's subjective, which I agree, it is subjective. But we don't like subjectivity. Culturally, we're not very happy with subjective concepts, especially in science and technology, like we want objective assurance that we're doing the right thing. But I've argued that we need to embrace subjectivity in order to be creative, in order to innovate, to make discoveries. And so, it's okay that it's subjective, but still that raises the question, what exactly is it? Well, what it is to me, it's the accumulation of all of your experiences and the entire legacy of your evolutionary history, all bottled up into one thing, which is then applied to deciding which direction looks most interesting to you to go. So in other words, it's informed by enormous amounts of information.

[00:27:51]  Kenneth: Astronomical amounts of information are coming to bear on the question of what's interesting, because the only people, or the only things, the only entities in the universe that make this kind of assessment up to very recently are humans. And so humans are the ones who decide what's interesting, and humans have an entire lifetime of experience to come to bear on this. And so what I'm saying is that your lifetime of experience is probably, it doesn't mean that you are able to assess every type of domain in terms of what's interesting. Like if you spent your life training as a physicist, and then somebody wants to know whether a certain recipe, like a cooking recipe, is going to be interesting, you don't necessarily have useful information to go on there. You're not necessarily going to have interesting insights into whether that's actually a good trail of stepping stones or playground to enter. But in the area where you do have a lot of experience, you have enormous amounts of built up intuition. And the thing is that we'd spend enormous amounts of investment in people, in individuals, to build up that kind of intuition over a lifetime. Like if you think about getting a PhD in something, but it's not really just about PhDs. I don't want people to think I'm just talking about higher degrees. It's like becoming an expert in anything, like an expert athlete or an expert chef or whatever it is that you're getting into.

[00:29:09]  Kenneth: You spend a lifetime, several decades, becoming at the cutting edge of your field, usually, there's a lot of public investment that went into that, at least with the public education and often beyond that, with like graduate school and things like that, grants that were given, whatever it was that went into creating that mind that you have after all that experience. Then suddenly, at the end of that, you could be 30 years old, we say, let's just shut it all off and just make it illegal to use all those intuitions. Because the only thing we care about now is whether there's a performance curve that's going up, which even a kindergartner could look at to see if the line is going up and to the right. That's the optimization, that's the objective following. Of course, we should be admitting people's intuitive understanding of what's interesting after building up all of this information over decades of experience in order to decide what will be an interesting experiment to perform, what will be an interesting hypothesis to pursue. That's what interestingness is about, or even what's an interesting art form, or what's an interesting picture, whatever it is that you're in. The accumulated experience of decades provides enormous substrate to make really interesting subjective decisions. One last caveat to that is, we should be able to discuss why something's interesting. One reason people worry about this argument is they feel like it's opaque.

[00:30:30]  Kenneth: It's like you could just come to me and say, I have decades of experience thinking about computers, and so you should give me a million dollars to do this. That would be a crazy argument. Again, I'm not obligated to give you a million dollars just because you're such and such older, been working for so many years. You should be able to explain why it's interesting. That's still something we should be able to do. That's what I think is missing from the discourse, is that we don't allow ourselves to discuss why things are interesting because it's subjective and we're afraid of subjectivity because we need an objective security blanket. But remember, it's just a security blanket because it's deceptive. So it's no more of an assurance that you're going to achieve what you want to achieve in life because you're pursuing an objective, then not pursuing an objective and just following the path of the interesting. And so we should be able to have a really complex, grounded discussion about what's interesting. Assuming that we're peers in an area, like a computer scientist and a chef may not be able to have a great discussion about this. But a computer scientist and computer scientist could discuss what's interesting in a productive way. And that means really, what does this playground open up? Look, I want to do this thing.

[00:31:39]  Unidentified speaker: This is the project that I think is interesting.

[00:31:41]  Kenneth: I don't know where it's going, because I'm not going through the objectives. But what I can tell you is, it's a whole new frontier. And here's why, because these things have never been looked at before. The phenomena underneath these, once we look at those, will change our perspective undoubtedly, even though I can't tell you how. And things like that should be able to be made explicit, so that we could have a genuine subjective conversation about why you have the beliefs that you have about what's interesting. And by the way, I just want to add one other thing. I said that up to now, it's been humans who make this judgment. We're just getting to the point where maybe computers will start to have something to say about what's interesting. So that gets kind of interesting too. Can computers actually be a participant in this process of deciding where to invest interestingness and research? But it's sort of like at the edge of possible right now.

[00:32:30]  Unidentified speaker: All right.

[00:32:30]  Unidentified speaker: Thank you.

[00:32:32]  Bruce: So now, when you took a look at the grand challenge of the problem of open-endedness and started, which seems to be related to what had happened with your Picbreeder and the fact that you had discovered this idea that having objectives could be deceptive, it seems like you tried to come up with, what would a search look like if it didn't have an objective? And what came out of that was what you call novelty search. Can you maybe describe what novelty search is and how you came up with it and how it works?

[00:33:10]  Unidentified speaker: Yeah.

[00:33:11]  Kenneth: So, yeah, it helps maybe to give the background of how we came up with this idea. It does go back to that Picbreeder experiment that you just alluded to, where we were allowing people to breed pictures, which is, for a lot of people, it sounds strange, but so maybe to make it a little more concrete, if you see a number of images on your screen, you could pick the one that you find interesting and it would have children. And if that sounds strange, it's just like if you were breeding horses or dogs or something. It's just that on a computer, we can artificially cause the image to have offspring. So just like if a horse had children, if this image has children, they resemble the parent, but they're not exactly the same. And so you could breed over generations and people bred to amazing ends. Like they found things like butterflies and cars and planets and all kinds of artifacts of the real world that look familiar in a system that knew nothing about any of this stuff. It was purely through breeding. And so what was interesting though, is we discovered in that system that when people made these discoveries, which by the way were hard, like they were hard in the sense that these were extremely rare needles in a haystack, because like the vast majority, like 99.999% of the space of the Picbreeder is just total garbage, just like wallpaper patterns and static and things that mean nothing. And so people were finding these needles in a haystack consistently, and we looked into how we're interested, like how are they making these discoveries?

[00:34:38]  Kenneth: This is related to open-endedness actually in some way, like how people actually make discoveries in an open-ended search process. And we discovered something that to me was totally astonishing, shocking really, which is that people were only making the discoveries when they were not looking for what they ultimately found. So in other words, the only way to find something in Picbreeder is to not be looking for it. And it's totally paradoxical and totally counterintuitive. And up until that moment in my life, I believed in objectives too. So I just noticed right away that this contradicted that fundamental belief. I was trying to reconcile this with what I had been taught. Because I [unclear] engineering and computer science background. I've been taught you set objectives and then you strive towards them. And there's different ways. There's top-down design and bottom-up design and optimization and gradient descent. There's lots of ways to talk about getting to an objective. But that's how you do stuff. But I was looking at Picbreeder and it was clear that is not how you do stuff here. And I was trying to figure out what's the lesson to draw from this contradiction. I was totally obsessed with this. I mean, this is where the beginning of all the things we're talking about was, was this discovery inside of the Picbreeder system. And I was sure that there's something really fundamental going on here that I needed to understand. And I eventually started to understand the principles that we've been talking about.

[00:35:59]  Kenneth: But the natural thing for me to do, being a computer scientist, was not to jump into social critique, that came way later, but to just try to think of an algorithmic explanation or something that could distill the insight into an automated search algorithm, like something in AI or machine learning. Because that's basically my job. That was basically why we're doing the research. I want to understand how algorithms that discover interesting things should work. And so the lesson that I drew here is that it's actually sometimes better not to have an objective in order to make certain kinds of discoveries. And so what I was trying to think of was how can you mechanize that insight, like turn it into an algorithm, and take humans out of the loop? And so it's very hard to mechanize interestingness because it has never been formalized successfully, because it's, like I said, it's as complex as the human mind. There are some theories about interestingness, like Schmidhuber would cite himself here, and his whole website's on this. But while I think they make a dent in the question, like theoretically what interestingness is, I think interestingness is the accumulation of all of knowledge over all of our lifetimes. So it can't just be formalized simply into an equation. It's something massive. It's more like a large language model set of information than it is like a small formula. So to me at the time, I was thinking I need a proxy that's relatively simple, because I'll never be able to actually formalize interestingness. That's why I appealed to novelty.

[00:37:26]  Kenneth: That's where the name novelty search comes from. Because novelty is close to interestingness, close enough to be a pretty good proxy to see what something like that would do algorithmically. In the sense that not everything that's novel is interesting, but I would assert that everything that's interesting is novel. Novelty is a pretty good proxy. As long as I could define novelty in some domain, then I could say follow the gradient of novelty, instead of following an objective gradient. In other words, you have no objective, we don't even know where we're going, but we try to do novel things. Then the hypothesis is if I cause this machine learning algorithm to keep on trying to do novel things, it will make discoveries that are truly useful and interesting. I think this is not obvious to most people. If you just say, well, it doesn't have any objective, it doesn't even know what it's trying to do, and I'll just tell it to keep doing novel things, then something interesting will happen. Most people initially are like, that's crazy. What kind of algorithm is that? A lot of people use the word random, but like I argued, that is not random. That's why your intuitions are wrong. Because if it was random, you would be right to think it would do nothing.

[00:38:28]  Unidentified speaker: But novelty is based on information.

[00:38:30]  Kenneth: It's a very information-rich operation to move along the gradient of novelty. And so you shouldn't predict that what it will do is random.

[00:38:37]  Unidentified speaker: The question is, what should you predict?

[00:38:38]  Kenneth: And what I would predict is it will do interesting things. It will accumulate information. Eventually, it will do something interesting and possibly solve a problem that you have. But you shouldn't think about it that way because that would be an objective way of thinking about novelty, which is not what it's about. But it does solve some problems. It's just we don't know what problems it's going to solve. So we tried this algorithm.

[00:38:57]  Unidentified speaker: We built it.

[00:38:58]  Kenneth: We tried it out in some experimental domains. We tried a biped robot. We just said, just keep doing new things.

[00:39:04]  Unidentified speaker: And it learned how to walk.

[00:39:06]  Kenneth: It learned how to walk better than if we tried to optimize walking, which is a paradoxical result that is very confusing often to people.

[00:39:13]  Unidentified speaker: But again, it just goes back to deception.

[00:39:15]  Kenneth: The stepping stones to walking don't necessarily resemble walking. So if you're trying to optimize walking, you might actually miss better stepping stones that initially don't look like walking, like oscillation, for example. And so we saw result after result, like the results accumulated over years, not just from our lab, but many. It started to be clear that there is a real phenomenon here. And eventually the interpretation of that phenomenon led to the book and all of the kind of social implications that come out of this as well.

[00:39:43]  Peter: I was thinking of when you say interesting, like how I kind of interpret it is, would you say it's synonymous with, well, David Deutsch has this idea called the fun criterion, where if you're choosing between two theories, there's more to it than this. But if you're choosing between two theories, you kind of go with what's fun. Or there's the concept of meaningful. I'm thinking of like Viktor Frankl, is that the guy's name who wrote the book, Man's Search for Meaning, where you should, it's man's unique place in nature really, that we pursue what is meaningful or joyous. Or there's all kinds of words you could kind of—related words you could use there. Are these all sort of interwoven concepts to you? Meaning and joy and fun and interesting or is that fair?

[00:40:33]  Kenneth: They're certainly related. I mean, they're certainly related. So I think it's fair, but it's important also to point out that the beauty of the idea of interestingness is that none of us agree on what's interesting. Every individual person has a different definition, if you could write it down, which you can't, of what interesting is. What you find interesting is not what I find interesting. There may be some overlap, but it's not a complete match. So what that means is that we get more diversity. You're going to pursue things I wouldn't pursue, and I'll pursue things you wouldn't pursue. The beauty of it is that I might find a stepping stone that ends up instrumental to you because of that, which you would never have uncovered because you're not interested in it. But it's a good thing I found it interesting. And so this is one of the fuels of open-ended systems, is the fact that we don't all agree on what's interesting. It allows us to get divergence and to collect a diversity of stepping stones, which means that the next generation of discoveries is even more diverse, and we can get to even more places from that. A collection of stepping stones is the power of open-ended. It's like the more jumping off points, you have the more places you can get. Concrete examples of this is like in Picbreeder, we observed clearly that people had accounts where they could save their discoveries. We could see the kinds of discoveries people have. People have vastly different kinds of stuff that they're discovering, depending on their personality.

[00:41:57]  Kenneth: There was one woman whose whole account was full of bugs. It was like spiders and beetles. My account had lots of faces. These are often things you don't know about your own self. She asked me, because she was just one of my students, and she asked me, is Picbreeder designed to produce bugs? That's a really funny question. It's not designed to produce bugs. It doesn't know anything about bugs. It was that she didn't know that about her own self, that this is something that for some reason has valence, resonates to her. She's attracted to bug-like forms. It's completely her fault. So it's a mirror showing something back to her about herself. I apparently like faces. I didn't necessarily know that about myself, but we're different. We're different in the kinds of things we pursue, and it's a good thing because now Picbreeder has both bugs and faces on it. It's not something we need to get a universal, holistic definition out of, but we do want to understand some of the underlying factors. It's not like I would dismiss your point because I think the idea of beauty or fun, like those hide behind the diversity, like of course, like those are factors in it, and they are worth trying to disentangle and further scrutinize. I mean, I think it's true because I mean, from an algorithmic perspective, eventually we'd like to understand, like what does it mean to have fun? Like because we want eventually these models, artificial models to participate in this kind of process. And so to understand better where all of that originates is worth thinking about.

[00:43:32]  Kenneth: But it's just that it's not going to be the same for any two people.

[00:43:36]  Bruce: So can you go into some detail on how novelty search actually works? Like at a high level, your book explains it that you've got the virtual robot and maybe it initially just falls over and kicks its legs. So like I can understand it at a certain level. I struggled to understand how you would actually implement the concept of detecting novelty in a program like this. And I'm very curious how, and it almost seems like you would have to have a concept of what kind of novelty you were interested in. And I don't see how you'd get around that or if you did, that would be something that I just haven't been able to think of.

[00:44:17]  Unidentified speaker: Yeah, yeah.

[00:44:17]  Kenneth: So there's a long history of thinking about this question. And it's true that the most immediate thing and the thing that we did initially is to leave it on the shoulders of the human to decide how to measure novelty in a particular domain. So that means it is domain specific, at least as a first stab. And so you have to think about, like you said, what you're interested in. And so, yeah, there's a little bit of the magic is coming from the human. You could say it's almost like it's like a little little cheat there, like the human has to decide what are the features that are interesting here.

[00:44:54]  Unidentified speaker: I don't actually think it's cheating.

[00:44:55]  Kenneth: I think it's actually if this leads to discovering things that are really useful, interesting or insightful for us, what does it matter whether a human was involved or not? Like the important thing is that it did something we otherwise couldn't do. So but it's true though, that like it would be at first, early days of knowledge search is a domain specific question [wording uncertain]. So like very early novelty search experiments were robots and mazes. It was sort of like a nice canonical platform for just demonstrating it conceptually because you could see all of the paths that were explored through the robot that was running novelty search. So it's very easy to see how it's working and get an intuitive understanding of what's going on. So a robot in a maze, it makes sense then for novelty to be measured with respect to where the robot ends up in the maze or the trajectory. So it's like a sequence of x, y positions. It could be the novelty measure or the novelty. We called it actually the behavior characterization. That's what we called this vector of information, which is how do you describe the behavior. Then you would compare one behavior characterization to another. So it's like one trajectory to another you could compare. Or really what it would be is you're comparing a trajectory to a whole archive of prior trajectories to see if it's novel. That would make sense there. For the walking robot, it's something like the x, y positions of where the feet hit and the times where the feet were hitting the ground. I mean, there are other things you could do.

[00:46:16]  Kenneth: It's subjective, of course. Ultimately, what should we care about? You could imagine if the robot in the maze had a light on its head, we could also measure the pattern of flashing of the light. But the problem is there that it's probably not interesting because it's completely orthogonal to the trajectory through the maze. Presumably, the reason we have a maze is because we care where it goes, and so the light is irrelevant and a human would know this. And so it's true that human intelligence is part of deciding what we care about and what's interesting to us here, and that was at play. Now, over time, as the field progressed and people came up with ideas, it was an obvious thing to think, is there such a thing as a generic measure? Is there a way to take the human out of the loop? Of course, people were thinking about this, and people tried information theoretic measures of things to try to do. One thing you could say is like, let's just look at the inputs and the outputs of the neural network, and just forget about what's going on in the outside world, and just track those patterns in some way, characterize those patterns, and just see if we get new patterns. We don't have no idea what's going on in the domain. This could be a generically applied to any neural network that's running a novelty search. Those show some degree of success. But what I would say is, as is usual in machine learning, any completely generic approach, it tends to do better across the board, but worse on any specific case.

[00:47:40]  Kenneth: It's like if I really wanted to work the best I could possibly do in this domain, probably better to use human insight than the generic measure. But if I can't use human insight or something like that, then the generic measure is the second best option. And maybe we can get some progress through measuring it generically.

[00:47:57]  Unidentified speaker: Okay.

[00:47:58]  Bruce: So what type of limits did you bump into with novelty search? It seems like I recall reading somewhere that eventually it led to the robot coming up with clever new ways to walk down a hallway that maybe to a human didn't seem as interesting.

[00:48:14]  Unidentified speaker: Yeah.

[00:48:14]  Kenneth: So there are limitations to novelty search. Novelty search to me is more of a pedagogical point than a utilitarian algorithm. Like it's not actually clear what it's for exactly. Now there are successors to novelty search that might be more useful as a practical matter. Like there's a whole field that now is called quality diversity algorithms, which grew out of novelty search, which I think are more practically oriented. But the original novelty search algorithm is making a philosophical point almost. So it's not necessarily trying to do something that's useful, although occasionally it does something useful. But what I mean is that it's not objectively oriented at all. It's as far in the other direction as you could possibly go. So you aren't telling it. You have no mechanism to tell it what you want it to do. So you have to take that out of your head. And you also have no guarantees that it will do anything useful at all, because it's just being asked to do new things. And so, of course, sometimes, if you call it a pathology, the pathology is that it will go off into the vast space of insanity that exists in the domain that nobody cares about.

[00:49:22]  Unidentified speaker: That can happen.

[00:49:23]  Kenneth: Now, the more simple the domain, the less likely that is to happen. In other words, that's why you do see sometimes interesting results out of novelty search like learning how to walk, because there just aren't that many things to do in that domain. And so, it eventually finds the pockets of possibility that are interesting there. But in a vast, vast space, it's not necessarily going to do something that's useful for you. But it makes a point, like I said, a conceptual philosophical point, because what I think is really, really salient about novelty search is, it's just embarrassing and crazy that this algorithm that doesn't know what it's trying to do can optimize a better walker, I shouldn't even use the word optimize, because that's the wrong word. It can discover a better walker than an algorithm that's trying to explicitly optimize walking. Like that should be an absolute embarrassment to every objective-based algorithm on earth. And it should raise huge red flags all over the place. It's a crazy result. And so, some people take the wrong lesson from that and say, Oh, well, what he's saying here is, I should just use novelty search for everything. Like that's actually the best algorithm in the world. That's obviously not correct. Novelty search doesn't provide any guarantee you're going to solve anything whatsoever. The lesson here is not that. The lesson is that there's something wrong with objectives. Like this should not be happening.

[00:50:46]  Kenneth: And the fact that it's happening raises a lot of questions and should change our understanding of what it means to set an objective and to optimize and what we're really asking an algorithm to do.

[00:50:56]  Unidentified speaker: Okay.

[00:50:57]  Bruce: You mentioned something as a successor to novelty search. Quality something.

[00:51:01]  Unidentified speaker: Can you say that again?

[00:51:04]  Kenneth: Quality diversity algorithms.

[00:51:06]  Bruce: Quality diversity algorithms.

[00:51:07]  Unidentified speaker: Okay.

[00:51:07]  Bruce: I'm going to have to look that up after the show.

[00:51:09]  Unidentified speaker: That sounds actually very interesting.

[00:51:12]  Bruce: So that's actually maybe related to what my next question was going to be. Novelty search seems very clever to me and I can see how it's kind of a proxy for interestingness and therefore kind of mimics what a human is doing when we have this kind of open-ended search of ideas and we're using our own interestingness as a way of creating the open-endedness. It's less clear to me how that would relate to biological evolution, because biological evolution, I would assume, doesn't have, maybe I'm wrong, but I would assume it doesn't have any sort of novelty search like structure to it. Have you given any thought to that as to, and you meant this quality diversity algorithm, does that show maybe more potential to matching up to what biological evolution does or have you given any thought to how novelty search does or does not relate to biological evolution and why biological evolution is able to do open-endedness without having interestingness as a guide?

[00:52:18]  Kenneth: Yeah, certainly I've thought about it a lot. I mean, I was originally entered into the field mainly interested in evolutionary algorithm. I was inspired by evolution. I was trying to understand really how evolution in nature leads to astronomical complexity. That was one of my original motivations for a lot of the work that I did. Going back before novelty search, like all the way back to the NEAT algorithm, or NeuroEvolution of Augmenting Topologies and things that succeeded that. So I was thinking about evolution the whole way through. My view is that the issue with evolution, like I'm not really—I don't—I'm not necessarily interested in, like, modeling what actually evolution does. So it's true that novelty search is not what evolution does in some one-to-one mapping. The reason is because I'm actually trying to understand is what in evolution actually accounts for the interesting things it does. It's that when you observe a natural phenomenon, there are some properties that it has which are undeniably factually there, but that actually are not the fundamental explanation for the part of it that really is captivating you. There are examples of this in embryogeny too. I also did a lot of work in artificial embryogeny and trying to find, what are the actual things that lead to embryogenesis being such a good way of representing organisms. You can look at the natural phenomenon and just say, well, I can tell you what happens. It starts out as a single cell and then it's splitting and splitting and the genes are interacting, the gene regulatory network and blah, blah, blah.

[00:53:51]  Unidentified speaker: But that's not the question.

[00:53:52]  Kenneth: I'm not asking what happens because that is what happens. I'm asking what's the deeper level of abstraction, or I should say the higher level of abstraction that actually explains the interesting part of the phenomenon. When it comes to evolution, to me, the interesting part of the phenomenon is the open-endedness, not the optimization. I don't deny that evolution does perform an optimization function in some pockets. There is optimization going on there, but I don't find that interesting. I think of that as the uninteresting part of the problem. In some way, if you hear the way David Chalmers talked about consciousness, like he said, there's the hard problem and there's the easy problem. I think that's true of a lot of subjects. In terms of evolution, the easy problem is to explain the optimization process. The hard problem is to explain the open-endedness. Most textbooks and biological explanations focus on the easy problem. It's like, yeah, selection and competition and all this stuff. It's all convergent types of processes.

[00:54:49]  Unidentified speaker: Competition is a convergent process.

[00:54:50]  Unidentified speaker: It's a death match.

[00:54:51]  Kenneth: It leads to a winner. The thing that's really mysterious and intriguing is the divergent process.

[00:54:59]  Unidentified speaker: What explains divergence?

[00:55:00]  Kenneth: Competition does not explain divergence. What does explain divergence? Like explicit divergence. How much could you get if you just extracted that one property from the process? It's divergent nature. That I would call more like an ablation study. Let's cut all the other chaff out of evolution, which most people would think is interesting, but I don't. All of these other things, like trying to compete with respect to fitness and things like that, and just distill it down to a fundamental essence, which is a high, high level of abstraction of the thing I'm interested in, which is the divergent property, and see what it does.

[00:55:35]  Unidentified speaker: What are you left with?

[00:55:37]  Kenneth: I think that's what novelty search is. It's a distillation to a very high level of abstraction of just the divergent property and only the divergent property. It's like taking the divergent property of evolution and putting the foot down on the accelerator. Because it's true, like you say, the actual novelty searching aspect of evolution is not explicitly like novelty search. In other words, there's not as many explicit mechanisms pushing towards novelty. There's not zero, though, should note. There is like mate-centered novelty-based selection. Sometimes mates do have preference for novelty. There's also niche founding, where it can be actually useful to be different because you get separated off from everybody else that you otherwise would have had to compete with. There's actually is an incentive for novelty in some cases. It's not like there's no novelty in there, but it's a big complicated mess of things that are all interacting. It's not like just been distilled into this one thing, the way novelty search is. Novelty search is, let's take that metaphor, let's abstract it out, let's just go all with that, all in, all exit in one basket, and just put the foot on the accelerator and just push it towards divergence in a way that evolution doesn't have an explicit push, and just see what happens then if we just focus on this one property.

[00:56:52]  Kenneth: What we find is, I think it is very surprising that it has a lot of the properties of evolution without what, even though it's missing, I mean, in terms of the outcomes, that's what I mean, a lot of the outcome properties of evolution, but it's missing all of the mechanisms, except for this one aspect of like a divergence. That's the way I think it's highly illuminating about understanding what's interesting about evolution, even though it's such a high level abstraction, that it's not at all mechanistically the same thing.

[00:57:18]  Unidentified speaker: Interesting.

[00:57:19]  Bruce: By the way, one of my favorite quotes from the Problem of Open-Endedness article that I've mentioned a couple of times is, It is now becoming clear to us that open-endedness, while perhaps simple, involves a kind of mind trick that would force us to re-examine all our assumptions about evolution. The whole story about selection, survival, fitness, competition, adaption, it's all very compelling and illuminating for analysis, but it's a poor fit for synthesis. Doesn't tell us how to actually write the process as a working open-ended algorithm. To pinpoint the reason we see open-endedness in nature and hence, become able to write an algorithm with analogous power, likely requires a radically different evolutionary narrative than we're used to, which is similar to what you were just discussing right now. Now, I know that this is not your wheelhouse. After reading your article, I became interested in the fact that there are some biologists that are trying to rewrite neo-Darwinian evolution to have a different story. And I don't necessarily see any strong connections between your theories and what they're doing. But Ray and Dennis Noble come to mind, as scientists that have kind of tried to push towards a rewrite of evolution to have a different story as to how it works. I confess that when I read their books, I don't always understand what they're getting at. But like I've also read James Shapiro's book, and he has pushed for something called natural genetic engineering. He has this idea that evolution doesn't really work by just natural selection.

[00:59:00]  Bruce: That is a part of it, obviously. But that the genes actually have sensors, they actually have knowledge of how to take mutations and move them in the genome and engineer what they want to have happen. Do you have any familiarity with any of these theories? Do any of them seem like they have any overlap to where you're going, or does this just seem like it's totally unrelated to what you're working on?

[00:59:23]  Kenneth: Yeah, I have some passing familiarity, but nothing intimate. It's like I just want to give that disclaimer at the front. I'm not an expert on these theories. For those authors, I don't want to be pretending to have an expert critique here. But I understand the gist of what this is about. And from the gist of it, I don't think it's on the same page with me. And it's not just in a superficial way. I mean, it's deeply different, I think, in terms of what they're exploring. Because they're looking at mechanisms that, I would probably say, aren't the most interesting. But they're actually trying to show that there are interesting ways that these mechanisms occur. So I don't mean to suggest that it's not interesting research. Obviously, someone should be looking at these questions, and it actually is worth understanding that there's more there, maybe, than meets the eye. But what I mean is, from a macro-level perspective, the whole evolutionary process, what interests me is, like I said, divergence or open-endedness, explaining how it can diverge over eons. I mean, we're talking about over more than a billion years, it's almost like forever. To create all of living nature in a single run, which is like literally biblical, like it's all of creation right there. So I don't think what they're looking at is part of the explanation of that. One reason for that is because these are actually optimization-centric questions that they're addressing, or you could think of them objective questions.

[01:00:56]  Kenneth: It's like any mechanism that allows you to improve your performance with respect to some metric at the current time is a form of optimization, and optimization processes lead to convergence, not divergence. It's like if I can more quickly converge my species towards what is the locally optimal behavior for survival, well, then the species will converge. It's just like the opposite of explaining divergence. To the extent that the mechanisms are available, you would get dramatically less diversity, fewer stepping stones, fewer species over time, and evolution would be less spectacular. And so I think it's still important to get to the facts of the matter in terms of how these things work, because there's more, I mean, I'm not claiming that what I'm interested in is the only thing we should care about. I'm just saying this is what I'm interested in. We should also want to know how things actually work at a mechanistic level, and they're doing that, and I'm not, so I'm hardly arguing that we should stop that kind of research. But I'm saying, from the purposes of trying to understand open-endedness and the creation of all living nature in a single run, this is probably not going to end up being extremely illuminating. And it sort of relates to the point I made about the fact that there's so many seemingly salient phenomena that are very hard to take your eye off of when you look at a natural phenomenon like the unfolding of an embryo, that it can be really hard to sort out what really matters here for understanding at a fundamental level what's going on.

[01:02:28]  Kenneth: Because the magic of just the unfolding process itself is just so salient, it's hard to take your eye off of it. And so I think this is the case where it's like seeing all of these mechanisms is just absolutely magical, to see that it could actually engineer its own genome. It's an insanely incredible process, but it can distract you. It can distract you from seeing something that's even deeper and more fundamental in terms of explaining how something like what happened on this planet is possible.

[01:02:55]  Bruce: Okay, let's talk about practical applications for your research, particularly how it relates to artificial general intelligence. So you had some, you had some suggestions in your book about how maybe funding of academia is too objective oriented. And then you made this really interesting point, which I'm sure I'm not going to be able to do justice to, that artificial intelligence is in some sense the search for search algorithms and that we don't even apply what we know about search algorithms in the search for search algorithms. Maybe I'm saying this poorly, so please correct me if I'm doing a poor job of representing your views here.

[01:03:43]  Kenneth: No, no, I think that's a good statement.

[01:03:49]  Kenneth: There are clearly practical implications. Now, we're talking at the level of institutions in some sense. I think there are practical applications of these insights because it's true that we run almost all our institutions through objective means. I would claim that as a result, we have a lot of unintended consequences that are a result of the objective paradox. In other words, we are asking for deception. We don't want to be asking for it, but in effect, that's what we're doing by running everything in terms of objectives. There's a lot of sociological analysis you could do to understand why are we like this? Why did we get to a point where every single thing that we do has to be objectively justified? Whatever the reasons are, and they're fun to talk about too, but whatever they are, what the analysis in the book shows is that they shouldn't be. Not everything should be objectively driven, especially when it comes to the pockets of our institutions and our personal lives, where what we're actually after is innovation, or discovery, or creativity. In those pockets, this is actually really hurting us. Those pockets are very, very important because that's how progress happens. It happens through innovation. And so there's a lot of institutions that have been set up explicitly because they're supposed to facilitate innovation, because like all the problems in the world that have not been solved will only be solved through innovation. And so there's an enormous weight on these institutions that are trying to do this.

[01:05:23]  Kenneth: And so if they're making a mistake because of the principle of the objective paradox, because of the fact that when you pursue an objective that's too far away, you end up deceiving yourself and blinding yourself to other stepping stones that might lead to other important discoveries, as well as stopping yourself from making the discovery you're trying to made, then we are slowing down all of innovation across society inadvertently, unintentionally. But obviously that's bad. We don't want to be doing that. And so that leads to some kind of prescriptive solutions potentially that are based on the ideas like the insights in novelty search and the whole theory behind these kinds of algorithms, these open-ended algorithms to the extent that we've worked this out, about better ways that these kinds of institutions could be run. And I'm thinking of like granting agencies, like innovation labs and research labs inside of corporations, like investors, especially venture investors, thinking about the next big thing. Like the government attempting to solve big, big problems in society and in the world, trying to cure diseases, environmental problems. There's all kinds of extremely ambitious objectives littered across society that are subject to the objective paradox. So it's something to be worried about because we approach these things objectively. And so the example you gave of artificial intelligence is one. Artificial intelligence is obviously a very ambitious objective, or really you could call it AGI these days.

[01:06:49]  Unidentified speaker: It's the way we articulate it.

[01:06:50]  Kenneth: Artificial general intelligence, it's like the objective of AI is AGI, you could say. Well, as soon as we really, if we take that to heart and really pursue things that way, then yes, you start to see all these pathologies coming out that come from pursuing deceptive objectives. And you see it really explicitly in this field. Maybe part of the reason is because I'm in the field, so it's an easy example for me to refer to because I've been watching it. But it's very objective how it works because we have benchmarks, and benchmarks are basically a way of making things objective. And people are really, really worried about knowing whether we're really making progress. That's what objectives mean. So everywhere I've been in academia and industry, everywhere I've been, there's a constant drumbeat of like, we need better benchmarks, we have a way of knowing. Is that really better than that? Get me more granularity and information here so I can know whether this is better than this. And so it's just an obsessive preoccupation.

[01:07:47]  Peter: I'm a high school teacher and it's every day. You wouldn't believe it anyway. I'm sorry, I just had to interject that.

[01:07:53]  Kenneth: Education is another place that this is just absolutely pathological. I mean, the entire education system is saturated in objectives. It's like how much funding is going to go to which school. It's like there's some objective measure of how the school is performing, there's objective measures of how the teachers are performing, and there's the objective measures of how the kids are performing on the test.

[01:08:13]  Unidentified speaker: You could say, well, how could all this be bad?

[01:08:14]  Kenneth: Like obviously you have to do these measurements.

[01:08:16]  Unidentified speaker: But just think about what I'm saying.

[01:08:17]  Kenneth: If our goal was to get every student to get 100 on every test across the country, then it's obviously deceptive.

[01:08:24]  Unidentified speaker: It's going to be extremely complex to do that, which means that sometimes we're going to have to explore something that might, horrible as it is, cause test scores to go down a little before they go up, and we are not allowed to do that.

[01:08:38]  Kenneth: So it's preventing us to do any kind of exploration at all. This is across all of the problems of society, and education is a big one. The thing in AI that we have trouble with is to just say, actually, this thing should be published because it's interesting. It's a new stepping stone, and I would argue that a lot of the major innovations in AI are because somebody did actually buck the trend and just decide to invest in something because it was interesting, like somebody had the courage to do that. Obviously, it happens, but it happens despite the way that we've structured things, not because of it. That's why I'm saying we could improve the structure of all of these institutions, including academic communities, to take this objective paradox seriously and make things, including especially, I could even say especially education work better, because I think education is just a crying shame, because there's real victims of the way that this stuff works. It's disgusting, like what it's done to kids. And nobody wants to ignore, like, you know— Actually, I've talked to so many diverse communities about this because of the book, but the one that I found the hardest to get access to is the educational community.

[01:09:47]  Unidentified speaker: They just don't want to hear about this.

[01:09:48]  Kenneth: They love their standardized tests. Other communities are more receptive. At least they like hearing something contrarian. But the educational, they don't even want to talk to me.

[01:09:57]  Unidentified speaker: There's exceptions.

[01:09:58]  Kenneth: I spoke to a superintendent in Kansas once. This guy was really interested in this stuff, but he's like an exception. It's just kind of interesting that it's so entrenched in that particular culture.

[01:10:11]  Bruce: So actually, you've mentioned that a couple of times now, this idea that we have this psychological need for objectives. Explain that a little bit further. Why do you think we have such an almost pathological need for objectives? How did that come to be? Is that something natural, part of human nature, if you will, or is that something cultural? I mean, obviously, it's probably, you know, anytime you do genetics versus nature, you know, nurture versus nature, it's almost always a false dichotomy. But is it something that's more cultural, that has come up because something in American culture or whatever has led to this? Or is this something that like is just a psychological pull that all humans have that just kind of make us want to go towards objectives?

[01:10:58]  Kenneth: Yeah, this is a really intriguing question. I like to think about that. Why are things the way that they are?

[01:11:04]  Unidentified speaker: And it is hard to disentangle.

[01:11:05]  Kenneth: I mean, you identified the right part of it because it's hard to disentangle the cultural from the psychological, the intrinsic psychological needs that we have. So my guess is a bit of both, that from the psychological side, you've got a desire for certainty, which I think is deep seeded. We just want to feel confident and sure that things are going to be okay in the future. Objectives provide what I think is a false security blanket for creating a veneer of certainty. It makes you feel like you can feel like you're doing the right thing, like you're confident, you know you're on the right path. That's something we really crave and want. It's hard to create or simulate that without an objective. Following the path of novelty or the path of interestingness does not create any certainty at all.

[01:11:51]  Unidentified speaker: It's about risk actually.

[01:11:52]  Kenneth: That's an important point to note. Like when I advocate following interesting paths, I'm actually advocating taking risks. Because there's no guarantee that because you follow an interesting path, you actually get a reward at the end.

[01:12:04]  Unidentified speaker: That's the whole point.

[01:12:05]  Unidentified speaker: We don't know where we're going.

[01:12:06]  Kenneth: A lot of these paths will lead to dead ends. But if you follow enough interesting paths, you will find something interesting. You just don't know what it is. But so there's a lot less certainty in that kind of an argument. It's just not as psychologically satisfying for the people who want to feel like they know where they're going. And because of that, it's important to acknowledge that you have no obligation as an individual to take risks. It's not like important or noble.

[01:12:30]  Unidentified speaker: It's just a personal preference.

[01:12:32]  Kenneth: So if you do want certainty in your life, if you don't want risk, you want to minimize risk, by all means follow objectives. What will happen is that you won't achieve anything remarkable. You'll have modest achievements, but you'll live a relatively risk-free life, and it's completely admirable. It's like you'll support your family, you'll go through life without suddenly losing your job and having to figure out what to do next. Like there's nothing wrong with it.

[01:12:55]  Unidentified speaker: It's just a personal decision.

[01:12:56]  Kenneth: I think the only time where you're making a wrong decision on this is if you're just not aware of the risks. Like that's a mistake. Like if you go out on a path and you think, oh, well, he said, do what's interesting, and so I'm going to guarantee to become a billionaire. That's a mistake, and that's actually you should try to avoid. You should be informed about the risks you're taking. But as long as you're okay with the risks, then you should be allowed to take those risks. Like that's something you can do. And institutions and governments have to think about when it's viable to take risks. You know, like research organizations can take risks. Like the National Science Foundation, which is giving out money to do research, is supposed to take risks. So it's okay if some projects fail. And that's why following the interesting should be okay. And I argue it's not very okay in that institution, but it should be. That institution needs to be reorganized. But something like the economy, it's not actually okay to take huge risks. Like you can imagine doing experiments with the economy because they're interesting.

[01:13:48]  Unidentified speaker: That's just off the table.

[01:13:49]  Kenneth: Like we can't afford to have like thousands or millions of starving people just to find out if something actually turns out interesting. So we are satisfied to live in that local optimum and simply optimize objectively because the risks are not worth it. So it's a question of where the risks are worth it. Now, also the social component, that's the part I didn't mention. The social, the component of culture, like how much is this influencing our obsession with objectives?

[01:14:16]  Unidentified speaker: I think it's big.

[01:14:17]  Kenneth: I think it's not just the psychological component. I think if you go back farther in history, you get less objective cultures. Like if you go back really far, you get to these philosophers like Lao Tzu, who I don't even know is a real person, but he was attributed thousands of years ago with saying something like a good traveler has no fixed plans and is not intent on arriving. And you hear these kinds of Confucian or Buddhist or Taoist philosophies, ancient philosophies that sound much more like something that came straight out of our book. That's actually why we use that quote in the book. We found all kinds of quotes that are, we actually put them at the top of chapters if anybody reads the book, we put these quotes because we admit that we're not the first to make this kind of a point. I think what we're the first to do is to try to put it into some kind of empirical framework. We actually have evidence of it. But many have made this kind of philosophical point that it's often better not to know where you're going if you want to get somewhere really great. I mean, we hear things like that, like when people talk about especially love, like the best way to find love is to not be seeking it and things like that. Like this is a direct expression of the philosophy of the objective paradox. So in some corners of our culture, like it still exists and people still talk about it. But in terms of like most of the culture, it has been drained out of the system.

[01:15:38]  Kenneth: I think the reason is because we have become increasingly sophisticated at creating checks and balances inside of institutions to try to prevent anything from going wrong. Like nothing bad should happen anywhere. I mean, it's bureaucracy is what we're talking about. So bureaucracy has become very sophisticated. So we continually want to try to use this sophistication to mitigate risk. The problem is that objectives look like a really good option for mitigating risk. It's like, how do I know that I'm not going to be wasting investment on something? It says I should be able to have an objective measure for it. So we feel like we are becoming increasingly sophisticated at managing risk, and making sure that we're putting money into things that actually are going to pay off. But the unintended consequence of it is that we're killing our abilities to diverge and to do open-ended exploration. It's a worldwide phenomenon. It's not just the US or the West. Eastern cultures have also lost their ability to do this kind of divergent thinking. I mean, if you look at Chinese culture, it is very top-down. It's very goal-oriented, both at the government level and the individual family level. It's like, this is what you need to do to succeed in life, and you do that for your parents. And the reason I mentioned China is because our book recently came out in China. So I actually got to know a lot of people in China because of that. And the book was particularly potent in China. It did very well there because it's such a clash with the culture there. The culture is totally objective in China.

[01:17:09]  Kenneth: And so you can go almost anywhere and see this has happened in the modern world. It's just a calamity across the modern world, in my view, which is why we wrote the book. There needs to be some force pushing in the other direction.

[01:17:22]  Bruce: So let's talk about how this would apply to actually funding research. So I might be reading too much in here from your book, but it seems like you kind of advocated for two kinds of research. Maybe one kind that would be done by companies that would be more objective oriented because a company needs to turn a profit. So they're going to try to move one step towards some sort of goal. And then another kind that maybe was funded by the government, that was more open-ended of just trying to follow interestingness so that we collect stepping stones. Again, my apologies if I'm misquoting or misrepresenting your views here. But I got the sense that you were kind of advocating for having both kinds of research available and doing both because they both have some sort of importance.

[01:18:16]  Unidentified speaker: Yeah.

[01:18:17]  Kenneth: So I don't remember if I exactly said that, but it's certainly points that are related to things I would have said that, you know, because the part that I don't completely mean to convey is that it's necessarily a dichotomy between the corporate world and the government world.

[01:18:34]  Unidentified speaker: Okay.

[01:18:34]  Kenneth: It doesn't have to be, although it does tend to be like you do tend to be, see more objective pursuits in corporations because of the profit motive. And the government through its funding agencies can afford to be less profit oriented. So that's true. But I do think that, like, generally, there's a breakdown in both in terms of, like, being able to do non objective pursuits. Like the government is not very good at actually doing non objective pursuits. By the government, I mean the funding agencies like the National Science Foundation, like those kinds of things or DARPA, for example. There's like different funding agencies in the government. And so these agencies do tend to ultimately force you to put your cards on the table objectively, unfortunately. But the real question here is not what actually is in practice, but what could be in practice, like where should things be happening and how should they be happening? And I think that you could have non objective pursuits in both environments. It's true it's a little more challenging in industry, but it can be justified, especially in large corporations, you know, because they have they have research labs, which are which are basically research institutions, and they need to have them. They're not just having them for fun, because like what large companies are dealing with is the potential for disruption. It's the possibility that somebody is going to come along with some completely unanticipated new technology out of left field, and just make everything obsolete at this big company.

[01:19:58]  Kenneth: And so what is your defense against disruption? It's research. Like you have to like look at the possible future horizons and inventions yourself to anticipate the future world. And so that's why, you know, there's a Google brain or deep mind at Google, or like a Meta has its AI labs. And like these organizations have their big labs doing research often these days in AI, but also like things like Bell Labs, Xerox PARC, like they have these things for good reasons. And so the question though is, can these kinds of corporate labs exist in an objective free environment? When the company itself obviously is very strong objective requirements, like it needs to have a profit. I mean, it's just answering to investors. So it literally is a legal fiduciary responsibility. It can't just be like, oh, we don't care. It does care. And so it's hard for the CEO to look at their innovation lab or their research lab and think, oh, we don't really need objectives here. Like, let's get rid of them or we shouldn't have them because they feel like everything needs to be aligned with the cause of profit. But the thing is that I think that's an incorrect view because the thing is like, these are your shield against disruption. The shield against disruption works differently than every other component of your organization. Because you cannot protect against disruption by following an objective. The only protection against disruption is open-endedness. And the open-ended component of the organization has to actually be open-ended, which means it cannot be ultimately objectively oriented.

[01:21:31]  Kenneth: And this point is extremely difficult to absorb for leadership in companies. It's not quite impossible, I hope, but I don't think it's quite, there are some exceptions actually, I know of some CEOs who try to actually follow this philosophy. But it's extremely difficult. But the thing that you have to recognize is that, if you can't get it all the way up the chain, all the way to the CEO, that they actually believe in this, then it's doomed.

[01:21:56]  Unidentified speaker: Forget it.

[01:21:57]  Kenneth: You can call it whatever you want, you can call it a research lab, you can call it basic research. It ultimately will not actually be that, because people will understand that there's an objective incentive, which is to affect the company's bottom line, and they won't do what they otherwise would have done, which is do what they thought was interesting, because they'll understand that they're going to lose their job. So they're going to do the thing that they think will increase the bottom line, which is objective, and it's not what they think is interesting, and you will destroy all innovation. And most companies work that way. The government, it's easier for them to extricate this from the objective pressure, because they don't have a profit motive. But for some reason, they still can't seem to do it. Because it's like the NSF, which I was very familiar with, because I was a professor begging for the money myself, it forces you to explain the deliverables that you're going to get. Like you actually write these statements at the end of your grants, and that's basically like laying out objectives. Like people wanna know, well, what are we gonna get at the end? Well, what if I don't know what I'm gonna give you? Like I'm just doing this because it's interesting. Like that was actually why I did Picbreeder. I have no idea that this theory would result from doing Picbreeder, but I did know, or at least I thought, that if we do this experiment where people evolve pictures online, we're gonna learn something really interesting about discovery. I don't know what it is though.

[01:23:09]  Unidentified speaker: So I couldn't write it down.

[01:23:10]  Kenneth: And NSF, of course, they rejected the grant application for that reason. Like they said it wasn't clear what the point was.

[01:23:17]  Unidentified speaker: Like what's the point of having all these people evolving pictures on the internet?

[01:23:20]  Kenneth: It's like a completely pointless endeavor. So they failed to see that my argument is not that there is any deliverable. It's just that it's interesting. So we're gonna open up a new playground and discover new things. So they're both not doing well. It's harder to do in industry, but I think it's possible and it's easier to do in the government. But they're still not doing it anyway.

[01:23:39]  Bruce: We're just about out of time, but I have one more question I really wanted to ask you. So you talk about creativity as a kind of search. Here's a quote from your book from page 6. So we can think of creativity as a kind of search. I suspect that idea, equating creativity with search, would surprise many people in and of itself. How literally do you mean that? Is that literally true, or is that more like an analogy in your mind? And I should probably give my own biases here. I actually think it's literally true, so I found that interesting. But I wasn't sure if you intended it as literally true or more as an analogy.

[01:24:17]  Kenneth: I think I mean it literally too, yeah. I say I think because there's some ambiguity about what search even means, like that's something you can get in a split hairs on that, like what is a search? But I basically think you're searching through a space, like that's what you're doing when you're being creative. In other words, there exist, of course it's a philosophical question, what does it mean that they exist? But things exist in the space of images that haven't yet ever been written down or haven't been drawn yet. They in some sense theoretically exist. You're searching for those things in that giant space that actually are interesting, whatever those are, like if you draw a picture or make a piece of art. I think of it as searching through that vast space, which is like uncountably vast.

[01:25:08]  Bruce: That's an interesting idea in and of itself, the idea of search as related in some way to creativity or to intelligence. Although I did notice that the most famous textbook for AI is the Stuart and Russell book, and they basically put search as Chapter 2, because then they're going to reference that all throughout the entire rest of the book. There does seem to be something basic about search and AI that go together.

[01:25:37]  Unidentified speaker: Yeah, that's true.

[01:25:38]  Kenneth: Yeah, I'm trying to think like where or why I'm so comfortable calling everything search. It does go back to something like those textbooks, because I read those textbooks when I was in introduction to AI, and yeah, it sort of frames everything in terms of search, and it starts to be a framework for thinking about how a learning algorithm moves through a space. Once that clicked for me, it actually took a few years, I think, to really click what that means. It made it easier to think about, you can visualize more what's happening. You think of these points in the space, and you can think of the algorithm moving along, this point moving through the space, where you are in the space, and approaching an optimum is like moving towards it in the space. You're searching for the optimum. It's just a metaphor that's helpful, I think, to think about it visually to some extent, and to couch it in something that you can actually grasp on to.

[01:26:40]  Bruce: Yeah, even the very concept of a fitness landscape and things like that, where they try to visualize all the algorithms in terms of some sort of search is kind of interesting.

[01:26:51]  Unidentified speaker: All right.

[01:26:52]  Bruce: Well, we're out of time, but thank you so much. This has been a massive pleasure to be able to meet with you and to be able to ask all my burning questions after reading your book. And I think that our listeners will love what you had to say on the subject here.

[01:27:08]  Peter: And Ken, I just want to say your ideas have lit a fire in my brain. I just it's one of the most interesting concepts I've come across in a long time. And I really appreciate you coming on our show.

[01:27:21]  Kenneth: Well, thank you for that, Peter, and thanks to both of you. This was great. I loved these questions. It's great to get these points out there, which often you don't get to talk about at this level. So, I'm really glad that this is now recorded and down on video, so people can actually hear these ideas.

[01:27:38]  Unidentified speaker: Thank you.

[01:27:39]  Kenneth: Thanks a lot.

[01:27:39]  Unidentified speaker: All right.

[01:27:40]  Unidentified speaker: Thank you.

[01:27:48]  Peter: Hello again. If you've made it this far, please consider giving us a nice rating on whatever platform you use, or even making a financial contribution through the link provided in the show notes. As you probably know, we are a podcast loosely tied together by the Popper Deutsch Theory of Knowledge. We believe David Deutsch's Four Strands tie everything together. So we discuss science, knowledge, computation, politics, art, and especially the search for artificial general intelligence. Also, please consider connecting with Bruce on X at BNielsen01 [handle spelling uncertain]. Also, please consider joining the Facebook group The Many Worlds of David Deutsch, where Bruce and I first started connecting.

[01:28:32]  Unidentified speaker: Thank you.