What do we want from AI?

Or, what I've been up to all year.

Hi all! It’s been almost a year since I’ve posted to this substack. Sorry. My wife is about to have a baby and this is a great forcing function for sending an update, even though I don’t exactly have a thesis right now.

My interests this year diverged quite a bit from “keeping up with AI technical developments” and I was sheepish about posting it here. I kept thinking “it’s cool to use this time to explore weird stuff, but the newsletter people want technical AI development analysis and I’m sure I’ll get back to that soon.” But I kept going in weirder directions!

What I'm going to do is end the paid tier, readjust expectations, and hopefully actually start posting again. Although it may be a slow start when the baby comes. Let me know if you want a refund and I'll try to figure out how to do that (I don't know exactly how stripe works). This post contains a set of stubs that I .

What do we want from AI?

I started off the year helping out with some AI capabilities projects and doing cofounder dating. I figured this would give a lot of posting content. But while I was doing that I started asking, what do we really want from AI? And that sent me down some rabbit holes. The topic that set me off was the Yudkowsky view that AI is about to kill us all.

I find his argument hard to refute it on its own terms. Humans exist and our behavior seems highly underoptimized and is tied to biological bodies. Human agency is an existence proof, this implies that it is physically possible to create something agentic and smart like humans. And most likely it is possible to do it in a way that is more optimized and not constrained by physical bodies. Claims about specific timelines I find less convincing, and the biggest question mark is whether this is actually likely to arise by accident in the course of AI research. Yudkowsky’s level of certainty (>99%) is epistemologically suspicious: At the very least Knightian uncertainty should give him more humility. But more important, it doesn’t work for me as a driving principle of the way I plan to live my life. If you really think humanity is about to be turned into paperclips by an unstoppable superintelligence maybe you should focus on making the best use of what little time we have left instead of frantically bombing data centers.

Nevertheless, the world will massively change in our lifetimes, with AI likely playing a big role. Holden Karnofsky points out that all possible views about humanity’s future are wild:

There’s no continuation of this graph that isn’t wild. Growth continuing as it has is wild. Growth stopping would be wild. And you can’t rule out more apocalyptic scenarios. Because every future outcome is wild, the stakes are high. We live in the most important century. Since that’s the case, how do we build a future we’d actually want to live in?

Here's some of the weird directions I've explored intellectually in an attempt to gain some kind of footing on that question:

Solving Information Overload

Even without AGI, technology and AI 1.0 (e.g. algorithmic feeds) has already create upheaval in our in our information space. For an especially pessimistic take, see David Chapman’s e-book Better Without AI, in particular this chapter. Although it’s easy to blame this on Mooglebook, it’s possible that a lot of trouble is caused by information superabundance itself. David Perell has written about the paradox that arises here: A few infovores like Tyler Cowen live in a paradise of riches and are much better off. But many (most?) become worse off, overwhelmed by the firehose of stimulating meaning fragments. Teens are depressed, QAnon is popular. Even well functioning adults spend way too much time staring at endless feeds on our phones (myself included). We’ve seen the effects of superabundance in the food we eat: Modern agriculture ended shortages of food but also gave rise to fast food and obesity. How can we benefit from what’s good while limiting the bad? One hypothesis might be a cultural movement in favor of “healthy information consumption,” akin to the movement towards healthy food consumption. This would create a market for tools that could help.

One interesting recent development along these lines is Twitter’s community notes. They are not perfect but make an intentional effort to reduce partisanship. Vitalik Buterin writes up some details here. I’d love to see more work developing tools to enable people to make their information diet more healthy.

Meditation

If we want to get the benefits of information technology without losing our minds, do we need to become masters of our own attention? Meditation is like a workout that trains attention. I’ve explored this deeply over the past year. I’ve made sure to meditate at least 30 minutes per day in 2023 (often much more). I took a 10 day silent retreat, I have been taking part in Toby Sola’s digital sangha Brightmind, and I’ve had some really great experiences at the Berkeley Alembic. It has been transformational for me, providing a powerful inner resource. I find it easier to break out of fixated states. But it also continues to feel like a work in progress. People tell me I’ve gotten more loving and affectionate since starting. I still waste time on my phone. I’ve heard things from meditators like, “oh yeah it takes about 1000 hours to really break through.” Is there a way to accelerate this and make it more broadly accessible? Stephen Zerfas and the team at Jhourney are taking an engineering-first approach to solving this.

Predictive Processing

My PhD was in Cognitive Science. Since I left the field in 2013 there have been promising developments about unifying AI and neuroscience stemming from a theory by Karl Friston known as Predictive Processing (Nature Reviews Neuroscience here). Friston claims to provide the general-purpose algorithm brains implement to help their host organisms act agentically. Briefly, brains have an objective function that minimizes something Friston calls free energy. Free energy is the sum of prediction error plus the KL-divergence of feature estimates to their priors. Although it’s a learning algorithm, it explains motor function: one way to minimize free energy is to update the model, another way is to manipulate the world to match expectations. Homeostatic behavior comes for free if the organism has a strong prior for “organism continues to thrive.” Several things turn out to be interesting about this:

1. the theory makes predictions about neuroscience phenomena like priming effects, autism, attention, and how to train a blob of human neurons to play pong.

2. It generates a number of hypotheses as to why meditation might be useful that former DeepMind researcher Shamil Chandaria has been investigating (video here, conversation with Michael Taft here), as well as inspiring Laukkonen and Slagter (2021, here).

3. Taken seriously, it could inspire recipes for creating effective AI agents.

Scott Alexander reviews Andy Clark’s book about this here and makes an attempt to explain free energy in more detail here.

So what?

Stewart Brand’s diagram of Pace Layering, article here.

I started this year focused on accelerating capabilities in AI. As I’ve explored the space, I’ve become more interested in the impact technology and media have on our deep well being. The future can be 10x better than the past has been, but I don’t trust it to happen by default. It will require agency on our part to build a future worth living in. I don’t have the answers yet to how we do that. I believe it will involve combining a deep understanding of the tools we have available with an even deep understanding of what enables human thriving.

What’s Next?

For this Substack: The plan is to lower my threshold and post what’s on my mind without worrying so much if the theme is right. It will be a slow start due to the baby’s imminent arrival. As I said at the beginning, I’m turning off the paid tier, please get in touch if you’d like a refund! But stay tuned if you’re interested in where this is leading.

In my life: We’ll find out! My plan is to get serious about taking on new things after I’ve spent some time adapting to parent life. I anticipate that the baby will massively change my mindset and I want my choices to reflect that. But I am focused on the question of how I can be of service.

Gratitude

I’ve been on sabbatical for over a year now. South Park Commons has been an extremely supportive place to explore widely. Ruchi Sanghvi, Aditya Agarwal, Mitra Lohrasbpour, and many more created an amazing place for technologists to go from –1 to zero. A few founders I’ve met there that you should watch out for:

• Bryan Davis @ Centari

• Alexandra Johnson @ Rubber Ducky Labs

• Ryan Orban @ Cadea

• Avi Fein @ Meebo

• Samantha Whitmore (still in stealth but iterating really fast)

…and not SPC but I learned a lot from Derek Chen about how to actually make AI dialogue work.

I’ve also gotten a lot out of working with the Alembic community: As I mentioned Kathryn Devaney is doing really cool work on the neuroscience of enlightenment, and Stephen Zerfas is building a startup called Jhourney to enable normal people to access deep meditative states known as jhanas. If you want to go take a walk on the wilder side of non-denominational dharma practice, check out Michael Taft. I’d recommend his podcast, or the book he edited, The Science of Enlightenment here.