Jurassic Park is a bad bioengineering parable but a great AI alignment allegory

Jurassic park is a vividly entertaining movie, but few people watch it for the chaos theory.  At most watchers come away with a first-order “bioengineering is dangerous and we shouldn’t play God” message.  Which isn’t completely unintended, but is a very shallow version of the lesson Crichton wrote into the (more thoughtful, but less flashy) book of the same name, which is about inevitable unpredictable failures in complex control systems. 

(the only place this gets airtime in the movie is the scene where Malcom flirts with Elie in the car, rolling water droplets off her hand, wherein the chaos theory is overshadowed by Malcom’s slightly scandalous horndog antics) 

But long story short, the core of the story is a blow-by-blow illustration of the debate between the billionaire Hammond and the mathematician Ian Malcom, and if you abstract away the dinosaur flesh and cut down to the bones, you get this conversation:

Hammond: “We have created a novel complex system of living agents, and we can harness it to do immense good for the world!”

Malcolm: “But you can’t possibly control a system you don’t fully understand!”

Hammond: “Nonsense, we bought state-of-the-art technology, hired the best engineers, and haven’t had any serious accidents!”

Malcolm: “You can’t control this system because you think of these agents as your playthings, but they think of themselves as agents, and their goal is to survive.  Without understanding the full system, it’s hubris to guess how or when it will fail, but I’m telling you it will fail.

(and of course, two hours or two hundred pages later, Hammond admits Malcom is right, the dinosaurs escape, and a lot of people get eaten).

It’s hard to take this seriously as a parable about existential risk, because at the end of the day you can make a T-Rex really scary but it’s hard to shake the feeling that a couple tactical bazookas would bring the T-Rex back into containment (and even in-universe they have to fudge this heavily with an inconvenient hurricane and evacuations, although I suppose this is one of the “unpredictable failure” modes)

But as Matt Yglesias has pointed out recently we actually kind of suck at writing approachable stories about existential risk, and in particular about AI risk, and I want to make the case here that Jurassic Park is actually best viewed as of 2022 as a story about how hard it is to align a superintelligent agent with human utility, and I think there are some concrete parallels to AI alignment, at least as it exists today:

  1. Systems which operate as agents eventually optimize their own objective function, not yours

In the story, “dinosaur safety” researchers built in not one but two failsafes to ensure containment:

  • All dinosaurs were deficient in the amino acid Lysine
  • All the dinosaurs were female

The dinosaurs of course, did not even realize they were supposed to be contained by these failsafes, and responded by (unpredictably) converting from female to male like a frog, and by eating lysine-rich foods.

In the generic version of that conversation above, you just end up describing the problem of AI alignment as is normally framed: AI when built as a tool (ex, translators, image detection, protein folding) is likely safe from dangerous outcomes (to the extent that people don’t use it to say, design novel pathogens), but as soon as you turn that system into an agent with goals, it becomes extremely difficult to keep the agent optimizing for human-oriented goals.  

The classic parallel here is obviously the paperclip maximizer scenario: a friendly agent whose goal is to make as many paperclips as possible (for humans!) , but decides that the most effective way to maximize the paperclip count is to first use all atoms in the observable universe to replicate itself, and consequently converts all humans into paperclip-maximizer motor oil.

The AI fails containment not even through maliciousness or deviance (which is a whole separate problem) but by treating its own failsafe as an obstacle to be overcome; because of course it has no inbuilt reason to respect the spirit of the law, or even a concept of what that means to us.

  1. The system designers honestly tried to be responsible about containment, but stopping (or even slowing down) was not an acceptable outcome

Robert Muldoon was the experienced game warden brought into Jurassic Park by Hammond to ensure the safety of the park.  Muldoon’s advice about how to contain the raptors (paraphrased)?  

Muldoon: “We should kill them all “

Hammond: “lolno, we’re not shutting the park down” 

AI safety is taken seriously by all the big players right now, but similarly has a “yes, and…” mandate.  If the DeepMind alignment team’s conclusion was “we can’t trust that any models with over 10 billion parameters are safe to release in a public-facing product”, Google is going to hire a new safety team.

  1. Early warnings where the agents cross tripwires and cause real harm are probably just going to get brushed under the carpet by lawyers and ethics committees 

The Nedry-instigated power loss and T-Rex escape wasn’t the first sign that Jurassic Park’s containment was fallible; the opening hook to both the book and movie is an animal handler’s death, and the characters were flown in as an oversight committee of sorts.  But the goal wasn’t an honest investigation; the goal was to put a plausible stamp of approval on the operation by a crew of friendly faces. 

Would DeepMind’s Ethics Board have the independence and freedom to shut down a model which seemed prone to cross the line from “aligned AI” to “unaligned AGI”?  Well

  1. Even perfect technological safeguards fail to (inevitable) human defection 

Although the dinosaurs in Jurassic Park were well on their way to escaping containment on their own (via sex-changes and Lysine-heavy diets), the catastrophic physical containment failures weren’t technological; it was when Nedry shut down the electric fences to steal a vial of embryos and sell on the black market! 

This one is straightforward; it doesn’t matter how responsible your AI alignment oversight committee is, if one of your engineers decides to steal and sell a dangerous model to a Russian crime syndicate for a few million crypto shitcoins.

  1. These agent-systems are built without significant popular or even regulatory input

Jurassic Park was built in secret.  This stretches narrative suspension of disbelief, but it’s a conceit we accept.  And from Hammond’s POV, it’s a surprise he’ll offer the world (although in practice any secrecy was mostly to deflect corporate competitors).

Modern AI isn’t a technical secret to the general populace (Google may even claim to try to inform the public about the benefits of their AI assistant technology), but functionally the general public has no concept of how close or far we are from an AGI which will, for better or for worse, upend their place in the world.

  1. At the end of the day, a new world where these freely-operating agents have completely escaped their control systems is presented as a fait accompli to the general public

Jurassic Park III, (I admit, an overall franchise-milker of a film) ends with a cinematic shot of Pteranodons flying off into the sunset, presumably to find new nesting grounds.  This is a romantic vision, undercut by the fact that those Pteranodons very recently tried to eat the film’s main characters, and presumably the humans who live in those nesting grounds will have no veto power over this new arrangement.

And likewise, AGI — or even scoped AI — absent dramatic regulatory change, is going to be presented as a fait accompli to the general public.   

What’s my point?

I don’t know if the lesson here is that some enterprising cinematographer can reskin Jurassic Park as AGI Park to get Gen Z interested in AGI risk or what, or if there are more bite-sized lessons about how to make hard-to-grok theoretical risks.  

But I do feel we’re missing opportunities to (lightly) inform the public when the contemporary cinematic treatment of technology which is about to turn the world upside down looks like uh, whatever Johnny Depp is doing here, and that really feels like an own-goal for the species (and maybe all sentient organic life if things go really off the rails).

Electoral Savings Accounts: One Person, One Vote (but you can save it for later)

The conversation around “high information” vs “low information” voters hypothesizes a world where voters lie somewhere on a spectrum of “well-informed” to “uninformed”:

Those who fret about “low-information voters” dislike that low-information and high-information voters all count the same at the ballot box, feeling this dilutes the opinions of those who are well-informed about the issues.

Segmenting voters into “high” and “low” information buckets oversimplifies, however, by dropping a dimension — time.  A particular voter’s informedness and enthusiasm (we’ll treat them the same, as a first-order approximation) vary widely over time and life circumstances:

The graph is different for everyone, but in this example, our voter partied through college, became more politically engaged after getting their first job, disengaged when their life got busy, and re-engaged after their kids went to college.

(Of course, we could break this down further into individual issues a voter cares about.  Interest in taxation, education, and environmentalism etc wax and wane with personal circumstances)

How does enthusiasm translate into voting patterns?  Our voter’s voting history may look something like this:

There’s clearly a correlation with enthusiasm and votes cast; in years where the voter is completely disengaged, he/she will likely not bother to vote at all. But in all but the most-disengaged years, the voter will cast the legal maximum of 1 vote.

But if we could design an optimized democracy, is this how we would structure representation?  Probably not.  In a theoretically optimized democracy, voters would cast a number of votes corresponding to how informed and enthusiastic they were about the issues at hand:

Is that possible?

Saving votes for later

We can’t just ask voters how informed they are (poll tests have a sordid history) or how enthusiastic they are (there’s no advantage in being honest).  And any system of buying & selling votes, to rebalance between people, is prone to corruption and disenfranchisement.

But we could let voters cast multiple votes by saving votes, and borrowing future votes, from themselves:

  • When a voter doesn’t choose to vote, the vote is “saved”, and they are free to use it in the future
  • A voter can “borrow” votes from up to 10 years in their future
  • When casting a vote, a voter can spend as many votes as they have available for a race — either dipping into their bank, or borrowing from their future.

This allows our example voter to cast votes that match their enthusiasm curve, with a bit of saving and borrowing:

  • In college, our voter was busy partying, and didn’t really care about the issues.  So they didn’t feel any pressure to vote —but there’s no disenfranchisement, because the votes are saved for later.
  • Once they sobered up and got a job, they really cared about the issues.  Maybe about taxes, or climate change, or both.  Not only did they spend the votes they’d banked from college, but they borrowed from elections into their 30’s.
  • In middle age, they again disengaged, but again not to any permanent disenfranchisement, because once their kids went to college, they were free to spend off their banked votes (or, just save them up).

Electoral Savings Accounts

The details of borrowing and saving votes sounds complicated, but the implementation is actually pretty simple:

  • When a voter turns 18, they start with an Electoral Savings Account (ESA) of 10 votes — ie, allowing them to “borrow” ten years into the future 
  • Each election, the voter’s ESA increases by one.
  • Each election, each race, voters may spend anywhere between none and all of the votes in their ESA balance.
  • Voters hold separate ESAs for each elected position; a voter can vote for school board candidates while abstaining from the presidency, or vice versa.

The last point, when coupled with the fact that people move to new jurisdictions, requires some inter-jurisdictional coordination, to categorize Seattle’s city council election in the same “bucket” as New York’s city council election.  But for the most part, the important races have direct analogues in other cities across the country, and it would not be challenging to build a national mapping from one elected position into a known bucket.

Why are ESAs good for democracy?

The headline reason for ESAs is that they align votes with the times voters are most interested, but there are other reasons they would incentivize a healthy democracy:

ESAs reward honesty by political parties (and punish dishonesty)

It is common for political parties, in their public stances and media advertisements, to frame every race in every election as a highest-priority, life-or-death issue (“the most important election of your lifetime”).  Currently, there is no downside to doing so, because angry voters are good for fundraising.

But using ESAs, a party which hypes up the importance of a non-critical election risks misleading their voters into wasting their entire ESA on an unimportant race.  On the other hand, a party which rightly acknowledges that their opponent is a boring centrist, can save up a war-chest of ESAs their voters can spend on a later, more important, election.

“Voting against everyone” isn’t self-disenfranchisement

Currently, when party primaries produce two terrible candidates, centrist voters are left with two unappealing options:

  • Vote for the slightly lesser evil
  • Voting for nobody

The second option — spoiling a ballot, or just not showing up — is usually unappealing because it amounts to self-disenfranchisement, and sends no clear message to the candidates.

But if an abstained vote goes directly into your ESA, there’s a great reason to skip an election to punish a slate of bad candidates — you can spend the vote later, on a candidate you actually like. 

Check on tyranny-by-majority

Even districts which consistently vote with 45% – 55% splits in a First Past The Post (FPTP) system are considered non-competitive districts, because the party with 55% of the vote almost always wins.  This is a bad deal for the 45% of the population who, despite having 45% of the population, receive 0% of the representation.

ESAs increase the representation of the minority by allowing them to use their votes “when it matters”.  Instead of constantly throwing their votes away on 45-55% elections, the minority party can save up votes to flip the election when an especially viable candidate is on the ballot:

(here, abstaining from voting, in the years with light green, and double-voting in the years with light red)

Disincentivize very polarizing candidates

When a candidate is running against voters who have a substantial ESA pool available, it does not pay to be antagonistic or polarizing. In a FPTP system — what we have now — winning 51% of the vote is Good Enough, and it is often good tactics to make the remaining 49% hate you.  But this is bad for society overall.

The problem is that in a FPTP system, “49% of the population who hate your guts” is electorally indistinguishable from “49% of the population that mildly dislikes you”.  Being very, very angry doesn’t matter.  But if the very angry minority has an ESA balance to spend, they can punish specifically infuriating candidates with electoral upsets.

What’s more, in an electorally efficient system — where candidates and voters behave rationally — this threat of upset votes via ESA spending is enough to motivate inclusivity (or at least a lack of outright antagonism).  And then when the ESAs aren’t actually spent, they’ll continue to motivate inclusivity in the next election, and so forth.

It’s healthy to not care about politics for a few years

High-intensity interest in politics is not good for mental health.  Allowing voters to check out of politics for a few years without sacrificing their representation relative to those who stay involved, allows voters to optimize for their own wellbeing.


ESAs are a simple mechanism, but would fundamentally change the dynamics between voters and elections for the better.  Instead of making votes a use-it-or-lose it opportunity — which cuts out voters who don’t have the time or energy  to research and vote for a whole slate of candidates — it trusts voters with a resource that they can spend when and where they please.

Given that the fundamental premise of democracy is that we do trust the people, it seems likely we could make democracy even more robust by trusting voters to cast ballots not just if, but when they see fit.

Appendix: Variations on ESAs

ESAs as proposed above are “as simple as possible”, but there may be opportunities for refinement at the margins, at the cost of higher complexity:

Expire banked votes

ESA votes as described compound neither positively or negatively over time; 1 vote saved in 1980 can be spent as 1 vote in 2025.

An (IMO unlikely) but possible scenario is if vote-hoarding becomes a destabilizing problem, because voters regularly procrastinate instead of casting votes.  A gentle way to nudge voters into voting sooner rather than later would be to cap the number of years a vote can be banked — for example, a vote not spent within 10 years would expire.

Age-cap borrowed votes

By giving voters a 10-year window of future votes to borrow (that is, by initializing their ESA with 10 votes), we’ve likely inflated the total vote supply.  This is because at the end of a person’s life, they are likely to have spent down their pool of votes, borrowing votes from years they are not alive.

This isn’t catastrophic, but if we wanted to re-normalize the total vote count, ESAs could stop accumulating votes for a corresponding decade, for example between the ages of 60-70 (since the majority of voters will make it to age 70).

Spend votes fungibly across races

If we are allowing voters to spend their votes fungibly across time — because their enthusiasm waxes and wanes over time — a natural extension is to allow voters to spend their votes fungibly across elected positions, in alignment with their enthusiasm.

There are complications in this system (should a vote for a municipal sewage administrator be equivalent to a vote for a senator?), but this is actually the same as quadratic voting, a system fully compatible with ESAs.

Legislative Performance Futures — Incentivize Good Laws by Monetizing the Verdict of History

There are net-positive legislative policies which legislators won’t enact, because they only help people in the medium to far future.  For example:

  • Climate change policy
  • Infrastructure investments and mass-transit projects
  • Debt control and social security reform
  • Child tax credits

The (infrequent) times reforms on these issues are legislated — which happens rarely compared to their future value — they are passed not because of the value provided to future generations, but because of the immediate benefit to voters today:

  • Infrastructure investment goes to “shovel ready” projects, with an emphasis on short-term job creation, even when the prime benefit is to future GDP.  For example, Dams constructed in the 1930s (the Hoover Dam, the TVA) provide immense value today, but the projects only happened in order to create tens of thousands of jobs.
  • Climate change legislation is usually weakly directed.  Instead of policies which incur significant long-term benefits but short-term costs (ie, carbon taxes), “green legislation” aims to create green jobs and incentivize rooftop solar (reducing power bills today).
  • (small) child tax credits are passed to help parents today, even though the vastly larger benefit is incurred by children who exist because the marginal extra cash helped their parents afford an extra child.

On the other hand, reforms which provide no benefit to today’s voter do not happen; this is why the upcoming Social Security Trust Fund shortfall will likely not be fixed until benefits are reduced and voters are directly impacted.

The issue is that while the future reaps the benefits or failures of today’s laws, people of the future cannot vote in today’s elections.  In fact, in almost no circumstances does the future have any ability to meaningfully reward or punish past lawmakers; there are debates today about whether to remove statues and rename buildings dedicated to those on the wrong side of history, actions which even proponents acknowledge as entirely symbolic.

But while the future cannot vote today, financial instruments exist to reward those who made wise choices yesterday — stocks. 

Legislative Performance Futures

Those who bet in 1990 that Apple would be a winner have been massively rewarded.  But politicians who bet in 1920 that segregation is Very Bad, or in the 1950s that the Red Scare was Very Bad, or in the 1980s that nuclear proliferation is Very Bad, suffered electorally for their stances, and in recompense get only faint praise from history professors.

Though we cannot electorally incentivize forward-thinking politicians, we can monetarily incentivize them.  Specifically, by monetizing the future public opinion of today’s legislators, we can provide lawmakers with an incentive to pass laws for which history judges them kindly.  

We can call this instrument a Legislative Performance Future (LPF) and it would work something like this:

  • In lieu of direct compensation, legislators receive LPFs, or shares, on their future job evaluations, which will be paid out 40 years from the date of issue.  For example, a 2020 Arkansas congressman on entering office will be granted 100 LPFs, in his or her name, maturing in 2070.
  • Each year, voters, in addition to electing current representation, “vote” among the representatives who served exactly 40 years ago.
  • A fixed fraction of GDP, in aggregate corresponding to very generous salaries — .01% of GDP or so — is paid out proportional to the above retrospective votes, to the holders of the corresponding LPFs.
  • LPFs are fully fungible, and can be inherited or sold like any other financial or physical asset.  Because legislators need money to live their lives, it is expected that they will immediately liquidate many (or most) of their shares into cash.

In simple terms, those who enacted legislation for which history thanks them (such as establishing the EPA) are rewarded.  Those who passed laws which history scorns — Jim Crow laws, internment caps — are not.

This system might work as-is, with legislators blinding voting their consciences and hoping that history someday rewards them.  But this feedback loop is long.  Luckily, by making LPFs tradable assets, we can do much better.

Markets Predict

Because legislators need money to live (those who don’t get book deals), and LPFs are monetized commodities, we expect that most legislators will sell (most of) their LPFs at market value.

But what’s the market value of an LPF?

The value of an LPF today is set not by the future, but by the markets of today predicting what the future will think.  It is not a leap to believe that markets will align the value of LPFs to future sentiment more accurately than legislators today do to their future reputation.

Today, when a legislator proposes major legislation, their staff closely monitors public opinion polls to gauge public sentiment.  The polls move up and down almost immediately (but in a manner only loosely correlated with likely future sentiments, because the voters of today vote primarily with their wallets and emotions).

But the buyers and sellers of LPFs are incentivized only to be correct.  They may hold personal stances on the issues, but have every reason to set those stances aside and let homo economicus perform brokerage transactions.  This produces good predictions. 

(a personal example: I am not a vegetarian.  I enjoy eating meat, and would not vote to outlaw meat.  But I recognize that the tides of history are clearly against animal farming and consumption, and history will consider me in the wrong.  So if a legislator in my district proposed banning pig farming, I would vote against them — but I would immediately buy into their LPF.)

Thus, LPFs change this dynamic.  Because a legislator’s LPFs trade on the open market, their value will move immediately when new legislation is proposed — legislators don’t have to wait to reap the verdict of history; they can immediately reap the verdict of projected history by proposing (and passing) laws which are primarily good only for the next generation, and then selling their LPFs at a now-higher price.

LPFs will not help politicians win elections.  But to the extent that politicians are corrupt and money-grubbing humans, LPFs will align that greed with the desire to pass good, forward-thinking, laws.

Implementation Questions

How will voters even know what politicians of 40-years-ago stood for (without extensive research)?

Easy — we just re-print the voter information guides they used in their own elections.  LPFs incentivize politicians to be very explicit about their policy positions in their candidate statements, in order to stand out during the LPF value-evaluation vote. 

Will voters even bother voting in LPF evaluations?

Many won’t.  But unlike in elections today (where the stakes are actually pretty high, even in “minor” races) there’s relatively little impact if only high-information voters bother to cast votes.  Money is redistributed, but the only way the system breaks is if voting is so random that politicians today lose faith in the connection between performance and payouts.

Likewise, there is (almost) no incentive for voters to ever vote tactically or against their own beliefs, because money is parcelled out proportional to votes (not winner-take-all).  There’s simply no reason for voters to be dishonest.

How do we actually start trading LPFs?

Practically, there’s a bootstrapping problem.  If legislators today (2021) were paid in 40-year-maturity LPFs, 40 years of LPFs would be bought and traded before a dollar was paid out.  The uncertainty of that process (confidence that investments would really pay out) is likely to depress prices.  We can instead bootstrap the process by gradually extending the LPF maturity dates:

  • In 2021, legislators are paid in LPFs which mature in 2023
  • In 2022, legislators are paid in LPFs which mature in 2025
  • In 2023, legislators are paid in LPFs which mature in 2027
  • … and so on, up to the final generational maturity window of 40 years.  

This way, the system will pay dividends as soon as 2 years from inauguration, to build confidence, but within half a generation (20 years) will become the long-term instrument we’ve intended.

What laws do I think LPFs will incentivize?

With the caveat that these are the stances I see history judging, not a function of personal opinion, I would personally invest in a LPF portfolio built around:

  • Carbon taxation
  • YIMBY policies which depress housing prices
  • Cuts to social security and medicare benefits
  • Child tax credits and maternity leave
  • Banning or restricting factory farming

(and if I’m wrong, the great thing about LPFs is that my money is where my mouth is)


LPFs will not fix all our problems of a broken lawmaking process.  Most fundamentally, they cannot get politicians re-elected.  But they may at the margin nudge politicians into politically unpopular stances which they believe will be looked upon favorably.

Possibly more importantly than the financial payout, at least for legislators truly interested in “doing the right thing”, is that in moments of doubt they can literally consult an “opinion poll from the future”  (or at least, our best guess at one).

Last but not least, by asking voters to cast their own retrospective votes, it will nudge them to view their own present-day votes by the light that their children’s children will judge them.  

In this at least, it is hard to see any downside.

The corporate / academic / public AI value gap

There is a huge gap between the benefits of Artificial Intelligence the public is being sold, the benefits of AI which are being marketed to corporate adopters, and the actual motivations of AI researchers.

  • Tech providers pitch AI as a driver of innovation (self-driving cars) and global good (mitigating global warming).  But the B2B case-studies pitched to corporate clients more often pitch AI solutions as better automation, mostly enabling cost-reduction (specifically, reducing human-in-the-loop labor).
  • While many AI researchers are motivated by genuine interest in improving the human condition, other motivations diverge — a desire to push the bounds of what we can do, a genuine belief in transhumanism (the desire for AI to replace, or transform into something entirely unrecognizable, humanity), or simply because AI pays bigly.

These drivers — replacing human employment, and perhaps humans themselves — are, to put it mildly — not visions the public has bought into.

But these internal motivations are drowned out by the marketing AI pitch by which AI is sold to the public: “AI will solve [hunger/the environment/war/global warming]”.  This leaves the people not “in the know” about AI progress — 99% of the population — not even thinking to use democracy to direct AI research towards a world the (average) person actually wants to live in.

This is not particularly fair.

Marketed AI vs Profitable AI

To the public, the tech giants selling AI solutions (Google, Microsoft, and Apple) pitch visions of AI for good.  

The public face of these advertising campaigns is usually brand advertising, perhaps pitching consumers on a software ecosystem (Android, iOS), but rarely selling any specific product.  This makes it easy to sell the public a vision of the future in HDR, backed by inspirational hipster soundtracks.

You all know what I’m talking about — you’ve seen them on TV and in movie theaters — but the imagery is so honed, so heroic, that we should look at the pictures anyway.

Google’s AI will do revolutionary things, like fix farming, improve birthday parties, and help us not walk off cliffs: 

Microsoft’s AI is safe.  You can tell because this man is looking very thoughtfully into the distance:

But if that is not enough to convince you, here is a bird:

Microsoft goes into detail on their “AI for Good” page.  The testimonials highlight the power of AI as applied to:

  • Environmental sustainability (image recognition of land use, wildlife tracking, maximizing farm yields)
  • Healthcare (dredging through data to find diseases)
  • Accessibility (machine translation, text to speech)
  • Humanitarian action and Cultural Heritage preservation

Even the Chinese search engine Baidu, not exactly known for their humanitarian work, has joined the OpenAI “safe AI” consortium, which is nominally dedicated to developing and selling only safe AI.

The theme among all these public “good AI” initiatives — the sales pitch to the public — is:

“We’re developing advanced AI, but we’re partnering with NGOs, hospitals, and more, to make this AI work for people, not against them.  Look at all the good we can do!”

This isn’t fake.  Microsoft is working with nonprofits, NGOs, and more, to deploy for-the-people AI.  But these applications don’t get us closer to the real question:

“What solutions are normal companies actually deploying with AI-as-a-service cloud technology?”

We can peek behind the curtain at Amazon.  Amazon’s AWS has been for the last decade synonymous with “the cloud”, and still has a full 50% market share.  The bleeding edge of AWS are plug-and-play machine learning and AI tools: Amazon Forecast (machine learning), Amazon Polly (text to speech), Amazon Rekognition (video object recognition), Amazon Comprehend (natural language processing), and more.

And Amazon, alone and refreshingly among tech giants, doesn’t even pretend to care why their customers use AI:

“We certainly don’t want to do evil; everything we’ve released to customers to innovate [helps] to lift the bar on what’s actually happening in the industry. It’s really up to the individual organisation how they use that tech”

Amazon sells AI to C-suites, and we know what the hooks are, because the marketing pitches are online.  AWS publishes case studies about how their plug-and-play AI and ML solutions are used by customers. 

We can look at a typical example here, outlining how DXC used AWS’s ML and AI toolkits to improve customer service call center interactions.  Fair warning:  the full read is catastrophically boring — which is to be expected when AI used not to expand the horizon of what is possible… but instead used to excise human labor from work which is already being done:

“DXC has also reduced the lead time to edit and change call flow messaging on its IVR system. With its previous technology, it took two months to make changes to IVR scripts because DXC had to retain a professional voice-over actor and employ a specialized engineer to upload any change. With Amazon Polly, it only takes hours”

Using Amazon Connect, DXC has been able to automate password resets, so the number of calls that get transferred to an agent has dropped by 30–60 percent.

DXC anticipates an internal cost reduction of 30–40 percent as a result of implementing Amazon Connect, thanks to increased automation and improved productivity on each call.

In total, what did DXC do with its deployed AI solution?  AI is being used to:

  • Replace a voice-over actor
  • Eliminate an operations engineer
  • Eliminate customer service agents

There’s nothing evil in streamlining operations.  But because of the split messaging being used to sell AI research to the public vs to industry — on one hand, visions of environmental sustainability and medical breakthroughs, and on the other hand, the mundane breakthrough of applying a scalpel to a call center’s staffing — the public has little insight (other than nagging discomfort) into automation end-game.  

The complete lack of (organized) public anger or federal AI policy — or even an attempt at a policy — speaks to the success of this doublespeak.

Research motivations

So why are actual engineers and researchers building AI solutions?

I could dredge forums and form theories, but I decided to just ask on reddit, in a quick and completely unscientific test.  Feel free to read all the responses — I’ve tried to aggregate them here and distill them into the four main themes.  Weighted by upvotes, here’s the summary:

Preface: none of these are radical new revelations.  They match, in degrees, what you’d find with a more exhaustive dragnet of public statements, blogs, or after liquoring up the academic research mainstream.

Walking down the list:

1. Improving the human condition

A plurality goal is to better the human condition, which is promising.  An archetypal response is a vision of a future without work (or at least, without universal work):

“I believe the fundamental problem of the human race is that pretty much everyone has to work for us to survive.

So I want to work on fixing that.”

It’s not a vision without controversy — it’s an open question whether people can really live fulfilled lives in a world where they aren’t really needed — but at minimum it’s a vision many could get behind, and is at root predicated in a goal of human dignity.

2. It pays

Close behind are crude economics.  Top comment:

“Dat money”

I don’t intend to sound negative — capitalism is the lever which moves the world, and in capitalism, money follows value.  But as shown by AWS, value can come from either revolutionary invention (delivering novel value), or cost excision (delivering cheaper value).

Either direction pays the bills (and engineers), and few megacorp engineers care to peek behind the curtain at which specific aspect of the AI product delivered to clients pays the bills.

3. Transhumanism

Here’s where the interests of true believers in AI diverge from the mainstream.  Top comment:

“I don’t really care about modern ML solutions, I am only concerned with AGI. Once we understand the mechanisms behind our own intelligence, we move to the next phase in our species’ evolution. It’s the next paradigm shift. Working on anything else wouldn’t be worth it since the amount of value it brings is so vast.”

“I’m in it for the money” is just realism.  “A world without work” and “making cheddar” are motivations which appeal to the mainstream, and is at least comprehensible (if frustrating) to those whose jobs are on the line.  

Transhumanism is different.  There’s a prevalent (although possibly not majority) philosophy among many AI researchers, practitioners, and enthusiasts, that the goal of developing strong (human-level) AI is not a tool for humans, but an end unto itself.  The goal is the creation of a grander intelligence beyond our own:

“Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.”

Or, step-by-step:

  • Humans create AI 1.0 with IQ human + 1
  • AI 1.0 creates AI 2.0, which is slightly smarter
  • AI 2.0 creates AI 3.0, which is WAY smarter
  • AI 3.0 creates AI 4.0, which is incomprehensibly smarter

And whatever comes next… we can’t predict.

This is not a complete summary of transhumanism.  There’s a spectrum of goals, and widespread desire for AI which can integrate with humans — think, nanobots in the brain, neural augmentation, or wholesale digital brain uploads.  But either way — whether the goal is to retrofit or replace humans — the end goal is at minimum a radically transformed concept of humanity.

Given that we live in a world stubbornly resistant to even well-understood technological revolutions — nuclear power, GMOs, and at times even vaccines — it’s fair to say that transhumanism is not a future the average voter is onboard for.

4. Just to see if we can 

And just to round it out, a full 16% of the votes could be summarized (verbatim) as:

“Why not?”

Researchers — and engineers — want to build AI, because building AI is fun.  And there’s nothing unusual about Fun Driven Development.  Most revolutionary science doesn’t come from corporate R&D initiatives; it comes from fanatical, driven, graduate students, startups, or bored engineers hacking on side projects.

Exploration for the sake of exploration (or with a thin facade of purpose) is what got us out of trees and into lamborghinis.

But at the end of the day, “for the fun” is an intrinsic motivation akin to “for the money”.  The motivation gives one engineer satisfaction and purpose, but doesn’t weight heavily on the scales when answering “should this research exist?” — in the same way we limit fun of experimental smallpox varietals, DIY open-heart surgery, and backyard nuclear bombs

Misaligned motivations

The public has been sold a vision of AI for Good; AI as an ex machina for some (or all) of our global crises, now and future:

These initiatives aren’t fake, but they also represent a small fraction of actual real-world AI deployments, many if not most of which focus on selling cost-reductions to large enterprises (implicitly and predominantly, via headcount reductions).

AI researchers and implementers, in plurality, believe in the potential good of AI, but more frequently are in it for the money, to replace (or fundamentally alter) humans, or just for the fun of it.

The public, and their elected governments, can’t make informed AI policy if they are being sold only one side of the picture — with the unsavory facts hidden, and the deployment goals obscured.   These mixed messages are catastrophically unfair to the 99% of humanity not closely following AI developments, but whose lives will be, one way or another, changed by the release of even weak (much less, strong) AI.

Ethically Consistent Cryonics

John woke.  First wet, then cold.  

“Hello John.  I’m Paul Mmmdcxxvi”

Paul’s face drifted into focus.  

“You froze to death (by the standards of the time) climbing Everest in 2005.  You were chipped out of a glacier last week.  Thanks to recent medical advances, you defrosted with minimal tissue damage.”

Hrrrngh.  “Hrrrrng” 

“It is no longer 2005” Paul helpfully added.  “But I need to explain a few things, to frame the real date.  Take your time.  Take a drink.”  Paul gestured at a bottle on a nearby table.

John sat upright, drank, and after a few minutes, looked around.  

The room was dimly lit, but vast.  Behind Paul, extending as far as John could see, stood row upon row of barrels, stacked upwards into complete darkness.  Nearby barrels misted slightly.  Between rows, up in the air, crawled… something?  Somethings?  

Behind Paul blinked a single standard LCD monitor — the only clear light.

“How are you feeling, John?”  Paul prompted.

“Better.” Surprisingly well, John realized.  “So it’s not 2005.  When is it?”

“You missed a lot, so I’ll need to start where you left off.  I apologize if I’m scattered; please do interrupt”  Paul paused, and started:

“Our civilization had two great breakthroughs in the middle of the 21st century.  

“The first was moral.  The earth struggled with ethics in the early 21st century.  We had made great advances in mathematics and physics, but our moral calculus was stagnant by comparison.   Good people put together moral frameworks like Utilitarianism, concepts like Quality Years of Life saved, and tools to compute the time-decay discounts for future lives.  

“Gradually, a formal field of research emerged — computational ethics.  Many of our best minds went to work, and researchers constructed some truly heroic Google Sheet formulas.   But at the end of the day, they couldn’t avoid the inherent contradictions.”

“Contradictions?” John objected.  “I’m no mathematician, but before I died, I lived a decent life.  I still feel bad about stealing Pokemon cards in 5th grade.”

“Not surprising, for a 20th century mind.   But you were optimizing for observable, personal moral impact.  Computational ethics tried to be scientific about optimizing human morality.  For example: how could you justify eating premium Ostrich burgers, while UNICEF struggled to fundraise for famine relief?”

“Well” John admitted, “I assumed my tax dollars mostly took care of that.”

“Naturally.  And that’s just the tip of the iceberg.  We started with the “easy” optimization, maximizing Quality Years of Life.  It worked well at first; we eliminated Polio, hunger, and male-pattern-baldness.  But we got stuck.  It turned out there was no way to optimize out suicide, but leave BASE jumping as a optional joie de vivre.  Or leave potato chips.  Or leave the fun game where teenagers shoot fireworks at each other.”

John mindlessly scratched at the old burns where his left ear once grew.  “That’s a shame, I had a really fun childhood.”

“But it got even worse.  When we ran the numbers, there was no consistent system which condemned murder but condoned voluntary vasectomies.  The average vasectomy destroyed billions of QALYs, by eliminating children, children of children, their grandchildren…”

Wait, what?  “Well, that’s ridiculous.  One is killing a real person, and one is just eliminating a hypothetical future.  If you evaluate your moral utility as a delta from what you could have done instead, you’re going to go crazy.”

“That’s what we eventually figured out.  Science would have moved much faster if gifted with clear thinkers like you.  So we cut all that ‘possible futures’ stuff out, and settled on a simple axiom.”

“Axiom?”  Is this some geometry thing?  “ I just figure that dying is bad, much worse than the other bad stuff.”

“Exactly.  That’s where we landed: ‘People dying is bad’.  It’s been our guiding law, in the few thousand centuries since.  We’ve rebuilt society accordingly.”

That seems fine.  John figured.  Wait.  The “thousand centuries” thing is concerning.  Paul seems like a nice guy, but storytelling isn’t his forte.  “Speaking of which, where is everyone?  Why is it so cold?  And what’s with all the storage tanks?  Why’d you defrost me in an Amazon warehouse?

“That gets me to the second great breakthrough: cryo-sleep.  

“You were incredibly lucky — most people who fell headfirst and naked into a glacier in 2005 ended up a decomposed mess.  But in the 2030s, we perfected supercooled nitrogen bath immersion, and could reliably freeze freshly-dead bodies, with minimal to no risk.”


“Cosmetic tissue loss, and nobody’s going to win any Nobel prizes.  But the point is, we can stop people from fully dying.  Once we figure out how to fix what killed someone, we can defrost them, fix them, and send them back out to enjoy a full life.

Huh.  “That’s really the dream, then.  Live fast, die young, and …”  



“Eventually, perhaps.  But if you die too fast, nobody can put you back together.  We could save the gunshot victims, and stop some heart attacks.  But you know what they say — all the king’s horses and all the king’s men, can’t really do much, when Humpty’s parachute strings get tangled… so we couldn’t justify letting thrill-seekers skydive.”

Ok, so the future is kind of boring.  I can live with that, I guess. “So what do people do for fun now-days, if the risky stuff is off the table?”

“I’m getting there.  There was a bigger unsolved problem, a nut we haven’t cracked yet.  I’ve personally been working on it for the last forty years.”


“Old age.  Even if you eliminate the dumb deaths like snakes, train crashes, and undeployed parachutes, eventually, people just break down.  When a person makes it to the end — around 120 is the best we ever did— we run out of ways to keep them ticking.

“It’s likely fixable.  But it’s possible that we won’t ever be able to reverse aging, only forestall it.   Thermodynamics is a bitch.  So we decided it’s ethically unsound to ever let a person die of old age.  Cryo-sleep isn’t forever, but death from old age might be.   So we head it off.  When someone hits 80, they go into storage, and stay there until we’ve figured out how to stop aging”

That’s neat, but I’m only 30, and I’m also recently defrosted, cold, and hungry.  This doesn’t seem super important.  “I really appreciate this backstory, but I’d appreciate the cliff notes.  Is there anyone else who could swap in?”

“I’m getting there.  There isn’t anyone else.”

… fuck?

“We almost figured it out too late — by the late 21st century, the vast majority of our energy infrastructure was dedicated to cryogenics.  The more people who kept breathing, the more people who kept getting older.  When they hit 80, they filled up scarcer and scarcer cryo tanks.  

“We only had a few decades left before we hit Peak Cryo.  And if we run out of places to store people, it’ll be a catastrophe — we’ll have to pick and choose who to defrost and recycle.  We can’t let that happen!

“Obviously we can’t just give up on fixing ageing.  Everyone would end up dead!  But it doesn’t make sense to parallelize that research — haven’t you read The Mythical Man-Month?  We couldn’t waste effort — every hour counts.”

Ugh, I get it.  “So you froze everyone.  To stop the clock.”

“Precisely.  Some people were unhappy, but most understood the logic — that it was the only possible choice, given our consistent moral foundation.”

Being dead had benefits.  “I suppose that’s an insane but rational choice.  So how close are we…  you…  to actually solving ‘old age’?”

“Me? Honestly, I gave it an honest effort for a decade, but found the biology all very confusing.  I was just a furniture salesman before this, you know?  I’ve spent the last 25 years mostly organizing my predecessor’s notes, in the hopes I could give you a head start.”

John blinked several times, and then several more.  “Me?”

“Of course, you.  It’s really just dumb luck the excavator-bots dug you up (while freeing up some storage space on the north slope) right before I retired.  You’re the youngest person alive —or whatever — by a solid decade, so you’re next in line.

“It’s totally natural to not feel up to the task.  But don’t sweat it — it’s not a huge setback if you fail.  Once you’ve put in 50 years of the good-old-college-try, you’ll get retired, someone fresh will swap in.”

Uh, yeah, prepare for disappointment.   “And if it takes longer than that?  What if nobody currently ‘alive’ can solve it?”

“Worst case, we have a system for making new people, of course.  It was the first thing we developed.  But we won’t start unless it’s a true catastrophe, and run out of existing people.  Given storage constraints, it’s irresponsible to go around making more people except as a last resort.”

I read some sci-fi back in college.  Surely there’s a less-stupid way out of this.  “What about storing people off-world?  Maybe on the moon?”

“Dangerous.  Here let me — ah, I have the notes”  Paul swiveled back to the monitor.   “My predecessor spent a few years on this, and couldn’t figure out how to deal with the potential micrometeorite damage.  But I’d certainly encourage you to take a shot.  Fresh neurons, fresh ideas!”

Well.  “And if I say no?”

“I can tell you’re struggling.  It’s ok.  This is why decisions in the 21st century were so hard.  Your people had no real moral axis!

“If you say no, obviously there are backup plans.  The automation” Paul gestured up at the tangle of arms and wires menacingly crawling between stacks of barrels — cryo-tanks, John realized —  “would simply re-freeze you and put you in the back of the queue.  It would be quite unpleasant, but you have my promise you’d survive.  Someday we’ll all get defrosted and have a good laugh about it.”

Paul slowly rose, his right arm on a cane, waving his left arm as if to hail a cab.  “I don’t like to cut this off so soon.  But I’ve already given you far more time than the norm thanks to the unusual circumstances of your defrosting, and I really shouldn’t linger.  It’s far too dangerous, at my age.”

The closest mess of wires enveloped and lifted Paul as he shook John’s hand.

“Take your time and mull it over.  Just shout, and the help” (gesturing to the nearest arm) “will point you to a shower and a hot meal.  The research can wait until you’re ready.   One nice benefit of our ethical framework, is that no decision ever has to be rushed.”  

The wires disappeared quietly in the dark.  Uncomfortably alone, John stared at the monitor blinking in front of him.  This is the stupidest possible futureAt least Terminator had an awesome robot uprising.  

But at the same time, and what have I got to lose?  The future is already stupid, and I certainly can’t make it any worse.  I deserve a shower and breakfast first.  I can take a crack at this for at least a day.  I can try anything for a day.  It can’t be much worse than being properly dead.   

And it’s not like a bit of hard work is going to kill me John admitted, because dying is absolutely not an option anymore.

Peacetime Bureaucracy / Wartime Bureaucracy

Peacetime bureaucracy forces all tests through the CDC.  Wartime bureaucracy allows hospitals to run their own tests.

Peacetime bureaucracy priorities food labeling rules over functional supply chains.  Wartime bureaucracy gets food out of fields before it rots.

Peacetime bureaucracy creates ethical review boards.  Wartime bureaucracy allows volunteers to infect themselves to test vaccines.

Peacetime bureaucracy inventories ICU beds.   Wartime bureaucracy builds hospitals.

Peacetime bureaucracy certifies medical-purpose N95 respirators.  Wartime bureaucracy uses construction masks.

Peacetime bureaucracy waits for double-blind, controlled efficacy studies. Wartime bureaucracy tells shoppers to wear masks.  

Peacetime bureaucracy prioritizes fraud prevention.  Wartime bureaucracy writes checks.

Peacetime bureaucracy plays diplomatic games to stay funded.  Wartime bureaucracy takes advice from Taiwan.

Peacetime bureaucracy considers every life sacred.  Wartime bureaucracy balances QALYs saved against the price tag.

Peacetime bureaucracy prioritizes racially sensitive nomenclature.  Wartime bureaucracy stops international flights.

Peacetime bureaucracy requires HIPAA certification for telemedicine.  Wartime bureaucracy lets doctors use Skype.

Peacetime bureaucracy optimizes for eventual correctness.  Wartime bureaucracy treats time as the enemy.

Peacetime bureaucracy optimizes for public support in the next election cycle.  Wartime bureaucracy has a long-term plan.

Investors know the difference between peacetime CEOs and wartime CEOs, and trade them out when times demand change.  How do we build institutions which quickly exchange peacetime bureaucracy for wartime bureaucracy? 

Two months into COVID-19, we’re barely halfway there. Next decade (or next year) there will be a next disaster, war, or pandemic.  When that happens, we need wartime officials ready to act — not peacetime officials reluctantly easing into the role.  These public officials must be able to make hard choices with incomplete information.

We need to learn from COVID-19, but the preparation can’t stop at “stockpile masks and ventilators”. Preparation means having officials ready to eliminate red tape, make new rules, and make hard choices on day 1 — not day 30.

We got lucky this time, a trial-run on a pandemic whose victims are (predominantly) the old and sick. To fail utterly at curbing COVID-19 precipitates an ethical, but not civilizational, failure.

We’re unlikely to be so lucky next time. The future is full of darker, bloodier pandemics than COVID-19 — both natural ones, and man-made ones. When one strikes (and it will) we need a wartime bureaucracy and a playbook ready, telling us which of the old rules still matter — and which rules will not.


It’s easy to think without reading, but also easy to read without thinking.

I’ve started reading nonfiction again.  I have a good reason for stopping: I was stuck halfway through Proofs and Refutations for about a year and a half, and as a committed completionist, I couldn’t start any other books until it was done.  After powering through the dregs of P&R, I know a lot about…. proofs, less about polyhedrons, and I’m free to re-engage in educational literature.

It’s easy to read without reflecting, though.  I’d venture that 90% of “consumed content by volume” — especially online content — functions only to:

  1. Reinforce biases, in an (optimistically) intellectual circlejerk
  2. Get the reader frothing mad when they Read Stupid Opinions by Stupid People

I don’t think I’m uniquely bad at “intellectually honest reading” —  but “median human” is a low bar, and not one I’m confident I always clear.  If I’m going to go through the motions of reading brain books, I need a forcing function to ensure the input actually adjusts my priors;  if after having read a book, I haven’t changed my mind about anything, I’m wasting my time on comfortable groupthink.

My forcing function — until I get tired of doing it — will be to write something here.  There may be inadvertent side-effects (like accidentally reviewing the book, although I hope not), but my only commitment is: to outline at least one stance, large or small, the book has changed my mind on.  Or, lacking that, forced me into an opinion on a topic I hadn’t bothered to think about.

If I can’t find one updated stance, I’m wasting my time. Committing that stance to writing forces crystallization, and committing that writing to a (marginally) public audience forces me to make the writing not entirely stupid.

I make no commitment to keeping this up, but I waited to write this until I had actually written an un-review, so at least n=1, and by publicly declaring a plan, I can (hopefully) guilt myself into maintaining the habit.

Schrödinger’s Gray Goo

Scott Alexander’s review of The Precipice prompted me to commit to keyboard an idea I play with in my head: (1) The biggest risks to our humanity the ones we can’t observe, because they are too catastrophic to survive, and (2) we do ourselves a disservice by focusing on preventing the catastrophes we have observed.

Disclaimer: I Am Not A Physicist, and I’m especially not your physicist.

1. Missing bullet holes

The classic parable of survivorship bias comes from the Royal Air Force during WWII.  The story has been recounted many times

Back during World War II, the RAF lost a lot of planes to German anti-aircraft fire. So they decided to armor them up. But where to put the armor? The obvious answer was to look at planes that returned from missions, count up all the bullet holes in various places, and then put extra armor in the areas that attracted the most fire.

Obvious but wrong. As Hungarian-born mathematician Abraham Wald explained at the time, if a plane makes it back safely even though it has, say, a bunch of bullet holes in its wings, it means that bullet holes in the wings aren’t very dangerous. What you really want to do is armor up the areas that, on average, don’t have any bullet holes.

Why? Because planes with bullet holes in those places never made it back. That’s why you don’t see any bullet holes there on the ones that do return.

The wings and fuselage look like high-risk areas, on account of being full of bullet holes.  They are not. The engines and cockpit only appear unscathed because they are the weakest link.  

2. Quantum interpretations

The thought-experiment of Schrödinger’s cat explores possible interpretations of quantum theory:

The cat is penned up in a steel chamber, along with the following device: In a Geiger counter, there is a tiny bit of radioactive substance, so small, that perhaps in the course of the hour one of the atoms decays, but also, with equal probability, perhaps none; if it happens, the counter tube discharges and through a relay releases a hammer that shatters a small flask of hydrocyanic acid. If one has left this entire system to itself for an hour, one would say that the cat still lives if meanwhile no atom has decayed. The first atomic decay would have poisoned it. 

Quantum theory posits that we cannot predict individual atomic decay; the decay is an unknowable quantum event, until observed.  The Copenhagen interpretation of quantum physics declares that the cat’s state is collapsed when the chamber is opened — until then, the cat remains both alive and dead.

The many-worlds interpretation declares the opposite — that instead, the universe bifurcates into universes where the particle did not decay (and thus the cat survives)  and those where it did (and thus the cat is dead).

The many-worlds interpretation (MWI) is an interpretation of quantum mechanics that asserts that the universal wavefunction is objectively real, and that there is no wavefunction collapse. This implies that all possible outcomes of quantum measurements are physically realized in some “world” or universe.

The many-worlds interpretation implies that there is a very large—perhaps infinite—number of universes. It is one of many multiverse hypotheses in physics and philosophy. MWI views time as a many-branched tree, wherein every possible quantum outcome is realised. 

3. The view from inside the box

The quantum suicide thought-experiment imagines Schrödinger’s experiment from the point of view of the cat.  

By the many-worlds interpretation, in one universe (well, several universes) the cat survives.  In the others, it does. But a cat never observes universes in which it dies. Any cat that walked out of the box, were it a cat prone to self-reflection, would comment upon its profound luck. 

No matter how likely the particle was to decay — even if the outcome was rigged 100 to 1 — the outcome remains the same.  The cat walks out of the box grateful to its good fortune.

4. Our box

Or perhaps most dangerously, the cat may conclude that since the atom went so long without decaying, even though all the experts predicted decay, the experts must have used poor models which overestimated the inherent existential risk.

Humans do not internalize observability bias.  It is not a natural concept. We only observe the worlds in which we — as humans — exist to observe the present.  Definitionally, no “humanity-ending threat” has ended humanity.   

My question is: How many extinction-level threats have we avoided not through calculated restraint and precautions (lowering the odds of disaster), but through observability bias?

The space of futures where nanobots are invented is (likely) highly bimodal; if self-replicating nanobots are possible at all, they will (likely) prove a revolutionary leap over biological life.  Thus the “gray goo” existential threat posited by some futurists:

Gray goo (also spelled grey goo) is a hypothetical global catastrophic scenario involving molecular nanotechnology in which out-of-control self-replicating machines consume all biomass on Earth while building more of themselves

If self-replicating nanobots strictly dominate biological life, we won’t spend long experiencing a gray goo apocalypse.  The reduction of earth into soup would take days, not centuries:

Imagine such a replicator floating in a bottle of chemicals, making copies of itself…the first replicator assembles a copy in one thousand seconds, the two replicators then build two more in the next thousand seconds, the four build another four, and the eight build another eight. At the end of ten hours, there are not thirty-six new replicators, but over 68 billion. In less than a day, they would weigh a ton; in less than two days, they would outweigh the Earth

Imagine a world in which an antiBill Gates stands with a vial of grey goo in one hand, and in the other a geiger counter pointed at an oxygen-14 molecule — “Schrödinger’s gray goo”.  Our antiBill commits to releasing the gray goo the second the oxygen-14 molecule decays and triggers the geiger counter.

In the Copenhagen interpretation, there’s a resolution.  The earth continues to exist for a minute (oxygen-14 has a half-life of 1.1 minutes), perhaps ten minutes, but sooner or later the atom decays, and the earth is transformed into molecular soup, a giant paperclip, or something far stupider.  This is observed from afar by the one true universe, or perhaps by nobody at all.   No human exists to observe what comes next. [curtains]

In the many-worlds interpretation, no human timeline survives in which the oxygen-14 model decays. antiBill stands eternal vigil over that oxygen-14 atom: the only atom in the universe for which the standard law of half-life decay does not apply.

5. Our world

As a species we focus on preventing and averting (to the extent that we avert anything), the risks we are familiar with:

  • Pandemics
  • War (traditional, bloody)
  • Recessions and depressions
  • Natural disasters — volcanoes, earthquakes, hurricanes 

These are all bad.  As a civilization, we occasionally invest money and time to mitigate the next natural disaster, pandemic, or recession.

But we can agree that while some of these are civilizational risks, none of them are truly species-level risks.  Yet we ignore AI and nanotechnology risks, and to a lesser but real degree, we ignore the threat of nuclear war.  Why though?

  • Nuclear war seems pretty risky
  • Rogue AI seems potentially pretty bad
  • Nanobots and grey goo (to the people who think about this kind of thing) seem awful

The reasoning (to the extent that reasoning is ever given) is: “Well, those seem plausible, but we haven’t seen any real danger yet.  Nobody has died, and we’ve never even had a serious incident”

We do see bullet holes labeled “pandemic”, “earthquake”, “war”, and we reasonably conclude that if we got hit once, we could get hit again.  Even if individual bullet holes in the “recession” wing are survivable, the cost to human suffering is immense, and worth fixing.  Enough recession/bullets may even take down our civilization/plane. 

But maybe we are missing the big risks, because they are too big.  Perhaps there exist fleetingly few timelines with a “minor grey goo incident” which atomizes a million unlucky people.  Perhaps there are no “minor nuclear wars”, “annoying nanobots” or “benevolent general AIs”. Once those problems manifest, we cease to be observers.

Maybe these are our missing bullet holes.

6. So, what?

If this theory makes any sense whatsoever — which is not a given — the obvious followup is that we should make a serious effort to evaluate the probability of Risky Things happening, without requiring priors from historical outcomes. Ex:

  • Calculate the actual odds — given what we know of the fundamentals — that we will in the near-term stumble upon self-replicating nanotechnology
  • Calculate the actual odds — given the state of research — that we will produce a general AI in the near future?
  • Calculate the actual odds that a drunk Russian submariner will trip on the wrong cable, vaporize Miami, and start WWLast?

To keep things moving, we can nominate Nicholas Taleb to be the Secretary of Predicting and Preventing Scary Things.  I also don’t mean to exclude any other extinction-level scenarios. I just don’t know any others off the top of my head.  I’m sure other smart people do.

If the calculated odds seem pretty bad, we shouldn’t second guess ourselves — they probably are bad.  These calculations can help us guide, monitor, or halt the development of technologies like nanotech and general AI, not in retrospect, but before they come to fruition.

Maybe the Copenhagen interpretation is correct, and the present/future isn’t particularly dangerous.  Or maybe we’ve just gotten really lucky.  While I’d love for either of these to put this line of thought to bed, I’m not personally enthused about betting the future on it.

Ineffective Altruism

Texas Lt. Gov. Dan Patrick hung up a morality piñata when he had the audacity to state, on record, that he’d be willing to take risks with his life to prevent an economic meltdown:

No one reached out to me and said, ‘As a senior citizen, are you willing to take a chance on your survival in exchange for keeping the America that all America loves for your children and grandchildren?’ And if that is the exchange, I’m all in

Like many other moral-minded folk, Cuomo took a free swing at the piñata a few hours later, and snapped back with this zinger:

While this tweet is genetically engineered to be re-twatted, it is a ridiculous statement.  People put prices on human life all the time. Insurance companies price human lives. Legislators do it all the time when enacting regulations meant to protect human life, at a cost. 

On the flip side, it’s common to price out the cost of saving a life as well.  Effective Altruism is a niche but deeply principled movement which goes through great lengths to, with exacting rigor, price out the most effective way to save lives (and then, generally, donate 80% of their salary, time, and organs to those causes).

GiveWell is one of them.  They annually put together a version of this spreadsheet which calculates which charities are able to do the most work with the fewest dollars. 

It’s worth checking out. The math is more complicated than a naive observer would expect. It turns out that the shittiest things nature can do to a person often doesn’t kill them — Hookworms reduce educational attainment, blind children (!), and reduce life earnings, but rarely if ever… kill anyone. But because being blind and poor really sucks, many of GiveWell’s “most effective” charities attempt to eliminate hookworms and similar parasites.  

The way to massage this math onto a linear scale is to compute dollars per QALY saved, where QALY stands for Quality Adjusted Life Year — the equivalent of one year in perfect health.  By this measure, saving the life of an infant who would otherwise die in childbirth may save 75 QALYs, while saving the life of a senile, bedbound 80 year old may save 15 Quality Adjusted Life Hours.

This is a reasonable and principled method of making financial investments.  If you put stock in the Efficient Politics Hypothesis of government, stop here and feel good about the choices we’ve made.


We have decided to, as a nation, spend an indefinite amount of time-money at home eating ice cream and mastrubating to Netflix, in a crusade to stop* the spread of COVID-19.  

*by some difficult to define metric

How much time-money?  $2.2 trillion is the down-payment stimulus package.  It’s a very conservative lower-bound estimate of the cost of recovery (it’s not expected that this will fix the economy by any means), so we can run with it.

How many lives are we saving?  The high upper bound is 2.2 million (assuming a 100% infection rate (not generally considered realistic), with a fatality rate of .66% (best estimate)).

This works out to a conveniently round, very lower bound, $1,000,000 per life saved (in the outcome that the US quarantine does prevent the majority of those deaths).  What about those QALYs? I’m not going to try sum it out, but we can look at a few suggestive stats:

  • The average age at death (due to COVID-19) in Italy is 79.5.  Italy’s average life expectancy is, for reference, 82.5*
  • 99% of Italian COVID-19 victims had pre-existing conditions (same source).

*I understand that the average life expectancy of a living 79 year old is higher than 82, which is why I’m not doing the math, so please shut up so we can move on.

We can speculate that no, we will not be saving very many QALYs.  But we can use the crude ‘lives saved’ metric instead to generously lower-bound our math, and run with 2.2 million raw.

Effectiver Altruism

My question: how does this calculus compare to effective altruism?  I was genuinely curious, because $1,000,000 per life saved is somewhat disproportionate to the charity pitches you see on television:


“Save a Child for Only $39 per day*”

Exit right.

*assuming 70 year life expectancy @ $1,000,000 per life

I tried to find a toplist of “problems we can solve for X dollars to save Y lives per year”.  I did not find one. GiveWell (entirely reasonably) calculates the payout of donating to a specific charity, not of speculatively eliminating entire facets of human suffering.

So I put together a list.  These numbers aren’t precise.  They are very speculative.  My goal was to understand the orders of magnitude involved.

My focus was on problems we could solve that don’t involve serious tradeoffs, and don’t require hard political choices.  Trying to solve “war”, “suicide”, or “alcoholism” don’t cost money per se, they require societal committment we can’t put a price tag on.  For the most part, this leaves diseases.

I started with the highest-preventable-death diseases in the developing world, and ended up with 7 “campaigns” where we could non-controversially plug in money on one end, and pull extant wretched masses out of the other.   When considering the payout in lives saved from eradicating a disease, I used 30 years, because using “forever” is unfair (I’m sure there’s a time-decay value on life an actuary would prefer, but this was simple, and it doesn’t really change the conclusion).

Global Hunger

Hunger is the stereotypical “big bad problem”, and it wasn’t hard to find data about deaths:

Around 9 million people die of hunger and hunger-related diseases every year, more than the lives taken by AIDS, malaria and tuberculosis combined.

(for the record, this gave me some good leads on other problems).  How much would it cost to actually fix hunger?

Estimates of how much money it would take to end world hunger range from $7 billion to $265 billion per year.  

Pulling the high estimate, we get… 

Price Tag$265 billion
Lives Saved9,000,000
Cost Per Life$29,444 / life 


Malaria sucks, and a lot of smart people want to spend money to get rid of it.  How many people does Malaria kill?

In 2017, it was estimated that 435,000 deaths due to malaria had occurred globally

What would it take to actually eliminate Malaria?

Eradicating malaria by 2040 would cost between $90 billion and $120 billion, according to the Gates Foundation

We can highball this estimate to get …. 

Price Tag$120 billion
Lives Saved13,050,000
Cost Per Life$9,195 / life


Tuberculosis is still a huge killer in the developing world, but it’s a killer we can put rough numbers on:

The Lancet study said reducing tuberculosis deaths to less than 200,000 a year would cost around $10 billion annually… a chronic lung disease which is preventable and largely treatable if caught in time, tuberculosis is the top infectious killer of our time, causing over 1.6 million deaths each year.

Price Tag$10 billion
Lives Saved42,000,000
Cost Per Life$7,143 / life

The math here is fuzzier than I’m comfortable with, but works out in the same ballpark as Malaria, so I feel OK about the result.


Again, this wasn’t the cleanest math puzzle, but this report pegs the cost of ending AIDs at $26.2 billion/year for 16 years.  At 770,000 deaths per year from AIDs, we can (again, more mathematically than I like, ballpark the bill and lives saved over 30 years:

Price Tag$366.8 billion
Lives Saved10,780,000
Cost Per Life$33,963 / life

Maternal mortality

Like Tuberculosis, it ends up in the same ballpark as Malaria, so I’m inclined to believe it’s not more than half-asinine.

Dying in childbirth is bad, and kills people.  How much public health spending would it take to eliminate it?  Again, it was really hard to find good estimates, but we find that 

Researchers at UNFPA and Johns Hopkins University calculated that the annual cost of direct services, such as paying for medical staff, drugs and supplies when a woman is giving birth, will reach $7.8bn (£6.2bn) by 2030, up from an estimated $1.4bn last year.

To save how many lives?  

About 295 000 women died during and following pregnancy and childbirth in 2017

I’m honestly not sure how to compare these numbers, but if we ballpark that the $7.8 billion saves at least that number (?) each year, we work out to 

Price Tag$7.8 billion
Lives Saved295,000
Cost Per Life$26,440 / life

If you don’t like the fuzziness of the math, feel free to ignore it, or multiply it by 10.  Or whatever.


Measles is bad.  To convince you to vaccinate your children, I will attach a picture of a child with measles.

In the US, measles doesn’t flat-out kill many people, but in the developing world, it does:

Worldwide more than 140,000 people died from measles in 2018

What would it cost to actually eradicate measles?

Eradicating measles by 2020 is projected to cost an additional discounted $7.8 billion and avert a discounted 346 million DALYs between 2010 and 2050

Using the 30 year window I’ve been using, we end up with:

Price Tag$7.8 billion
Lives Saved140,000
Cost Per Life$1,857 / life

This is a shockingly low number, and can only conclude either that (1) I messed something up, or (2)  that we are a terrible, horrible, species for not having eradicated this decades ago.

Global Warming

Stepping outside of diseases, what about something big?  Global warming is big.

How many deaths might be attributed to climate change in the next century?  Obviously this is a make-believe number, but the number is definitely at least ‘several’:  

A report on the global human impact of climate change published by the Global Humanitarian Forum in 2009, estimated more than 300,000 deaths… each year

This is the lowest bound I can find short of flat-out AGW denialism.  It’s easy to find genocide-level projections, assuming crop failures in the developing world several orders of magnitude higher.  I won’t use them.

What’s the cost of fixing global warming?  Long-term, there’s no good answer yet, because the technology doesn’t exist.  But there are (speculative) ways we can slow it down for reasonable sums via carbon sequestration: 

Returning that land to pasture, food crops or trees would convert enough carbon into biomass to stabilize emissions of CO2, the biggest greenhouse gas, for 15-20 years… With political will and investment of about $300 billion, it is doable

We can use these numbers to price tag the cost/payoff of delaying global warming:

Price Tag$300 billion
Lives Saved6,000,000
Cost Per Life$50,000 / life

This is the most speculative guesstimate of all, so if you want to ignore it too, feel free.

Compare & Contrast

My original goal was to build a snarky visualization game which invited users to bin-pack global problem solving which worked out to less than $2T.  I was foiled, because you could do literally everything on this list for less — by my (fuzzy) calculations, you could solve global hunger*, malaria, tuberculosis, delay global warming 20 years, cure AIDs, eliminate maternal mortality, and eliminate measles, for “only” $1.4T.

*to be fair, this one is annual, not a permanent elimination.

But I had already invested the time learning how to use Data Studio, so I made the chart anyway:

(you can play with it yourself here)


What I feel confident saying — even using wildly generous numbers, since I am:

  • using the absolute upper bound for US COVID-19 deaths,
  • using crude deaths for a disease which primarily affects the elderly, instead of QALYs when comparing to diseases which affect primarily the young,
  • using just one ($2.2T) of many recovery packages we’re going to pay for, and
  • generously upper/lower bounding all the numbers

is that 

The COVID-19 economic shutdown is 20x as expensive per life as any other public health intervention the US could fund.  The most expensive intervention on this list — “delaying global warming” — cost $50,000/head.  We’re paying $1,000,000/head for COVID-19.

Now, there is a range of valid value statements, depending on your priors and beliefs in how creative fiscal policy can be:

  • “We should do both”
  • “We don’t have money to do either”
  • “Maybe civilization was a bad idea”

I’m not claiming to be a superhero here.  I’m not an Effective Altruist, and probably don’t register as an altruist at all.  But cheap platitudes annoy me, especially when used to shut down arguments.

In the end, the most meaningful, easiest, way Cuomo could have qualified his Tweet would have been 

We will not put a dollar value on American life

It’s not a great look, or a great tweet.  But as far as I can tell, it’s the only way to make the numbers — maybe — add up.

You Should be Angry

If you are under the age of 30, in the west, your future has turned grim.  In a fair world, you would be in the streets, screaming to get back to your job and building your life.

But instead, you are inside, while the police sweep empty streets.

As of late March 2020, the economies of the US, and most of western Europe, have been shut down.  This action does not have precedent, and it will cripple a generation in poverty and debt. Short term, this will likely mean 20% unemployment, vast GDP contraction, and trillions in debt.

This price will be paid by those under 30, to save — some of — those over 80.

It is not necessary, and is not worth the price.  It was an instinctive reaction, and I hope history will not be kind to the politicians who caved to it.  The best time to stop this mistake was before it was made. 

The second best time is right now.

You are being lied to

We have been asked to shut down the US for two weeks — and similar timeframes in Italy, France and elsewhere.  Two weeks (15 days, per the Feds) is a palatable number. Two weeks is a long Christmas break.  The technorati elite on Twitter think the shutdown is a vacation, and for them it is, because their checking accounts are overflowing from the fat years of the 2010’s.

Two weeks is not the goal, and it never was the goal.

The Imperial College report is the study which inspired the shutdowns — first of the Bay Area, then all of California, then New York.   This report measured the impact of various mitigation strategies. For those not “in the know” (aka, normal humans) there are two approaches to treating this pandemic:

  • Mitigation, where we “flatten the curve” enough to keep ICUs full, but not overflowing.  Eventually, we will build up herd immunity, and disease persists at a low level.
  • Suppression, where we eliminate the disease ruthlessly and completely.  

You don’t have to read the paper.  This graph tells you everything you need to know:

The orange line is the optimal “mitigation” strategy.  We try to keep ICUs full, try to keep businesses and schools running, and power through it.  But people will die.

The green line is suppression.  We shut down businesses, schools, universities, and all civic life.  Transmission stops, because there is no interaction with the outside world.  The economy does not depress — it stops.

We aren’t following the orange line, because: people will die.

That is the IC report’s conclusion: no amount of curve flattening gets us through this pandemic in a palatable timeframe.  Thus, we must suppress — for 18 months or longer — until we have a vaccine.  I’m not paraphrasing. This is the quote:

This leaves suppression as the preferred policy option…. this type of intensive intervention package … will need to be maintained until a vaccine becomes available (potentially 18 months or more)

Italy, France, California, New York, Illinois, and more in the days to come, have nearly shuttered their economies.  All schools, universities, and social gatherings are cancelled, at risk of ostracization or police enforcement. This is the green line.

By enacting the green line — closing schools, universities, and businesses — the US is choosing to give up on mitigation, and choose suppression.  This doesn’t mean 2 weeks of suffering. It means 2 years to start, and years of recession to follow.

We are eating the young to save the unhealthy old

COVID-19 does not kill, except in the rarest of exceptions, the young.   Old politicians will lie to you. The WHO and CDC will lie to you — as they lied about masks being ineffective — to nudge you to act “the right way”.  Do not trust them.

Here are the real, latest, numbers:

In South Korea, for example, which had an early surge of cases, the death rate in Covid-19 patients ages 80 and over was 10.4%, compared to 5.35% in 70-somethings, 1.51% in patients 60 to 69, 0.37% in 50-somethings. Even lower rates were seen in younger people, dropping to zero in those 29 and younger.

No youth in South Korea has died from COVID-19.  Fleetingly few of the middle aged. Even healthy seniors rarely have trouble.  The only deaths were those seniors with existing co-morbidities.  In Italy, over 99% of the dead had existing illnesses:

With the same age breakdown for deaths as South Korea:

As expected, the numbers for the US so far are the same:

More than raw numbers, the percent of total cases gives a sense of the risk to different age groups. For instance, just 1.6% to 2.5% of 123 infected people 19 and under were admitted to hospitals; none needed intensive care and none has died… In contrast, no ICU admissions or deaths were reported among people younger than 20.

These numbers are under-estimates — the vast majority of cases were never even tested or reported, because the symptoms don’t even exist in many of the healthy.   The vast majority of the young would not even notice a global pandemic, and none — generously, “fleetingly few” would die.

The young — the ones who will pay for, and live through, the recession we have wrought by fiat — do not even benefit from the harsh medicine we are swallowing.  But they will taste it for decades. 

This is not even necessary

To stop COVID-19, the west shut itself down.  East Asia did not. East Asia has beaten COVID-19 anyway.

China is where the disease started (conspiracy theories aside).  Through aggressive containment and public policy, the disease has been stopped.  Not even mitigated — stopped:

There are two common (and opposite) reactions to these numbers:

  1. China is lying.  This pandemic started on Chinese lies, and they continue today.
  2. China has proven that the only effective strategy is containment

Neither is true.  But we also know that China can, and has, used measures we will never choose to implement in the west.  China can lock down cities with the military. China can force every citizen to install a smartphone app to track their movement, and alert those with whom they interacted.

So we can look at the countries we can emulate:  South Korea, Japan, Taiwan, and Singapore.  None (well, at most one) of them are authoritarian.  None of them have shut down their economies. Everyone one of them is winning against COVID-19.

South Korea

South Korea is the example to emulate.  The growth looked exponential — until it wasn’t:

South Korea has not shut down.  Their economy is running, and civic life continues, if not the same as normal, within the realm of normal.  So how did they win, if not via self-imposed economic catastrophe?  Testing.

The backbone of Korea’s success has been mass, indiscriminate testing, followed by rigorous contact tracing and the quarantine of anyone the carrier has come into contact with

Their economy will suffer not because of a self-imposed shutdown, but because the rest of the world is shutting itself down. 


Singapore won the same way: by keeping calm, testing, and not shutting down their economy.

Singapore is often the “sure… but” exception in policy discussions.   It’s a hyper-educated city-state. Lessons don’t always apply to the rest of the world. But a pandemic is different.  Pandemics kill cities, and Singapore is the world’s densest city. If Singapore can fix this without national suicide, anyone can.  So what did they do? 

  • Follow contacts of potential victims, and test
  • Keeping positives in the hospital
  • Communicate
  • Do not panic 
  • Lead clearly and decisively

I could talk about Japan and Taiwan, but I won’t, because the story is the same: Practice hygiene.  Isolate the sick. Social distance. Test aggressively.   

And do not destroy your economy.

“The economy” means lives

The shutdown has become a game — fodder for memes, fodder for mocking Tweets, and inspirational Facebook posts, because it still feels like Christmas in March.  

It is not.  If your response to the threat of a recession is:

  • “The economy will just regrow”
  • “We can just print more money”
  • “We just have to live on savings for a while”

The answer is simple: you either live a life of extraordinary privilege, are an idiot, or both.  I can try to convince you, but first, ask yourself:  how many people have the savings you’ve built up — the freedom to live out of a savings account in a rough year? I’ll give you a hint:  almost none.

Likewise, working from home is the correct choice, for anyone who can manage it. Flattening the curve is a meaningful and important improvement over the unmitigated spread of this disease. But the ability to work from home is a privilege afforded to not even a third of Americans:

According to the Bureau of Labor Statistics, only 29 percent of Americans can work from home, including one in 20 service workers and more than half of information workers

If you are able to weather this storm by working from home, congratulations — you are profoundly privileged. I am one of you. But we are not average, and we are not the ones who risk unemployment and poverty. We are not the ones who public policy should revolve around helping. The other 71% of Americans — who cannot — are the ones who matter right now.

To keep the now-unemployed from dying in the streets, we will bail them out.  And that bailout, in the US alone, just to start, will cost a trillion dollars.  That number will almost certainly double, at a minimum, over the next two years.  

What else could we spend two trillion dollars on?   To start, we could void all student debt:  $1.56 trillion, as of 2020.  We could vastly expand medicare or medicaid.  You can fill in the policy of your choice, and we could do it.  But we won’t.

We are borrowing money to pay for this self-inflicted crisis.  We should be spending that money investing in the future — perhaps freeing students from a life of crippling debt — but instead, we are throwing it at the past.

The rest of the world

The US is not the world.  The US will muddle through, no matter how poor our decisions, usually (but not always) at the cost of our futures, not our lives.  The rest of the world does not have this luxury.

GDP saves lives.  GDP is inextricably linked to life expectancy, child mortality, deaths in childbirth, and any other measure of life you want to choose.  This is so well proven that it shouldn’t require citations, but I’ll put up a chart anyway:

A depression will set back world GDP by years.  The US and Europe buy goods from the developing world.  The 2008 recession — driven primarily by housing and speculation in western markets — crushed the economies not just of developed nations, but the entire world:

we investigate the 29 percent drop in world trade in manufactures during the period 2008-2009. A shift in final spending away from tradable sectors, largely caused by declines in durables investment efficiency, accounts for most of the collapse in trade relative to GDP

If you are unswayed by the arguments that a self-inflicted depression will hurt the working poor in the US, be swayed by this — that our short-sighted choices will kill millions in the developing world.

How did we get here?

Doctors are not responsible for policy.  They are responsible for curing diseases.  It is not fair to ask them to do more, or to factor long-term economic policy into their goal of saving lives right now.   We elect people to balance the long-term cost of these decisions.  We call them politicians, and ours have failed us.

The solution to this crisis is simple — we do our best to emulate East Asia.  We isolate the sick. We improve sanitization.  We mobilize industry to build tests, ventilators, and respirators, as fast as we can — using whatever emergency powers are needed to make it happen.   And we do this all without shutting down the economy, the engine which pays for our future.

We do the best we can.  And accept that if we fail, many of the sickest elderly will die.  

Next time will be different.  We will learn our lessons, be prepared, and organize our government response teams  the way that Taiwan and South Korea have. We will have a government and a response which can protect every American.

But now, today, we need to turn the country back on, and send the rest (the 71% who can’t work from home) back to work. We owe them a future worth living in.