In other Anthropic news today, I’ve been thinking about this bill that Anthropic proposed where it’s not a show-stopper if only 50 people might be killed or seriously injured:

The bill would let the government block a model’s release or use only if it poses a foreseeable, imminent risk of more than $1 billion in damages, or of killing or seriously injuring more than 50 people, according to the report.

It’s weird to even put a number on this. But also, if gun manufacturers had to meet a similar standard before putting semi-automatic rifles on the market, the world would be a better place.

@manton TBU (true, but useless) we’ve tried to move this discussion many times in the US but the destruction to human lives is, as they say, “priced in”. The same for cars and alcohol. The same is not true for whatever we’re calling OpenAI and Anthropic.

@manton Probably inspired by MIL-STD-882E or similar (see Table III, page 13), although $1B is 100x the “catastrophic” threshold in 882E, which also requires only 1 death, not 50.

Example: 20 deaths and $500M in damage is way beyond DOD’s definition of catastrophic, but below Anthropic’s threshold.

@manton The more important takeaway, at least from the reporting, is that in their proposal there’s no graduated response. It’s “nothing can be done” all the way up to $1B/50 deaths.

Meanwhile, we already have at least one existing, commonly used standard for mitigating risks with far more nuance.

@jatowler Interesting, thanks for the link to that.

@manton That is utterly depraved. Remind me why we are comparing computer systems with guns and considering military standards? How about full liability for anything more than $50 in damage? Death or injury to any human should not even be on the table. Unacceptable.

@manton I assume part of the reason for specifying this, is to distract people and make them argue about what levels of damage are reasonable - thereby already accepting Anthropic's preferred framework.

@manton At the end of the day all kinds of systems have to do grim estimates like this (value of a soldier in war etc).

But we wouldn’t want to inhibit self-driving cars, proven safer than humans already, just because some will inevitably have tragedy strike.

There’s also wide variance in cause and direction. Human directing AI vs. it self directing are very different situations.

No technology will be perfect, this one will likely have the biggest impact on human longevity than others ever over time with medicine and other things. You can’t have liability so low we never get that future benefit at all.