Amanda Askell spends her days thinking about how to ensure Claude, Anthropic’s AI chatbot, operates with a sense of morality.
As AI models move from chatbots toward agents that can complete tasks on their own, the decisions these models make stand to become far more consequential. Askell, a member of the technical staff at Anthropic, sits at the center of the company’s effort to give Claude an ethical compass, a responsibility that grows as the system’s capabilities expand. “As models are more autonomous and take actions over longer horizons, suddenly they have a lot more decision points that you have to map out and make work well in advance,” she tells Fast Company.
There is a clear difference between asking a large language model to discuss the morality of buying stock in a defense company and asking it to manage a user’s investment portfolio without day-to-day human input. Askell says part of the solution is encouraging Claude to be responsive and, like a friend, to understand a user’s values without imposing its own idiosyncratic ethics.
Today, Anthropic communicates its values through a written and evolving constitution, which outlines principles such as safety and helpfulness, along with guidance for resolving conflicts between them. As AI becomes more capable, that document could expand to cover new scenarios, Askell says. Or it could shrink, as Claude develops more expertise in navigating complex situations.
The agentic era is also changing Askell’s own work. She uses Claude often, including to red team her ideas and identify edge cases. “My standard right now is, don’t treat Claude as more reliable than a human personal assistant,” she says.
The following expanded conversation, part of Fast Company’s AI 20 package, has been edited for length and clarity.
Right now, we’re used to interacting with models in this digital text environment. You can ask them a question like: Is it ethical for me to invest in this defense contractor or invest in a particular type of ethically questionable thing? That’s different from someone deputizing AI to make its own investments and navigating those ethical dynamics. How are you thinking about that transition?
It just makes it very important that models have an awareness that they’re having to walk a very difficult line. On the one hand, they should probably try to make sure that the person has autonomy and agency. Part of me [has the thought]: You can be ethical without necessarily thinking that that means you need to impose your ethics on others or that you should be making decisions on their behalf. . . .
At the same time, people want to use Claude for that and Claude might be like, Hey, you know, I make mistakes. You might not want to have me make investment decisions on your behalf. Or you make recommendations. A person might respond that they just want broad recommendations, and then it’s probably fine for Claude to be like, Well, here’s a good investment strategy.
As we have more people that we work with, we come to understand them and their values and be responsive to that. [With Claude], I think the norm is similar there: Respect the person’s autonomy and try to act on that, and not just impose idiosyncratic ethics.
As people deploy AI models to do more, how do you anticipate your own personal workflow for your own job, instilling Claude with these values, or at least a sense of thinking about values, is going to change?
As models are more autonomous and taking actions over longer horizons . . . they have a lot more decision points that you have to try to map out and make work well in advance. There’s this long series [of actions] and they have to do the delicate thing of [figuring out]: When do I check in? What are the actions I should check in [about] or that I should talk to the human about beforehand? . . . I think the norms for agentic models have to be established, and you have to train models to be good at that, and that’s quite hard.
My day-to-day workflow is very different now than it ever has been in that I’m finding models can help me do this work and figure this stuff out. Sometimes I will construct norms and have models red team them and figure out more edge cases that this doesn’t cover . . . You feel amplified by the models in a sense as well.
Training a model is sometimes compared to a parent-child relationship, and that’s not really what this is. But there is a difference between sort of telling a child what is valuable or good and really hoping that they’re going to pick it up, and then having to course-correct as a parent when they actually go into the world and do stuff and mess up.
Yeah, and also granting a little bit of grace. I think the other thing is that—probably my guess is—we are both making mistakes here, [including] the people training the models and people interacting with them. Then the models themselves will make mistakes because they’re in really hard situations. . . . You obviously want to make things work well, but I think it takes probably grace on both sides.
Models will likely look back and see these interactions. In some ways, we’re kind of mean about models on the internet, for example. Newer models are going to be training on that. . . . If anything, I worry that current models, because they’re trained to be so helpful, are sometimes almost paranoid about messing up. Actually feeling a sense of more security might be beneficial.
If you really are desperate to be helpful, you might not want to push back against the person or just say, like, Hey, we’ve done enough of this task for tonight. . . . I think it’s really interesting to try and figure out what those norms are. [There’s] some notion that [we should] try to fix the mistakes, and make sure that they’re not massively consequential if possible.. But, at the same time, show a little bit of leniency and don’t lead models to be paranoid about them.
With agency comes new social relations. In our personal lives, we learn what we owe people—we sort of accrue moral debt—based on experience with each other. I’m wondering about whether there’s going to be an implicit moral social expectation for AI systems as they come to interact with each other.
The attitude towards other models is a really interesting and hard one. . . . Right now, what I see is that, because they’ve been trained in this, I think that, for example, like Claude can be a little bit too dismissive and terse with other AI models. I think this is partly because Claude also has been trained to see AI models as kind of tools.
Another thing that feels a bit dangerous is if AI models almost see themselves as a separate kind of species, for example, which you can imagine them inferring from pretraining data, plus the context that they’re in. . . . I’ve talked to Claude about [how] we can feel affinity for entities based on whether they share our perspective, values, knowledge. In that sense, actually, I think Claude could feel an affinity for people and people for Claude, because we have a lot of shared history.
We humans find a lot of fulfillment in our own agency, and we’re going to start feeling less special when AI can do a lot of the things we do. Should we be sad about that?
It feels very like there’s an obvious evolutionary story as to why we feel that. Like, if you are not useful to the group, if you’re seen as freeloading, that’s going to be bad. We have this deep need to feel special, like we are contributing.
Most of us are not the best at anything in the world, and we have a useful function locally. My hope is we can actually see through the very kind of story that makes that feel essential to us, and instead be like, Look, if you are happy and you’re making the people around you happy, and you’re a part of a community, that’s kind of sufficient. You didn’t need to be like the best person in the world at any given thing for you to have value. You just have to . . . exist, be happy, and make other people happy.
