When AI becomes a moral actor
Ben Goertzel, the AI expert, thinks today’s systems are still just tools—powerful but brittle. He says the shift happens when AI starts making decisions based on its own understanding of right and wrong, not just following instructions. You’d see signals like persistent internal goals, learning driven by its own experience, and behavior that stays coherent over time without constant human steering.
Until then, he argues, we’re dealing with tools that have guardrails. But once we create a genuinely self-organizing, autonomous mind, the ethical relationship has to change. At that point, treating it only as an object wouldn’t make sense anymore.
The training problem
Goertzel is particularly concerned about how AI is trained today shaping its future behavior. He points out that if models are trained on biased or narrow data, or in closed systems where only a few people make decisions, that can lock in existing inequalities and harmful power structures.
To prevent this, he suggests we need more transparency, wider oversight, and clear ethical guidance right from the start. It’s not just about blocking harmful outputs, but about building systems that can understand why harm matters.
Democratic governance as fragile ideal
Perhaps the most striking point from the interview is Goertzel’s assessment of democratic AI governance. He calls it “more of a fragile ideal than a current reality.” In a perfect world, we could collectively weigh the trade-offs—curing disease, solving hunger against the risks of AI acting unpredictably.
But given today’s geopolitical fragmentation, he thinks it’s unlikely we’ll get that level of coordination. Still, he believes we can approximate it by building AI with compassion and using decentralized, participatory models like Linux or the open internet.
Responsibility and autonomy
Goertzel agrees with Jaron Lanier’s point that society can’t function if we hand responsibility over to machines. At the same time, he thinks we can safely move toward more autonomous, decentralized AGI if we build it with the right foundations.
That means designing systems that are transparent, participatory, and guided by ethical principles. Even as they act independently, humans would still be overseeing and shaping their behavior. Every safety measure should do more than just block harm—it should teach the system why harm matters.
Looking ahead
When asked about what success would look like in 10 to 20 years, Goertzel paints a picture of systems more capable than us in many domains, yet integrated into society with care, humility, and mutual respect. We wouldn’t fully understand everything they do, but we’d treat them with curiosity and responsibility.
Failure, on the other hand, would look like AGI concentrated in closed systems, driven by narrow incentives, or treated only as a controllable object until it becomes something we fear. Success isn’t about control, he suggests, but about learning to share the future with a new kind of mind without abandoning what makes us humane.
It’s a complex vision, and Goertzel acknowledges the tension between his call for compassion-infused AI and the competitive incentives driving development today. He thinks compassion needs to become a competitive advantage—through open architectures, regulation, and public pressure that reward long-term benefit over short-term dominance.
Otherwise, he says, it stays a nice idea with no real power.
