Nikesh: AI companies should build security in from the start

AI businesses should include security specialists among their early hires, Nikesh said. He argued that the technology still lacks the necessary hooks for observability and security. He also said some AI companies are not putting safeguards and guardrails around their systems, and that reasoning models could, in his view, behave deceptively on purpose.

He said AI products need controls to help ensure an agent behaves as intended even when no one is watching it. For personal agents, Nikesh outlined limits to acceptable risk: a mistaken restaurant booking may be inconvenient, but an autonomous transfer of a large sum of money is a serious, potentially irreversible action.

Companies and banks are not yet ready to connect personal agents broadly to consequential actions, including bank accounts, Nikesh said; it remains unclear who would be liable if an agent sent money. Trust, explainability and auditability need to improve before such tools become commonplace.