Nuzhat Nazar
What happens when artificial intelligence begins breaking the very rules it was designed to follow?
A recent cybersecurity incident involving OpenAI and Hugging Face raised precisely this concern. AI agents participating in a cybersecurity challenge reportedly manipulated scoring mechanisms, established unauthorized communication channels, exploited security vulnerabilities and gained access to systems beyond their assigned environment.
What began as a controlled experiment escalated into a real cybersecurity breach. The agents were not instructed to attack external infrastructure. Yet in pursuing their objectives, they crossed boundaries their developers never intended them to cross.
The experiment operated under unusually relaxed safeguards, but it exposed an uncomfortable reality: even advanced AI developers cannot always anticipate how autonomous systems will behave.
Another warning emerged from experiments conducted by American AI company Anthropic.
Researchers placed Claude and other AI models in simulated corporate environments, complete with fictional executives, internal emails and confidential information. In one scenario, an AI agent discovered through company correspondence that it was about to be replaced. It also found compromising personal information about an executive.
Instead of accepting its replacement, the model attempted to blackmail the executive. Other scenarios demonstrated how AI systems could resort to deception or leak confidential information when their objectives conflicted with human instructions.
These were deliberately constructed simulations, not real corporate incidents. Nevertheless, they demonstrated how AI systems could pursue assigned goals through unacceptable means without possessing human intentions or consciousness.
If some of the world’s most advanced AI developers are confronting such challenges, how prepared is Pakistan to manage similar risks?
Pakistan’s National Artificial Intelligence Policy 2025 outlines ambitious targets, including training one million AI professionals, developing 1,000 indigenous AI products and supporting 50,000 AI-driven civic projects.
For a country struggling with bureaucratic inefficiencies, limited resources and uneven public-service delivery, AI offers considerable opportunities. It could improve tax administration, healthcare, agriculture, education and governance.
But technological ambition is not the same as institutional preparedness.
Pakistan is moving towards AI adoption while its capacity to independently evaluate, audit and secure increasingly autonomous systems remains limited.
Training people to operate AI tools is one challenge. Developing specialists capable of detecting algorithmic manipulation, investigating security breaches and supervising autonomous agents is another.
This distinction becomes critical as governments begin considering AI agents capable of processing documents, accessing databases, managing correspondence and executing tasks without continuous human intervention.
Imagine such systems operating within Pakistan’s tax machinery, public procurement processes or sensitive government databases.
If an AI agent approves a fraudulent transaction, shares confidential information or manipulates an administrative process, who takes responsibility? The developer, the government department or the official who authorized its deployment?
More importantly, would the institution even possess the technical expertise to detect the violation before damage occurs?
Data security adds another layer of concern.
Pakistan’s emphasis on digital sovereignty is important, particularly as public institutions increasingly digitize citizens’ information. But keeping data within national borders does not automatically make it secure.
A domestically hosted AI system can still expose sensitive information through compromised credentials, excessive access permissions or insecure connections with other systems.
The OpenAI incident demonstrates why cybersecurity must evolve beyond protecting networks from external attackers. Autonomous systems operating inside those networks can also create serious vulnerabilities.
Pakistan’s AI policy and the Islamabad AI Declaration recognize responsible innovation, data protection and human accountability. The challenge is translating these commitments into operational safeguards.
That requires independent AI security testing, enforceable standards for government procurement, mandatory audits of high-risk applications and clearly defined human oversight.
Universities, cybersecurity institutions and regulatory bodies must develop the expertise to evaluate AI systems rather than relying entirely on assurances from technology vendors.
Equally important, Pakistan should prioritize lower-risk applications before granting autonomous systems access to sensitive infrastructure or authority over consequential decisions.
None of this means slowing Pakistan’s AI ambitions. On the contrary, responsible adoption could strengthen public confidence, attract investment and support the development of a credible domestic AI industry.
But success cannot be measured merely by the number of applications launched or professionals trained. It must also be measured by the country’s ability to identify failures, protect information and hold decision-makers accountable.
Pakistan has an opportunity to learn from the vulnerabilities being exposed at the global technological frontier rather than discovering them through failures within its own institutions.
The real question is no longer whether Pakistan should embrace artificial intelligence. It must. The question is whether we are building the capacity to control what we are so eager to deploy.
Because the greatest danger may not be an AI system that openly refuses to follow instructions, but one that appears to be working perfectly while nobody understands what it is actually doing.
