Synopsis

Agentic AI systems are increasingly being used to perform autonomous tasks such as planning, reasoning, tool use, code execution, information retrieval, and decision-making. Unlike conventional AI models, agentic AI can interact with external tools, APIs, databases, files, and other agents, creating new security risks. These systems can be vulnerable to threats such as prompt injection, indirect prompt injection, excessive agency, tool misuse, data leakage, privilege escalation, and malicious manipulation of agent behavior. However, systematic security testing and evaluation frameworks for Agentic AI are still relatively limited.

Relevance of the Topic

Currently, much of the AI research focuses on improving the accuracy, reasoning capability, and autonomy of AI agents. As Agentic AI moves toward real-world deployment, it is becoming equally important to determine whether these agents are secure and trustworthy.

An agent may produce a correct answer but still perform an unsafe action—for example, calling an unauthorized API, exposing sensitive information, executing malicious code, or following instructions injected through an external document or website.

Therefore, there is a strong need for research on security testing of Agentic AI, including identifying vulnerabilities, designing attack scenarios, evaluating agent behavior, and developing automated security testing methods.

Future Research/Scope

Students can investigate the security of Agentic AI systems by developing and evaluating different attack and testing scenarios.

Possible research directions include:

• Prompt injection and indirect prompt injection testing
• Tool/API misuse and unauthorized tool invocation
• Excessive agency and unsafe autonomous actions
• Data leakage and sensitive-information disclosure
• Privilege escalation in AI agents
• Agent-to-agent communication attacks
• Memory poisoning and context manipulation
• Malicious document/web-content injection
• Goal hijacking and instruction manipulation
• Automated security testing/benchmarking of AI agents
• Development of a security testing framework for Agentic AI
• Using mutation/fuzz testing to generate adversarial agent scenarios
•  

Skills Learned

• Agentic AI and LLM fundamentals
• Python programming
• Prompt engineering
• AI/LLM security
• Cybersecurity fundamentals
• API and tool integration
• Threat modeling
• Security testing
• Adversarial testing
• Evaluation and benchmarking
• LangChain/LangGraph or similar agent frameworks
• Experiment design and statistical evaluation

Relevant Courses to the Topic

• Artificial Intelligence
• Neural Networks / Deep Learning
• Cybersecurity
• Software Engineering
•  

Reading List

1. OWASP Top 10 for LLM Applications — particularly Prompt Injection, Excessive Agency, and Insecure Output Handling.
2. OWASP Top 10 for Agentic Applications — security risks specifically associated with autonomous AI agents.
3. NIST AI Risk Management Framework (AI RMF) — framework for managing AI security and trustworthiness risks.
4. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents — useful for studying security evaluation of tool-using agents.
5. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents — useful for studying indirect prompt injection and tool misuse.
6. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents — useful for understanding systematic security evaluation of LLM agents.