Data governance and compliance in AI

From: aidotengineer

AI agents are autonomous systems capable of reasoning, creating their own workflows, performing tasks, and using tools to make API calls [00:01:09]. Tasks are specific actions that might utilize large language models (LLMs), Retrieval-Augmented Generation (RAGs), or other tools [00:01:19]. Tools are functions designed to retrieve data from the internet, databases, or service APIs [00:01:33], while memories are contexts shared across agents, tasks, and tools [00:01:39].

Security Challenges in AI Agents

A significant challenge arises because most agent frameworks operate as a single process, meaning the agent, tasks, and tools reside within the same environment [00:02:15]. This configuration means that if a tool requires access to a database or API, it needs to possess the necessary credentials or share tokens [00:02:24]. These credentials often carry super admin privileges [00:02:30].

This shared process introduces security vulnerabilities, as one tool could technically access credentials or prompts intended for another [00:02:49]. This creates a “zero-trust” issue and a high attack surface [00:03:33]. Given the autonomous and non-deterministic nature of AI agents, which can generate their own workflows, predicting their exact actions becomes difficult, leading to “unknown unknowns” in security [00:03:30].

Key challenges include:

• Security: Improperly designed or implemented agents can lead to unauthorized access and data leakages of sensitive and confidential information [00:03:58].
• Safety and Trust: Unreliable models or an unsafe environment where prompts can be altered may lead to incorrect results [00:04:14].
• Compliance and Governance: Many organizations struggle with making AI agents enterprise-ready due to neglected compliance and governance requirements [00:04:26]. For instance, a credit bureau customer treats AI agents similarly to human users, requiring adherence to strict onboarding, training, and regulatory processes, including regional data regulations (e.g., California resident data consent, Europe data access rules) [00:05:21]. Without such adherence, agents cannot be moved into production [00:05:24].

Multi-Layered Solutions for Safe and Reliable AI Agents

Addressing these challenges requires a multi-layered approach, as there is “no silver bullet” in security and compliance [00:05:46]. The proposed solution involves three layers: pre-production evaluation (evals), in-production enforcement, and post-deployment observability [00:05:50].

1. Pre-Production Evaluation (Evals)

Evals determine the criteria for an agent to enter production, focusing on security and safety rather than just model performance [00:06:12]. The goal is to generate a risk score to decide if an agent can be promoted to production, even if it’s a third-party agent [00:06:30].

Similar to traditional software development, AI agents require:

• Code Quality: Ensuring adequate test coverage [00:07:41].
• Vulnerability Scanning: For Docker containers and third-party software, including checking for Common Vulnerabilities and Exposures (CVEs) [00:07:58].
• Penetration Testing: To identify vulnerabilities like cross-site scripting [00:08:06].

Specific evals for AI agents include:

• Use Cases and Baselines: Defining clear use cases and ground truth to ensure that changes (e.g., prompt modifications, new libraries, frameworks, or LLMs) do not alter the baseline behavior [00:08:27].
• LLM and Library Vulnerability: Ensuring third-party LLMs are not poisoned and are scanned for vulnerabilities [00:08:36]. Third-party libraries must meet minimum vulnerability criteria [00:08:45].
• Prompt Injection Testing: Verifying that the application has controls to block prompt injections, even if LLMs are generally doing this [00:09:04].
• Data Leakage Evals: Especially critical in enterprise AI, agents must not leak sensitive or confidential data. This requires testing to ensure malicious users cannot exploit loopholes to access unauthorized information (e.g., an HR agent leaking salary benefits) [00:09:53].
• Unauthorized Actions: For agents that can initiate actions or change data, ensuring these actions are performed by authorized entities [00:10:11].
• Runaway Agent Testing: Testing for scenarios where agents enter infinite loops due to bad prompts or task/agent configurations [00:10:32].

These evaluations collectively produce a risk score, providing confidence in an agent’s readiness for production [00:10:40].

2. In-Production Enforcement

Enforcement ensures that an agent’s strong build is maintained during operation, particularly in a zero-trust environment [00:10:57].

• Authentication and Authorization: Crucial for preventing impersonation and theft of confidential information [00:11:44]. When a user requests an agent, the user’s identity must be propagated through the agent, tasks, and tools to the final API call or database access point [00:11:52]. This ensures that the agent’s actions and data access are enforced based on the user’s permissions, preventing agents from exceeding their designated roles or accessing data the user is not authorized for [00:12:20].
• Approvals: Agents can automate many tasks, but there’s a need for automated approval mechanisms [00:12:51]. This can involve another agent overseeing approvals, with defined thresholds for automatic approval and guardrails to involve human intervention when limits are exceeded [00:13:11].

3. Post-Deployment Observability

Observability is vital for AI agents due to the numerous variables involved and the rapid evolution of models and frameworks [00:14:25].

• Continuous Monitoring: Unlike traditional software, AI agent behavior is highly subjective to user input [00:14:35]. Monitoring is needed to track how user inputs affect responses and to detect any anomalous outflow of sensitive or confidential data [00:15:11].
• Thresholds and Metrics: It’s impractical to monitor every request [00:15:19]. Establishing thresholds and metrics, such as failure rates, can trigger alerts for investigation when anomalies occur, potentially indicating misbehaving agents or malicious users [00:15:53].
• Anomaly Detection: Similar to user behavior analytics in traditional security, anomaly detection for agents aims to ensure they operate within accepted boundaries [00:16:24]. This leads to a real-time security posture, indicating how well the agent is performing in a live environment [00:16:33].

In summary, a comprehensive approach involves preemptive vulnerability evaluation for a risk score, proactive enforcement with guardrails and sandboxing for secure operation, and robust observability for real-time monitoring and anomaly detection to fine-tune agents [00:17:24]. Private AI has open-sourced their Safety and Security Solutions called page.ai to contribute to this field [00:17:34].