Multiagent systems in technical workflows

From: aidotengineer

Multiagent systems are gaining prominence in technical workflows, offering enhanced capabilities beyond traditional single-agent or Retrieval Augmented Generation (RAG) approaches. They are particularly useful for tackling complex, multi-step tasks and integrating diverse data sources.

Definition and Purpose

A multi-agent system can be understood as a collection of agents that interact and collaborate to achieve a common goal [03:01:01]. For Yhar.ai, multi-agent systems are used to break down complex white-collar workflows, such as those in the legal industry, into specific, testable steps. Each step can have different requirements, frequencies, and states, often controlled by a graph [03:10:01].

Applications and Use Cases

AI Copilot Development

Multiagent orchestration can be used for developing AI copilots by handling complex taxonomies and decision-making processes [03:00:01].

Network Analysis

Cisco’s outshift group uses multi-agent systems to address challenges in network change management, aiming to reduce failures in production environments [02:47:17]. This includes tasks like impact assessment, testing, and reasoning about potential network failures [02:48:39]. The system integrates a natural language interface, a multi-agent system, and a network knowledge graph (digital twin) [02:48:49].

Key aspects of Cisco’s approach include:

• Data Ingestion: Processing data from various sources (controllers, SIM systems, telemetry) and transforming it into a standardized schema like OpenConfig [02:53:08].
• Layered Graph Structure: The knowledge graph is built in layers, allowing agents to access specific information efficiently without traversing the entire graph for every task (e.g., accessing raw configuration files for drift checks or multiple layers for reachability tests) [02:54:18].
• Agent Interaction: An “assistant agent” orchestrates tasks among other agents, such as a “query agent” that directly interacts with the knowledge graph [02:57:04].
• Performance Optimization: Fine-tuning the query agent with schema information and example queries significantly reduced token consumption and retrieval time, improving efficiency compared to initial RAG attempts [02:57:48].
• Standardization: Cisco is working on an open-source collective, Agency.org, to enable agents from different systems and vendors to communicate and integrate without heavy re-engineering [02:55:02].

Legal Industry Workflows

Yhar.ai uses multi-agent systems to automate complex workflows in the legal industry, particularly for finding class action and mass tort cases [03:07:03]. Their process involves:

• Web Scraping: Mass scraping of the web to gather information [03:18:43].
• Lead Qualification: Filtering and structuring scraped data based on specific schemas provided by law firms [03:19:05].
• Report Generation: Creating personalized reports for lawyers, backed by consistent, pruned graph schemas [03:20:01].
• Legal Discovery: Extracting and structuring information from large datasets like emails, which can then be augmented with visual graphs for expert review [03:16:19].
• Case Research: Identifying complaint patterns for products (e.g., cars catching fire) by tracking language across various online sources, building a density of complaints for specific issues, and detecting leads significantly earlier than traditional methods [03:21:00].

Agentic Firewalls in Cybersecurity

A proposed use case for multi-agent systems is an “agentic firewall” that provides a human language shell for interacting with machines in cybersecurity [02:02:21]. By plugging in agentic memory (temporal/episodic memory of shell history and machine context), agents can understand user behavior and control code execution [02:02:48]. This system would ensure that agents executing code in enterprise environments go through a controlled, secure pathway [02:05:39].

Benefits and Advantages

• Handling Complex Tasks: Multi-agent systems excel at breaking down complex workflows into manageable, testable steps, each with specific requirements [03:10:01].
• Orchestration and Decision Making: They provide a framework for orchestrating various tasks and enabling agents to make informed decisions [03:00:01].
• Accuracy and Control: In fields requiring high accuracy, like legal applications, multi-agent systems allow for precise control, structure, and schema, enabling pinpoint testing of what goes right or wrong at any point in time [03:10:40].
• Contextual Understanding: By integrating with knowledge graphs, agents can leverage rich contextual information and relationships, which is crucial for tasks like competitive analysis [03:01:01].
• Scalability: An open framework for building agents is considered critical for scalable systems [03:59:57].
• Overcoming LLM Limitations: Multi-agent systems help mitigate the probabilistic nature of LLMs by imposing structure and allowing for specific control [03:10:34].

Challenges and Considerations

• Workflow Imbalances: Some parts of a multi-agent workflow might be more critical or complex than others [03:11:02].
• Agent Quality: The effectiveness of a multi-agent system depends on the quality of individual agents. A poorly designed prompt can lead to a “bad agent” [03:11:12].
• Accuracy Chaining: The cumulative accuracy of sequentially chained agents can drop significantly, as a 95% accurate agent chained five times yields only 77% expected accuracy [03:12:16]. This highlights the challenge of decision-making under uncertainty [03:12:39].
• Evaluation: Establishing quantifiable metrics for evaluating the performance of multi-agent systems, especially in combination with knowledge graphs and digital twins, is an ongoing learning process [03:03:19].
• Integration with Existing Systems: Ensuring seamless integration of AI agents into existing infrastructure and support for multiple vendors is crucial for adoption [02:51:48].

Integration with Other AI Components

• Knowledge Graphs: Multi-agent systems frequently leverage knowledge graphs to manage state, provide structure, and control information across workflow steps [03:10:12].
• LLMs: LLMs can be used to “glue” together predefined tasks within multi-agent systems [03:09:56] or to translate commands into human language [02:02:16]. However, LLMs alone might not be sufficient for workflows demanding high accuracy [03:10:34].
• RAG (Retrieval Augmented Generation): While some initial attempts might use RAG for knowledge graph querying, fine-tuning specific agents can offer significant performance improvements in terms of tokens consumed and response time [02:57:48].
• Episodic Memory: The concept of agent continuations with episodic or temporal memory, like in “agentic firewalls,” allows agents to track historical events and user behaviors within a shell, providing crucial context for decision-making and security [02:03:26].
• Simulation Arenas: Since the evolution of agentic graph memory is unpredictable, simulation arenas (like a “web arena but for memory”) are necessary for evaluating and optimizing AI agents and workflows [02:07:00].