Knowledge graph and its role in AI

From: aidotengineer

Knowledge graphs are an increasingly vital component in modern artificial intelligence systems, particularly within the realm of Retrieval Augmented Generation (RAG). They offer a structured way to represent information and its complex relationships, addressing limitations of traditional semantic search methods [00:03:07].

What is a Knowledge Graph?

A knowledge graph is fundamentally a network that represents relationships between different entities [00:02:33]. These entities can be anything from people, places, concepts, to events [00:02:37]. The crucial aspect is the “edge of relationship” between two entities, which only graph-based networks or knowledge graphs can effectively exploit [00:02:58].

Knowledge graphs are essential for preserving wisdom by connecting information and creating a network of interconnected relationships [02:41:40]. They represent the thought process and a comprehensive taxonomy of a specific domain of expertise, enabling AI systems to offer advice rather than just retrieve data [02:49:50]. The universe itself can be thought of as a temporal network database with a graph structure, preserving causal links through relationships [01:36:54]. This ability to maintain causal links is key to solving hallucinations in AI and optimizing hypothesis generation and testing [01:38:44].

Knowledge Graphs vs. Traditional RAG

While RAG (Retrieval Augmented Generation) is not considered “dead,” its traditional vector database-based approach has limitations. If a basic RAG system can solve the problem in production, a more complex agent system may not be necessary [00:00:28]. However, traditional RAG systems based on vector databases primarily rely on semantic similarity, which may not be the same as relevance [02:26:50].

Advantages of Knowledge Graphs in RAG:

• Detailed Information Capture: Knowledge graphs capture information between entities in much greater detail, providing a comprehensive view of knowledge [00:03:49].
• Contextual Understanding: Unlike semantic vector databases, knowledge graphs excel at exploiting relationships between entities, which is crucial for deep contextual understanding [00:09:40].
• Data Organization: They have the ability to organize data from multiple sources effectively [00:04:14].
• Temporal and Relational Reasoning: Traditional RAG lacks native temporal and relational reasoning [00:40:56]. Knowledge graphs, especially those with temporal awareness like Graffiti, can model “why” things change over time and enable richer reasoning [00:41:20].
• Explainability: Graph RAG offers better explainability by visualizing and analyzing the structured nodes and semantics, unlike statistical probabilities from vector databases [02:29:28].
• Reduced Hallucination: Knowledge graphs can help reduce the hallucination rate in LLMs [00:33:51].
• Efficiency: Using knowledge graphs in conjunction with techniques like fusion and decoder can improve efficiency and lower costs [00:33:35]. Early studies show a three times improvement in accuracy with Graph RAG compared to SQL-based RAG [02:31:20].

Knowledge Augmented Generation (KAG)

KAG is an enhanced language model approach that integrates structured knowledge graphs for more accurate and insightful responses [02:55:00]. Unlike simple RAG, KAG understands and advises, rather than just retrieves [02:57:07]. Experts often have common ways of thinking and decision-making, which knowledge graphs can perfectly fit to store and leverage [02:59:01].

Building a Graph RAG System

Building a graph RAG or hybrid system involves four key components: data, data processing, graph creation (or semantic embedding vector database creation), and inferencing [00:04:31].

1. Data Processing and Graph Creation

The quality of data processing directly impacts the quality of the knowledge graph and subsequent retrieval [00:04:34].

• Triplet Creation: The goal is to create triplets (Subject-Predicate-Object) that define the relationship between entities, which is a strength of graph-based systems [00:03:23].
• LLM-powered Extraction: Large Language Models (LLMs) are crucial for extracting structured information and triplets from unstructured documents [00:07:26].
• Ontology and Prompt Engineering: Defining a clear ontology based on the use case and using precise prompt engineering are critical [00:07:46]. This stage is where developers might spend about 80% of their time to ensure a high-quality ontology and accurate triplets [00:08:11]. Noisy triplets lead to noisy retrieval [00:08:22].
• Data Cleaning: Small tweaks like removing apostrophes or other irrelevant characters can significantly improve results in triplet generation [01:53:08], leading to better accuracy [01:14:50].

2. Semantic Vector Database Creation

This is a more straightforward and well-studied process [00:08:53]. Documents are broken into chunks with appropriate overlap (to maintain context), converted into vector embeddings using an embedding model, and stored in a vector database [00:08:56].

3. Graph Enrichment

Once the base knowledge graph is constructed, it can be further enriched:

• Graph Algorithms: Running graph algorithms like PageRank, community detection, or link prediction can enrich data and identify cross-document topics or communities [02:36:39].
• LLM-Generated Summaries: LLMs can generate summaries for identified communities or clusters in the entity graph [02:36:49].

Retrieval in Graph RAG

Retrieval is the process of querying the knowledge graph to answer user questions [00:10:06].

• Multi-Hop Search: A crucial advantage of knowledge graphs is the ability to exploit relationships through multiple nodes (hops) [00:10:32]. While deeper hops provide richer context, they also increase retrieval latency [00:11:05]. Finding a “sweet spot” between depth and latency is important for production environments [00:11:16].
• Accelerated Search: Libraries like cuGraph (integrated with NetworkX) can accelerate graph searches, allowing for deeper traversal while reducing latency and improving performance [00:11:37].
• Hybrid Retrieval: A graph RAG retriever can perform an initial index search (e.g., vector, full-text) to find entry points in the graph, then follow relationships to retrieve additional context based on relevance and external user context [02:37:27]. Modern LLMs are increasingly trained to process graph structures, allowing them to handle node-relationship-node patterns as additional context [02:38:33].

Evaluating Performance

Evaluating a graph RAG system involves multiple factors and specialized tools:

• Metrics: Key metrics include faithfulness, answer relevancy, precision, recall, helpfulness, collectiveness, coherence, complexity, and verbosity [01:28:20].
• Ragas Library: Ragas is a Python library specifically designed to evaluate RAG workflows end-to-end, assessing the response, retrieval, and query interpretation [01:42:36]. It uses an LLM (defaulting to GPT but allowing custom models) for evaluation across several parameters [01:32:57].
• Reward Models: Specialized reward models, like Nvidia’s Lanimotron 340B, can judge the responses of other LLMs based on specific parameters [01:44:03].

Optimization Strategies

Improving the performance of a graph RAG system, especially the last 20% of the job, can take 80% of the time due to iterative fine-tuning [01:50:02].

• Knowledge Graph Quality: The better the knowledge graph, the better the results [01:51:21].
• LLM Fine-tuning: Fine-tuning LLMs can significantly improve the quality of generated triplets and overall accuracy. For example, fine-tuning a Llama 3.1 model improved accuracy from 71% to 87% in one experiment [01:57:07].
• Data Pre-processing: Simple data cleaning steps, like removing apostrophes or unnecessary characters, can lead to better triplet generation [01:58:34].

Applications and Use Cases

The choice between a semantic RAG system, a graph-based RAG system, or a hybrid approach depends on the data and the specific use case [01:14:14].

Data Characteristics

• Structured Data: Data from retail, FSI (Financial Services Industry), or employee databases often have good predefined structures, making them excellent use cases for graph-based systems [01:51:40].
• Unstructured Data: If a good knowledge graph can be created from unstructured data, experimenting with a graph path is worthwhile [02:06:58].

Specific Use Cases

• Complex Relationship Understanding: If a use case requires understanding complex relationships to extract information for responses, graph-based systems are beneficial [02:12:41].
• Agent Memory: Knowledge graphs are excellent for building real-time, dynamic, temporal memory for AI agents [00:38:52]. They enable agents to reason about temporal changes and causal relationships, which traditional vector database RAG cannot [00:40:52]. Zep’s Graffiti framework, for instance, uses temporal graphs for agent memory [00:41:45].
• Domain-Aware Memory: Allowing developers to model their business domain on the graph enables highly relevant memory retrieval specific to an application (e.g., financial goals for a finance coach vs. media preferences for a music assistant) [01:57:51].
• Competitive Analysis: Knowledge graphs can power expert AI systems for tasks like competitive analysis, turning data into strategy by mapping insights, experience, and situations [03:00:10].
• Network Operations: In complex network environments, knowledge graphs can represent network components, their relationships, and configuration data, serving as a digital twin for multi-agent systems to perform tasks like impact assessment and test plan generation [02:48:49].
• Legal Industry: Knowledge graphs are crucial for legal applications where accuracy and explainability are paramount [03:10:22]. They facilitate:
  • Legal Discovery: Structuring vast amounts of unstructured data (e.g., emails) into a graph format allows immediate identification of relevant information, augmenting it for visual decision-making by experts [03:15:31].
  • Case Research: Scraping web data, filtering it down to specific signals based on lawyer-defined schemas, and generating personalized reports is possible [03:19:21]. This enables early identification of potential lawsuits (e.g., product complaints) [03:21:03].

The Future of AI Memory

The future of AI memory lies in agentic systems that follow the network database principle, requiring graph databases to recover causality from fuzzy data [01:41:14]. Tools like GraphRack Chat Arena are being developed for simulating and evaluating evolving agentic graph memory [02:07:00]. The adoption of knowledge graphs in enterprise AI is trending upwards, as shown by Gartner’s hype cycle, indicating their increasing importance in breathing life into the AI ecosystem [02:30:12]. Leading organizations are already deploying production applications leveraging knowledge graphs [02:30:41].